From 77ddf260bf9a3de3dcb8456a60f4f809d67401fd Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Mon, 16 Nov 2026 10:08:11 -0600 Subject: [PATCH] fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid Runner Full-Stack E2E workflow proves chat features against a real source server > - Since master f7a3497e9d, every paid cell fails before any test runs: the source server cannot find `@paperclipai/plugin-sdk/dist/index.js` > - The supervisor builds the plugin SDK before it starts Paperclip, but its own static imports now load the SDK first > - This pull request removes the heavy import edge, adds a regression test for that class, and builds and checks the SDK once per campaign > - The benefit is that paid campaigns run again, and a missing workspace build fails once in the build job instead of in 199 paid cells ## Linked Issues or Issue Description Refs #21791 **What happened?** Chat-eval cohort 17 (master f7a3497e9d) failed in all 199 cells. Each cell exited in the "Run paid cell" step. The private Playwright log shows `ERR_MODULE_NOT_FOUND` for `server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported from `server/src/services/reusable-sandbox-resume-error.ts`, and then `Process from config.webServer was not able to start`. Cohort 16 (master 819f61f1fd) passed 169/199 with the same workflow steps. **Root cause** #21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"` to `native-runner-checkpoint.ts`. `tests/runner-e2e/server.ts` statically imports `controller-recovery-observer.ts`, which imports the checkpoint reader. The new edge connects it to the full heartbeat graph: `native-runner-checkpoint -> native-controller-recovery -> native-finalization-notifications -> external-agents -> muse -> heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error -> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated `dist`. The supervisor builds that `dist` in `prepareSourceServerDependencies`, but ES static imports load before that call. The workflow installs with `--ignore-scripts` and never built the SDK, so the supervisor crashes at load. #21554 is not involved: it changes only release workflows. **Expected behavior** The supervisor loads without generated SDK output. The campaign build job builds and checks the SDK once, and each cell restores it. **Steps to reproduce** 1. Remove `packages/plugins/sdk/dist`. 2. Run `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/server.ts`. 3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error, which shows that all static imports loaded. ## What Changed - Moved `NATIVE_RETAINED_OWNER_WAIT` and `NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module `native-recovery-wait-codes.ts`. `native-controller-recovery.ts` re-exports them, so other importers do not change. `native-runner-checkpoint.ts` imports the leaf. This also removes a checkpoint/controller-recovery import cycle. - Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses esbuild to walk the static import graph of `tests/runner-e2e/server.ts` and fails with the full chain if it reaches `@paperclipai/plugin-sdk`. - `runner-full-stack-e2e.yml`: the "Build reusable runner campaign artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same certified helper the cell uses), checks that `shared` and `plugin-sdk` dist and their completion markers exist with a clear `::error::`, and ships both dist trees in the campaign bundle. The cell restore step checks both entry points with a clear `::error::`. The helper's fingerprint receipt lets the cell reuse the shipped output instead of a rebuild. - `workflow-security.test.ts` pins the build, packaging and guard. ## Verification - Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts server-startup-import-graph` fails with the old import and passes with the fix. - Local reproduction above: fails before, loads after. - `npx vitest run --config tests/runner-e2e/vitest.config.ts workflow-security server-startup-import-graph native-completion`: 137 passed. - `node --test tests/runner-e2e/native-completion-git-source.test.mjs`: 36 passed. - Server `tsc --noEmit`: clean. - Paid proof of the source fix: one cell (`chat-features.runner-opencode.local.access.private-resource`), dispatched with `--ref master -f target_branch=fix/runner-e2e-plugin-sdk-dist`: https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That run uses the master workflow and this branch's source, so it proves the import-graph fix alone restores the cells. - The workflow changes (SDK build, bundle and guards) cannot run in a paid campaign before merge, because paid runs use the default-branch workflow. `workflow-security.test.ts` pins them. A 1-case paid check from master follows the merge. - The failed "Authorize paid campaign" check on this commit is from a dispatch with a branch `--ref`. The default-branch gate rejected it as designed. It is not a PR-event check. ## Risks - Low risk. The constants keep their values and are re-exported from the old module. - The campaign bundle grows by the `shared` and `plugin-sdk` dist trees (small). If the fingerprints do not match in a cell, the helper rebuilds, as before. ## Model Used - Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 --- .github/workflows/runner-full-stack-e2e.yml | 18 +++++++ .../native-controller-recovery.ts | 5 +- .../native-recovery-wait-codes.ts | 5 ++ .../native-runner-checkpoint.ts | 2 +- .../server-startup-import-graph.test.ts | 51 +++++++++++++++++++ tests/runner-e2e/workflow-security.test.ts | 5 ++ 6 files changed, 83 insertions(+), 3 deletions(-) create mode 100644 server/src/services/native-runtime/native-recovery-wait-codes.ts create mode 100644 tests/runner-e2e/server-startup-import-graph.test.ts diff --git a/.github/workflows/runner-full-stack-e2e.yml b/.github/workflows/runner-full-stack-e2e.yml index 297f74da7ae..df00eb932fa 100644 --- a/.github/workflows/runner-full-stack-e2e.yml +++ b/.github/workflows/runner-full-stack-e2e.yml @@ -797,6 +797,16 @@ jobs: if [ "$NEEDS_NATIVE_BINARIES" = true ]; then pnpm --filter @paperclipai/paperclip-runner build:runner-binaries fi + # The source server imports the plugin SDK from its generated dist. + # Build it once here, with the same certified helper the cell uses, + # instead of in every paid cell. + node scripts/ensure-plugin-build-deps.mjs + for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do + if [ ! -f "$entry" ] || [ ! -f "$(dirname "$entry")/.paperclip-build-complete" ]; then + echo "::error::Workspace build output $entry is missing; paid cells would fail before any test runs." + exit 1 + fi + done - name: Package immutable campaign outputs env: @@ -814,6 +824,8 @@ jobs: ) archive_paths=( packages/paperclip-eval-kernel/dist + packages/shared/dist + packages/plugins/sdk/dist ) if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then test -d packages/paperclip-runner/dist @@ -1235,6 +1247,12 @@ jobs: --file runner-e2e-build/runner-e2e-build-bundle.tar.gz \ --directory "$GITHUB_WORKSPACE" test -d packages/paperclip-eval-kernel/dist + for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do + if [ ! -f "$entry" ]; then + echo "::error::Campaign outputs lack $entry; the source server cannot start without it." + exit 1 + fi + done if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then test -d packages/paperclip-runner/dist fi diff --git a/server/src/services/native-runtime/native-controller-recovery.ts b/server/src/services/native-runtime/native-controller-recovery.ts index c7523d642f0..806cc82b1b0 100644 --- a/server/src/services/native-runtime/native-controller-recovery.ts +++ b/server/src/services/native-runtime/native-controller-recovery.ts @@ -11,6 +11,9 @@ import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBoo import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js"; import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js"; import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js"; +import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js"; + +export { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT }; export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024; @@ -68,7 +71,6 @@ export type NativeControllerRecoveryOutcome = | { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown } | { kind: "no_prior_execution" }; -export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner"; export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string }; @@ -138,7 +140,6 @@ export async function deferNativeControllerRecovery(db: Db, input: { }); } -export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy"; export async function deferNativeRunForBusyAgentDirectory(db: Db, input: { companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number; diff --git a/server/src/services/native-runtime/native-recovery-wait-codes.ts b/server/src/services/native-runtime/native-recovery-wait-codes.ts new file mode 100644 index 00000000000..cd2b7d46422 --- /dev/null +++ b/server/src/services/native-runtime/native-recovery-wait-codes.ts @@ -0,0 +1,5 @@ +// Dependency-free failure codes shared by native recovery modules. Keep this a +// leaf: checkpoint readers import it, and those readers are also loaded by +// standalone tools that must not pull in the heartbeat service graph. +export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner"; +export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy"; diff --git a/server/src/services/native-runtime/native-runner-checkpoint.ts b/server/src/services/native-runtime/native-runner-checkpoint.ts index 3a76c84ac17..4a93568dc75 100644 --- a/server/src/services/native-runtime/native-runner-checkpoint.ts +++ b/server/src/services/native-runtime/native-runner-checkpoint.ts @@ -9,7 +9,7 @@ import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from " import { legacyControllerBootId } from "../legacy-controller-lease.js"; import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js"; import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js"; -import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"; +import { NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js"; import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js"; type Binding = { companyId: string; issueId: string; runId: string }; diff --git a/tests/runner-e2e/server-startup-import-graph.test.ts b/tests/runner-e2e/server-startup-import-graph.test.ts new file mode 100644 index 00000000000..d5d1e040f63 --- /dev/null +++ b/tests/runner-e2e/server-startup-import-graph.test.ts @@ -0,0 +1,51 @@ +import path from "node:path"; +import { build, type Plugin } from "esbuild"; +import { describe, expect, it } from "vitest"; + +const repositoryRoot = path.resolve(import.meta.dirname, "../.."); + +// The source-server supervisor builds the plugin SDK (scripts/ensure-plugin-build-deps.mjs) +// before it starts Paperclip. Its own static imports load before that build, so they must +// never reach a workspace package whose entry point is generated output. When they did, +// every paid source cell failed with ERR_MODULE_NOT_FOUND before any test ran. +const BUILT_ONLY_PACKAGES = ["@paperclipai/plugin-sdk"]; + +async function staticImportPath(entry: string, target: string) { + const reached = new Map(); + const externalBare: Plugin = { + name: "external-bare", + setup(context) { + context.onResolve({ filter: /^[^./]/ }, args => { + if (args.path.startsWith("@paperclipai/") && !BUILT_ONLY_PACKAGES.some(name => args.path === name || args.path.startsWith(`${name}/`))) return undefined; + if (args.path === target || args.path.startsWith(`${target}/`)) reached.set(args.path, args.importer); + return { path: args.path, external: true }; + }); + }, + }; + const result = await build({ + entryPoints: [path.join(repositoryRoot, entry)], absWorkingDir: repositoryRoot, + bundle: true, write: false, metafile: true, platform: "node", format: "esm", logLevel: "silent", + plugins: [externalBare], + }); + if (!reached.size) return null; + const parents = new Map([[entry, ""]]); + const queue = [entry]; + while (queue.length) { + const file = queue.shift()!; + for (const imported of result.metafile.inputs[file]?.imports ?? []) { + if (imported.external || parents.has(imported.path)) continue; + parents.set(imported.path, file); + queue.push(imported.path); + } + } + const importer = path.relative(repositoryRoot, [...reached.values()][0]!); + const chain = [target, importer]; + for (let file = parents.get(importer); file; file = parents.get(file)) chain.push(file); + return chain.reverse().join(" -> "); +} + +describe("runner E2E source-server supervisor", () => { + it.each(BUILT_ONLY_PACKAGES)("loads without the generated %s output", async name => { + expect(await staticImportPath("tests/runner-e2e/server.ts", name)).toBeNull(); + }, 60_000); +}); diff --git a/tests/runner-e2e/workflow-security.test.ts b/tests/runner-e2e/workflow-security.test.ts index 49c28b6b7f6..ecf320784b6 100644 --- a/tests/runner-e2e/workflow-security.test.ts +++ b/tests/runner-e2e/workflow-security.test.ts @@ -704,6 +704,11 @@ describe("public repository paid workflow security", () => { expect(buildJob).toContain( "node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs", ); + // The source server imports the plugin SDK from generated output. Build, + // check and ship it once so a missing dist fails here, not in every cell. + expect(buildJob).toContain("node scripts/ensure-plugin-build-deps.mjs"); + expect(buildJob).toMatch(/archive_paths=\(\n(?:\s+\S+\n)*?\s+packages\/plugins\/sdk\/dist\n/u); + expect(testJob).toContain("Campaign outputs lack $entry"); expect(buildJob).toContain("runner-e2e-build-bundle.tar.gz.sha256"); expect(buildJob).toContain("runner-e2e-provider-pack.tar.gz.sha256"); expect(buildJob).toContain(