fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The paid Runner Full-Stack E2E workflow proves chat features against
a real source server
> - Since master f7a3497e9d, every paid cell fails before any test runs:
the source server cannot find `@paperclipai/plugin-sdk/dist/index.js`
> - The supervisor builds the plugin SDK before it starts Paperclip, but
its own static imports now load the SDK first
> - This pull request removes the heavy import edge, adds a regression
test for that class, and builds and checks the SDK once per campaign
> - The benefit is that paid campaigns run again, and a missing
workspace build fails once in the build job instead of in 199 paid cells

## Linked Issues or Issue Description

Refs #21791

**What happened?**
Chat-eval cohort 17 (master f7a3497e9d) failed in all 199 cells. Each
cell exited in the "Run paid cell" step. The private Playwright log
shows `ERR_MODULE_NOT_FOUND` for
`server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported
from `server/src/services/reusable-sandbox-resume-error.ts`, and then
`Process from config.webServer was not able to start`. Cohort 16 (master
819f61f1fd) passed 169/199 with the same workflow steps.

**Root cause**
#21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from
"./native-controller-recovery.js"` to `native-runner-checkpoint.ts`.
`tests/runner-e2e/server.ts` statically imports
`controller-recovery-observer.ts`, which imports the checkpoint reader.
The new edge connects it to the full heartbeat graph:
`native-runner-checkpoint -> native-controller-recovery ->
native-finalization-notifications -> external-agents -> muse ->
heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error
-> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated
`dist`. The supervisor builds that `dist` in
`prepareSourceServerDependencies`, but ES static imports load before
that call. The workflow installs with `--ignore-scripts` and never built
the SDK, so the supervisor crashes at load. #21554 is not involved: it
changes only release workflows.

**Expected behavior**
The supervisor loads without generated SDK output. The campaign build
job builds and checks the SDK once, and each cell restores it.

**Steps to reproduce**
1. Remove `packages/plugins/sdk/dist`.
2. Run `node cli/node_modules/tsx/dist/cli.mjs
tests/runner-e2e/server.ts`.
3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this
change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error,
which shows that all static imports loaded.

## What Changed

- Moved `NATIVE_RETAINED_OWNER_WAIT` and
`NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module
`native-recovery-wait-codes.ts`. `native-controller-recovery.ts`
re-exports them, so other importers do not change.
`native-runner-checkpoint.ts` imports the leaf. This also removes a
checkpoint/controller-recovery import cycle.
- Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses
esbuild to walk the static import graph of `tests/runner-e2e/server.ts`
and fails with the full chain if it reaches `@paperclipai/plugin-sdk`.
- `runner-full-stack-e2e.yml`: the "Build reusable runner campaign
artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same
certified helper the cell uses), checks that `shared` and `plugin-sdk`
dist and their completion markers exist with a clear `::error::`, and
ships both dist trees in the campaign bundle. The cell restore step
checks both entry points with a clear `::error::`. The helper's
fingerprint receipt lets the cell reuse the shipped output instead of a
rebuild.
- `workflow-security.test.ts` pins the build, packaging and guard.

## Verification

- Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts
server-startup-import-graph` fails with the old import and passes with
the fix.
- Local reproduction above: fails before, loads after.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
workflow-security server-startup-import-graph native-completion`: 137
passed.
- `node --test tests/runner-e2e/native-completion-git-source.test.mjs`:
36 passed.
- Server `tsc --noEmit`: clean.
- Paid proof of the source fix: one cell
(`chat-features.runner-opencode.local.access.private-resource`),
dispatched with `--ref master -f
target_branch=fix/runner-e2e-plugin-sdk-dist`:
https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That
run uses the master workflow and this branch's source, so it proves the
import-graph fix alone restores the cells.
- The workflow changes (SDK build, bundle and guards) cannot run in a
paid campaign before merge, because paid runs use the default-branch
workflow. `workflow-security.test.ts` pins them. A 1-case paid check
from master follows the merge.
- The failed "Authorize paid campaign" check on this commit is from a
dispatch with a branch `--ref`. The default-branch gate rejected it as
designed. It is not a PR-event check.

## Risks

- Low risk. The constants keep their values and are re-exported from the
old module.
- The campaign bundle grows by the `shared` and `plugin-sdk` dist trees
(small). If the fingerprints do not match in a cell, the helper
rebuilds, as before.

## Model Used

- Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file
edit, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-16 10:08:11 -06:00
1 parent 9de343b3fe
commit 77ddf260bf
6 files changed
+83 -3

No files matched your search

@@ -797,6 +797,16 @@ jobs:
if [ "$NEEDS_NATIVE_BINARIES" = true ]; then
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
fi
# The source server imports the plugin SDK from its generated dist.
# Build it once here, with the same certified helper the cell uses,
# instead of in every paid cell.
node scripts/ensure-plugin-build-deps.mjs
for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do
if [ ! -f "$entry" ] || [ ! -f "$(dirname "$entry")/.paperclip-build-complete" ]; then
echo "::error::Workspace build output $entry is missing; paid cells would fail before any test runs."
exit 1
fi
done
- name: Package immutable campaign outputs
env:
@@ -814,6 +824,8 @@ jobs:
)
archive_paths=(
packages/paperclip-eval-kernel/dist
packages/shared/dist
packages/plugins/sdk/dist
)
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
test -d packages/paperclip-runner/dist
@@ -1235,6 +1247,12 @@ jobs:
--file runner-e2e-build/runner-e2e-build-bundle.tar.gz \
--directory "$GITHUB_WORKSPACE"
test -d packages/paperclip-eval-kernel/dist
for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do
if [ ! -f "$entry" ]; then
echo "::error::Campaign outputs lack $entry; the source server cannot start without it."
exit 1
fi
done
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
test -d packages/paperclip-runner/dist
fi
@@ -11,6 +11,9 @@ import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBoo
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
export { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT };
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
@@ -68,7 +71,6 @@ export type NativeControllerRecoveryOutcome =
| { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown }
| { kind: "no_prior_execution" };
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string };
@@ -138,7 +140,6 @@ export async function deferNativeControllerRecovery(db: Db, input: {
});
}
export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy";
export async function deferNativeRunForBusyAgentDirectory(db: Db, input: {
companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number;
@@ -0,0 +1,5 @@
// Dependency-free failure codes shared by native recovery modules. Keep this a
// leaf: checkpoint readers import it, and those readers are also loaded by
// standalone tools that must not pull in the heartbeat service graph.
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy";
@@ -9,7 +9,7 @@ import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "
import { legacyControllerBootId } from "../legacy-controller-lease.js";
import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
type Binding = { companyId: string; issueId: string; runId: string };
@@ -0,0 +1,51 @@
import path from "node:path";
import { build, type Plugin } from "esbuild";
import { describe, expect, it } from "vitest";
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
// The source-server supervisor builds the plugin SDK (scripts/ensure-plugin-build-deps.mjs)
// before it starts Paperclip. Its own static imports load before that build, so they must
// never reach a workspace package whose entry point is generated output. When they did,
// every paid source cell failed with ERR_MODULE_NOT_FOUND before any test ran.
const BUILT_ONLY_PACKAGES = ["@paperclipai/plugin-sdk"];
async function staticImportPath(entry: string, target: string) {
const reached = new Map<string, string>();
const externalBare: Plugin = {
name: "external-bare",
setup(context) {
context.onResolve({ filter: /^[^./]/ }, args => {
if (args.path.startsWith("@paperclipai/") && !BUILT_ONLY_PACKAGES.some(name => args.path === name || args.path.startsWith(`${name}/`))) return undefined;
if (args.path === target || args.path.startsWith(`${target}/`)) reached.set(args.path, args.importer);
return { path: args.path, external: true };
});
},
};
const result = await build({
entryPoints: [path.join(repositoryRoot, entry)], absWorkingDir: repositoryRoot,
bundle: true, write: false, metafile: true, platform: "node", format: "esm", logLevel: "silent",
plugins: [externalBare],
});
if (!reached.size) return null;
const parents = new Map<string, string>([[entry, ""]]);
const queue = [entry];
while (queue.length) {
const file = queue.shift()!;
for (const imported of result.metafile.inputs[file]?.imports ?? []) {
if (imported.external || parents.has(imported.path)) continue;
parents.set(imported.path, file);
queue.push(imported.path);
}
}
const importer = path.relative(repositoryRoot, [...reached.values()][0]!);
const chain = [target, importer];
for (let file = parents.get(importer); file; file = parents.get(file)) chain.push(file);
return chain.reverse().join(" -> ");
}
describe("runner E2E source-server supervisor", () => {
it.each(BUILT_ONLY_PACKAGES)("loads without the generated %s output", async name => {
expect(await staticImportPath("tests/runner-e2e/server.ts", name)).toBeNull();
}, 60_000);
});
@@ -704,6 +704,11 @@ describe("public repository paid workflow security", () => {
expect(buildJob).toContain(
"node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs",
);
// The source server imports the plugin SDK from generated output. Build,
// check and ship it once so a missing dist fails here, not in every cell.
expect(buildJob).toContain("node scripts/ensure-plugin-build-deps.mjs");
expect(buildJob).toMatch(/archive_paths=\(\n(?:\s+\S+\n)*?\s+packages\/plugins\/sdk\/dist\n/u);
expect(testJob).toContain("Campaign outputs lack $entry");
expect(buildJob).toContain("runner-e2e-build-bundle.tar.gz.sha256");
expect(buildJob).toContain("runner-e2e-provider-pack.tar.gz.sha256");
expect(buildJob).toContain(