mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 13:01:05 +01:00
fix(runner-e2e): keep the supervisor off the heartbeat graph and ship the plugin SDK build (#21860)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The paid Runner Full-Stack E2E workflow proves chat features against a real source server > - Since masterf7a3497e9d, every paid cell fails before any test runs: the source server cannot find `@paperclipai/plugin-sdk/dist/index.js` > - The supervisor builds the plugin SDK before it starts Paperclip, but its own static imports now load the SDK first > - This pull request removes the heavy import edge, adds a regression test for that class, and builds and checks the SDK once per campaign > - The benefit is that paid campaigns run again, and a missing workspace build fails once in the build job instead of in 199 paid cells ## Linked Issues or Issue Description Refs #21791 **What happened?** Chat-eval cohort 17 (masterf7a3497e9d) failed in all 199 cells. Each cell exited in the "Run paid cell" step. The private Playwright log shows `ERR_MODULE_NOT_FOUND` for `server/node_modules/@paperclipai/plugin-sdk/dist/index.js`, imported from `server/src/services/reusable-sandbox-resume-error.ts`, and then `Process from config.webServer was not able to start`. Cohort 16 (master819f61f1fd) passed 169/199 with the same workflow steps. **Root cause** #21791 (df9914dbd9) added `import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js"` to `native-runner-checkpoint.ts`. `tests/runner-e2e/server.ts` statically imports `controller-recovery-observer.ts`, which imports the checkpoint reader. The new edge connects it to the full heartbeat graph: `native-runner-checkpoint -> native-controller-recovery -> native-finalization-notifications -> external-agents -> muse -> heartbeat -> heartbeat/run-completion -> reusable-sandbox-resume-error -> @paperclipai/plugin-sdk`. The plugin SDK exports only its generated `dist`. The supervisor builds that `dist` in `prepareSourceServerDependencies`, but ES static imports load before that call. The workflow installs with `--ignore-scripts` and never built the SDK, so the supervisor crashes at load. #21554 is not involved: it changes only release workflows. **Expected behavior** The supervisor loads without generated SDK output. The campaign build job builds and checks the SDK once, and each cell restores it. **Steps to reproduce** 1. Remove `packages/plugins/sdk/dist`. 2. Run `node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/server.ts`. 3. Before this change: `ERR_MODULE_NOT_FOUND` for the SDK. After this change: the normal `PAPERCLIP_RUNNER_E2E_SERVER_LOG is required` error, which shows that all static imports loaded. ## What Changed - Moved `NATIVE_RETAINED_OWNER_WAIT` and `NATIVE_AGENT_DIRECTORY_BUSY_WAIT` to a new leaf module `native-recovery-wait-codes.ts`. `native-controller-recovery.ts` re-exports them, so other importers do not change. `native-runner-checkpoint.ts` imports the leaf. This also removes a checkpoint/controller-recovery import cycle. - Added `tests/runner-e2e/server-startup-import-graph.test.ts`. It uses esbuild to walk the static import graph of `tests/runner-e2e/server.ts` and fails with the full chain if it reaches `@paperclipai/plugin-sdk`. - `runner-full-stack-e2e.yml`: the "Build reusable runner campaign artifacts" job runs `scripts/ensure-plugin-build-deps.mjs` (the same certified helper the cell uses), checks that `shared` and `plugin-sdk` dist and their completion markers exist with a clear `::error::`, and ships both dist trees in the campaign bundle. The cell restore step checks both entry points with a clear `::error::`. The helper's fingerprint receipt lets the cell reuse the shipped output instead of a rebuild. - `workflow-security.test.ts` pins the build, packaging and guard. ## Verification - Red/green: `npx vitest run --config tests/runner-e2e/vitest.config.ts server-startup-import-graph` fails with the old import and passes with the fix. - Local reproduction above: fails before, loads after. - `npx vitest run --config tests/runner-e2e/vitest.config.ts workflow-security server-startup-import-graph native-completion`: 137 passed. - `node --test tests/runner-e2e/native-completion-git-source.test.mjs`: 36 passed. - Server `tsc --noEmit`: clean. - Paid proof of the source fix: one cell (`chat-features.runner-opencode.local.access.private-resource`), dispatched with `--ref master -f target_branch=fix/runner-e2e-plugin-sdk-dist`: https://github.com/paperclipai/paperclip/actions/runs/43804619166 . That run uses the master workflow and this branch's source, so it proves the import-graph fix alone restores the cells. - The workflow changes (SDK build, bundle and guards) cannot run in a paid campaign before merge, because paid runs use the default-branch workflow. `workflow-security.test.ts` pins them. A 1-case paid check from master follows the merge. - The failed "Authorize paid campaign" check on this commit is from a dispatch with a branch `--ref`. The default-branch gate rejected it as designed. It is not a PR-event check. ## Risks - Low risk. The constants keep their values and are re-exported from the old module. - The campaign bundle grows by the `shared` and `plugin-sdk` dist trees (small). If the fingerprints do not match in a cell, the helper rebuilds, as before. ## Model Used - Claude, `claude-opus-5-5`, via Claude Code with tool use (shell, file edit, GitHub CLI). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
9de343b3fe
commit
77ddf260bf
6 files changed
+83
-3
No files matched your search
@@ -797,6 +797,16 @@ jobs:
|
||||
if [ "$NEEDS_NATIVE_BINARIES" = true ]; then
|
||||
pnpm --filter @paperclipai/paperclip-runner build:runner-binaries
|
||||
fi
|
||||
# The source server imports the plugin SDK from its generated dist.
|
||||
# Build it once here, with the same certified helper the cell uses,
|
||||
# instead of in every paid cell.
|
||||
node scripts/ensure-plugin-build-deps.mjs
|
||||
for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do
|
||||
if [ ! -f "$entry" ] || [ ! -f "$(dirname "$entry")/.paperclip-build-complete" ]; then
|
||||
echo "::error::Workspace build output $entry is missing; paid cells would fail before any test runs."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
- name: Package immutable campaign outputs
|
||||
env:
|
||||
@@ -814,6 +824,8 @@ jobs:
|
||||
)
|
||||
archive_paths=(
|
||||
packages/paperclip-eval-kernel/dist
|
||||
packages/shared/dist
|
||||
packages/plugins/sdk/dist
|
||||
)
|
||||
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
|
||||
test -d packages/paperclip-runner/dist
|
||||
@@ -1235,6 +1247,12 @@ jobs:
|
||||
--file runner-e2e-build/runner-e2e-build-bundle.tar.gz \
|
||||
--directory "$GITHUB_WORKSPACE"
|
||||
test -d packages/paperclip-eval-kernel/dist
|
||||
for entry in packages/shared/dist/index.js packages/plugins/sdk/dist/index.js; do
|
||||
if [ ! -f "$entry" ]; then
|
||||
echo "::error::Campaign outputs lack $entry; the source server cannot start without it."
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
if [ "$NEEDS_RUNNER_TYPESCRIPT" = true ]; then
|
||||
test -d packages/paperclip-runner/dist
|
||||
fi
|
||||
|
||||
@@ -11,6 +11,9 @@ import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBoo
|
||||
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
||||
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
|
||||
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
|
||||
import { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
|
||||
|
||||
export { NATIVE_AGENT_DIRECTORY_BUSY_WAIT, NATIVE_RETAINED_OWNER_WAIT };
|
||||
|
||||
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
|
||||
|
||||
@@ -68,7 +71,6 @@ export type NativeControllerRecoveryOutcome =
|
||||
| { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown }
|
||||
| { kind: "no_prior_execution" };
|
||||
|
||||
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
|
||||
|
||||
export type NativeRetainedOwnerWait = { runId: string; controllerBootId: string };
|
||||
|
||||
@@ -138,7 +140,6 @@ export async function deferNativeControllerRecovery(db: Db, input: {
|
||||
});
|
||||
}
|
||||
|
||||
export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy";
|
||||
|
||||
export async function deferNativeRunForBusyAgentDirectory(db: Db, input: {
|
||||
companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number;
|
||||
|
||||
@@ -0,0 +1,5 @@
|
||||
// Dependency-free failure codes shared by native recovery modules. Keep this a
|
||||
// leaf: checkpoint readers import it, and those readers are also loaded by
|
||||
// standalone tools that must not pull in the heartbeat service graph.
|
||||
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
|
||||
export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy";
|
||||
@@ -9,7 +9,7 @@ import { createLocalEncryptedCipher, deriveLocalEncryptedInstanceSecret } from "
|
||||
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
||||
import { assertNativeSourceCheckpointConsumer, lockSourceRecoveryCoordinators } from "./native-source-recovery-ownership.js";
|
||||
import { runnerdStateHasNoProviderTurn, runnerdStateIsIncompleteBootstrap } from "./native-bootstrap-proof.js";
|
||||
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-controller-recovery.js";
|
||||
import { NATIVE_RETAINED_OWNER_WAIT } from "./native-recovery-wait-codes.js";
|
||||
import { OWN_PROCESS_START_EVENT_TYPES } from "../native-process-start-events.js";
|
||||
|
||||
type Binding = { companyId: string; issueId: string; runId: string };
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
import path from "node:path";
|
||||
import { build, type Plugin } from "esbuild";
|
||||
import { describe, expect, it } from "vitest";
|
||||
|
||||
const repositoryRoot = path.resolve(import.meta.dirname, "../..");
|
||||
|
||||
// The source-server supervisor builds the plugin SDK (scripts/ensure-plugin-build-deps.mjs)
|
||||
// before it starts Paperclip. Its own static imports load before that build, so they must
|
||||
// never reach a workspace package whose entry point is generated output. When they did,
|
||||
// every paid source cell failed with ERR_MODULE_NOT_FOUND before any test ran.
|
||||
const BUILT_ONLY_PACKAGES = ["@paperclipai/plugin-sdk"];
|
||||
|
||||
async function staticImportPath(entry: string, target: string) {
|
||||
const reached = new Map<string, string>();
|
||||
const externalBare: Plugin = {
|
||||
name: "external-bare",
|
||||
setup(context) {
|
||||
context.onResolve({ filter: /^[^./]/ }, args => {
|
||||
if (args.path.startsWith("@paperclipai/") && !BUILT_ONLY_PACKAGES.some(name => args.path === name || args.path.startsWith(`${name}/`))) return undefined;
|
||||
if (args.path === target || args.path.startsWith(`${target}/`)) reached.set(args.path, args.importer);
|
||||
return { path: args.path, external: true };
|
||||
});
|
||||
},
|
||||
};
|
||||
const result = await build({
|
||||
entryPoints: [path.join(repositoryRoot, entry)], absWorkingDir: repositoryRoot,
|
||||
bundle: true, write: false, metafile: true, platform: "node", format: "esm", logLevel: "silent",
|
||||
plugins: [externalBare],
|
||||
});
|
||||
if (!reached.size) return null;
|
||||
const parents = new Map<string, string>([[entry, ""]]);
|
||||
const queue = [entry];
|
||||
while (queue.length) {
|
||||
const file = queue.shift()!;
|
||||
for (const imported of result.metafile.inputs[file]?.imports ?? []) {
|
||||
if (imported.external || parents.has(imported.path)) continue;
|
||||
parents.set(imported.path, file);
|
||||
queue.push(imported.path);
|
||||
}
|
||||
}
|
||||
const importer = path.relative(repositoryRoot, [...reached.values()][0]!);
|
||||
const chain = [target, importer];
|
||||
for (let file = parents.get(importer); file; file = parents.get(file)) chain.push(file);
|
||||
return chain.reverse().join(" -> ");
|
||||
}
|
||||
|
||||
describe("runner E2E source-server supervisor", () => {
|
||||
it.each(BUILT_ONLY_PACKAGES)("loads without the generated %s output", async name => {
|
||||
expect(await staticImportPath("tests/runner-e2e/server.ts", name)).toBeNull();
|
||||
}, 60_000);
|
||||
});
|
||||
@@ -704,6 +704,11 @@ describe("public repository paid workflow security", () => {
|
||||
expect(buildJob).toContain(
|
||||
"node packages/paperclip-runner/scripts/materialize-opencode-binary.mjs",
|
||||
);
|
||||
// The source server imports the plugin SDK from generated output. Build,
|
||||
// check and ship it once so a missing dist fails here, not in every cell.
|
||||
expect(buildJob).toContain("node scripts/ensure-plugin-build-deps.mjs");
|
||||
expect(buildJob).toMatch(/archive_paths=\(\n(?:\s+\S+\n)*?\s+packages\/plugins\/sdk\/dist\n/u);
|
||||
expect(testJob).toContain("Campaign outputs lack $entry");
|
||||
expect(buildJob).toContain("runner-e2e-build-bundle.tar.gz.sha256");
|
||||
expect(buildJob).toContain("runner-e2e-provider-pack.tar.gz.sha256");
|
||||
expect(buildJob).toContain(
|
||||
|
||||
Reference in new issue
Block a user