From 250429942e27d568f20d3bc7dddcfc84a666480a Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 06:46:29 -0500 Subject: [PATCH 1/4] fix(agents): report actionable lifecycle failure steps (#16418) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip helps people manage AI agents for work. > - New agents must pass setup and execution checks before they can run. > - The lifecycle worker saved the same retry message for every failure. > - Users could not identify the failed step or correct its configuration. > - The UI also described failed work as work still in progress. > - This pull request names the failed step and gives a safe corrective action. > - Users can identify configuration problems and understand automatic retries. ## Linked Issues or Issue Description **What happened?** A new hire showed a paused status and “The lifecycle step failed. Retry the operation.” Repeated retries did not explain the failure. A successful retry cleared the original error. This change cannot recover an error that an older server already cleared. **Expected behavior** Name the failed setup step. Give a useful action for known execution test failures. Explain automatic retry. Keep provider output and credentials out of lifecycle errors. **Steps to reproduce** 1. Create a disposable agent with an unavailable model or invalid execution command. 2. Open its profile while verification fails. 3. Inspect the error, correct the configuration, and select Retry now. 4. Confirm that the failure disappears after the agent becomes ready. Related change: https://github.com/paperclipai/paperclip/pull/16155 shows pending setup progress in creation and assignment. This change improves the cause and action shown when setup fails. ## What Changed - Save the failed lifecycle phase and step in the existing lifecycle error field. - Classify authentication, model, command, directory, timeout, and response failures from execution test check codes. - Use fixed error messages. Do not retain raw provider messages, check details, or credentials. - Share the allowed failure codes with the lifecycle log. - Show failed-state labels and automatic retry guidance. Rename Retry to Retry now. - Extend worker, database, and UI regression coverage. Update the agent verification maps. ## Verification - Passed 105 focused tests across lifecycle diagnostics, database lifecycle commands, saved harness verification, and AgentLifecycleStatus. - Passed server TypeScript compilation with `pnpm --filter @paperclipai/server exec tsc --noEmit`. - Passed UI typecheck and production build. - Passed module boundary checks, token gates, character sync, map validation, and `git diff --check`. - Used Playwright to inspect the production component with a simulated model failure. Confirmed the failure text, retry guidance, retry callback, and recovered state. Captured a screenshot. This component check does not prove a live provider run. - Attempted `pnpm -r typecheck` and `pnpm build`. Both require Rust cargo, which is not installed in the local execution environment. - Started `pnpm test:run` locally. No final local suite result was recorded before handoff. The complete PR CI workflow passed. - Passed all latest-commit CI gates, including typecheck, build, general and serialized tests, Rust checks, and all eight browser test shards: https://github.com/paperclipai/paperclip/actions/runs/38726198774 - Greptile scored the latest commit 5/5 with no findings. There are no review threads to resolve. ## Risks - Failure classification depends on existing adapter check codes. Unknown codes use a safe fallback message. - A known code can describe only the checks reported by the adapter. - No database migration is required. The error field remains a string. - Retry schedules, leases, readiness, approvals, and budget controls keep their current behavior. - Full build verification requires a runner with Rust cargo. ## Model Used - OpenAI GPT-6 through Codex. Used reasoning, code editing, browser checks, and tool execution. The runtime does not expose the exact deployment ID or context window. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- maps/agent-access-map/agents.md | 2 + maps/feature-map/agents.md | 6 +++ .../agent-harness-verification.test.ts | 16 ++++++++ .../__tests__/diagnostics.test.ts | 26 ++++++++++++ .../__tests__/lifecycle.test.ts | 2 +- .../agent-lifecycle/application/failure.ts | 40 +++++++++++++++++++ .../agent-lifecycle/application/worker.ts | 15 ++++--- server/src/modules/agent-lifecycle/index.ts | 2 + .../services/agent-harness-verification.ts | 12 +++++- server/src/services/agent-lifecycle.ts | 6 +-- .../components/AgentLifecycleStatus.test.tsx | 4 +- ui/src/components/AgentLifecycleStatus.tsx | 13 ++++-- 12 files changed, 128 insertions(+), 16 deletions(-) create mode 100644 server/src/modules/agent-lifecycle/application/failure.ts diff --git a/maps/agent-access-map/agents.md b/maps/agent-access-map/agents.md index 5988d4eace8..26e162cf032 100644 --- a/maps/agent-access-map/agents.md +++ b/maps/agent-access-map/agents.md @@ -132,6 +132,8 @@ API: `present` — representative operation IDs: - `POST /api/agents/{id}/pause`. - `POST /api/agents/{id}/resume`. - `POST /api/agents/{id}/terminate`. +- `POST /api/agents/{id}/lifecycle/retry`. +- `GET /api/agents/{id}` returns the safe failed phase/step and action in `lifecycleError`. Agent access: `partial` — REST / native `call_api` resume under agent_config:update change-grant authorization; pause and termination remain board-only. diff --git a/maps/feature-map/agents.md b/maps/feature-map/agents.md index 621e271ed13..44433068977 100644 --- a/maps/feature-map/agents.md +++ b/maps/feature-map/agents.md @@ -80,6 +80,12 @@ Automated: [instruction service](../../server/src/__tests__/agent-instructions-s Manual: Save a harmless instruction edit, reload, and verify its revision and effect in the next eligible run. Pause and resume the disposable agent; confirm pause prevents new work and does not erase previous runs. +For failed setup, use an unavailable model or invalid command on a disposable agent. +Confirm the profile names the failed verification step and gives a relevant action. +Check that it explains automatic retry and offers **Retry now**. Fix the configuration, +retry, and reload; the failure must disappear only after readiness commits. +Provider output and credentials must not appear in the lifecycle message. + An explicit runner change is a configuration change, with normal revisions and session invalidation. Incompatible settings must produce a field-specific error; selecting another runner must not silently discard them or change an active run's diff --git a/server/src/__tests__/agent-harness-verification.test.ts b/server/src/__tests__/agent-harness-verification.test.ts index 4728cdabd18..3d43bdd0a80 100644 --- a/server/src/__tests__/agent-harness-verification.test.ts +++ b/server/src/__tests__/agent-harness-verification.test.ts @@ -8,6 +8,22 @@ describe("saved harness verification", () => { adapterType: "test", status, testedAt: new Date().toISOString(), checks: codes.map(code => ({ code, level: "info", message: code })), }); + it.each([ + ["opencode_hello_probe_auth_required", "harness_auth_failed"], + ["opencode_hello_probe_model_unavailable", "harness_model_unavailable"], + ["opencode_command_unresolvable", "harness_command_unavailable"], + ["opencode_cwd_invalid", "harness_cwd_invalid"], + ["opencode_hello_probe_timed_out", "harness_probe_timed_out"], + ["opencode_hello_probe_unexpected_output", "harness_probe_unexpected_output"], + ["private provider error", "harness_test_failed"], + ])("classifies %s without keeping provider output", (checkCode, code) => { + try { + assertHarnessTestPassed(result("fail", [checkCode])); + expect.fail("Expected verification to fail"); + } catch (error) { + expect(error).toMatchObject({ code, message: "The harness test failed" }); + } + }); it.each([ ["fail", []], ["warn", ["claude_hello_probe_auth_required"]], ["warn", ["gemini_hello_probe_timed_out"]], ["pass", ["claude_hello_probe_skipped_custom_command"]], diff --git a/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts b/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts index 06b7e246d49..39c7895c30b 100644 --- a/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/diagnostics.test.ts @@ -5,6 +5,7 @@ import type { LifecycleAgent, LifecycleStore, LifecycleDriver } from "../applica import { startAgentLifecycle } from "../index.js"; import { createAgentLifecycleEffects } from "../../../services/agent-lifecycle.js"; import { logger } from "../../../middleware/logger.js"; +import { lifecycleFailureMessage } from "../application/failure.js"; const agent: LifecycleAgent = { id: "agent", companyId: "company", lifecycleState: "verifying", lifecycleVersion: 3, lifecycleRequiredPluginIds: null, lifecycleError: null, @@ -66,6 +67,30 @@ describe("lifecycle readiness handoff", () => { }); describe("lifecycle failure diagnostics", () => { + it.each([ + ["host", "harness_model_unavailable", "Execution configuration check failed. The selected model is unavailable."], + ["plugin", "required_plugin_unavailable", "Execution configuration check: required plugin step failed. A required plugin is unavailable."], + ["select_plugins", "unknown private code", "Execution configuration check: finding required plugins failed. Paperclip could not complete this step."], + ["transition", "40P01", "Execution configuration check: saving progress failed. Paperclip could not complete this step."], + ] as const)("persists safe actionable errors for %s", async (stage, code, expected) => { + const { store, driver, worker } = fixture(); + const error = Object.assign(new Error("secret provider output"), { code }); + if (stage === "host") driver.runHost.mockRejectedValue(error); + if (stage === "plugin") driver.runPlugin.mockRejectedValue(error); + if (stage === "select_plugins") driver.requiredPluginIds.mockRejectedValue(error); + if (stage === "transition") store.change.mockRejectedValue(error); + await worker.process(agent.id); + expect(store.defer).toHaveBeenCalledWith(agent, expect.any(String), expect.stringContaining(expected), expect.any(Date)); + expect(JSON.stringify(store.defer.mock.calls)).not.toContain("secret provider output"); + expect(JSON.stringify(store.defer.mock.calls)).not.toContain("unknown private code"); + await worker.stop(); + }); + + it("does not treat prototype properties as allowed failure codes", () => { + expect(lifecycleFailureMessage({ stage: "host", phase: "verifying" }, { code: "toString" })) + .toContain("Paperclip could not complete this step"); + }); + it.each(["host", "plugin", "claim", "renew", "scan", "defer"] as const)("reports %s failures with the available context", async stage => { const { store, driver, report, worker } = fixture(); const error = new Error("private provider detail"); @@ -100,6 +125,7 @@ describe("lifecycle failure diagnostics", () => { it.each([ [{ code: "required_plugin_unavailable" }, "required_plugin_unavailable"], [{ code: "harness_test_failed" }, "harness_test_failed"], + [{ code: "harness_auth_failed" }, "harness_auth_failed"], [{ cause: { code: "40P01", message: "private query" } }, "40P01"], [{ code: "private credential value" }, "unclassified"], ])("logs known codes without provider error contents", (details, failureCode) => { diff --git a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts index fc84aa79212..725a4adf1c3 100644 --- a/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts +++ b/server/src/modules/agent-lifecycle/__tests__/lifecycle.test.ts @@ -140,7 +140,7 @@ const support = await getEmbeddedPostgresTestSupport(); const work = worker(createLifecycleDriver(db, {} as never).runHost); await work.process(agent.id); expect(probe).toHaveBeenCalled(); - expect(await current(agent.id)).toMatchObject({ lifecycleState: "verifying", lifecycleError: "The lifecycle step failed. Retry the operation." }); + expect(await current(agent.id)).toMatchObject({ lifecycleState: "verifying", lifecycleError: expect.stringContaining("Execution configuration check failed. Authentication failed.") }); } finally { probe.mockRestore(); } }); diff --git a/server/src/modules/agent-lifecycle/application/failure.ts b/server/src/modules/agent-lifecycle/application/failure.ts new file mode 100644 index 00000000000..0c5da80e34b --- /dev/null +++ b/server/src/modules/agent-lifecycle/application/failure.ts @@ -0,0 +1,40 @@ +import type { LifecycleFailure } from "./ports.js"; + +// Only fixed text may leave this boundary. Provider errors can contain secrets. +const reasons = { + harness_test_failed: "The execution test did not pass. Open Configuration and run the agent test to inspect the failed checks.", + harness_auth_failed: "Authentication failed. Check the agent's AI connection or sign in to its execution tool, then retry.", + harness_model_unavailable: "The selected model is unavailable. Choose a model supported by the agent's AI connection in Configuration, then retry.", + harness_command_unavailable: "The execution command was not found. Check the command and its installation in the agent's environment, then retry.", + harness_cwd_invalid: "The working directory is unavailable. Check the working directory in Configuration, then retry.", + harness_probe_timed_out: "The execution test timed out. Paperclip will retry automatically. If this continues, run the agent test in Configuration.", + harness_probe_unexpected_output: "The execution test did not return the expected response. Run the agent test in Configuration to inspect the failed checks.", + test_environment_unavailable: "The test environment is unavailable. Check the agent's environment in Configuration. Paperclip will retry automatically.", + required_plugin_unavailable: "A required plugin is unavailable. Check the plugin's status in Settings. Paperclip will retry automatically.", + invalid_lifecycle_result: "A required plugin returned an invalid result. Check the plugin's status in Settings. Paperclip will retry automatically.", +} as const; + +const transientCodes = new Set(["ECONNREFUSED", "ECONNRESET", "ETIMEDOUT", "40001", "40P01", "53300", "57P01"]); + +export function lifecycleFailureCode(error: unknown): string | undefined { + const candidate = error as { code?: unknown; cause?: { code?: unknown } } | null; + return [candidate?.code, candidate?.cause?.code].find((code): code is string => + typeof code === "string" && (Object.hasOwn(reasons, code) || transientCodes.has(code))); +} + +export function lifecycleFailureMessage(context: LifecycleFailure, error: unknown): string { + const phases = { + preparing: "Agent preparation", verifying: "Execution configuration check", pausing: "Stopping agent execution", + resuming: "Starting agent resources", terminating: "Stopping agent execution", cleaning_up: "Agent resource cleanup", + }; + const phase = context.phase && Object.hasOwn(phases, context.phase) + ? phases[context.phase as keyof typeof phases] : "Agent setup"; + const step = context.stage === "plugin" ? `${phase}: required plugin step` + : context.stage === "select_plugins" ? `${phase}: finding required plugins` + : ["save_plugins", "complete_host", "complete_plugin", "transition"].includes(context.stage) + ? `${phase}: saving progress` : phase; + const code = lifecycleFailureCode(error); + const reason = code && Object.hasOwn(reasons, code) ? reasons[code as keyof typeof reasons] + : "Paperclip could not complete this step. It will retry automatically. If this continues, contact your instance administrator."; + return `${step} failed. ${reason}`; +} diff --git a/server/src/modules/agent-lifecycle/application/worker.ts b/server/src/modules/agent-lifecycle/application/worker.ts index 3b90b227673..3c362dcbb11 100644 --- a/server/src/modules/agent-lifecycle/application/worker.ts +++ b/server/src/modules/agent-lifecycle/application/worker.ts @@ -1,6 +1,7 @@ import { randomUUID } from "node:crypto"; import type { LifecycleDriver, LifecycleStore, LifecycleFailure } from "./ports.js"; import { isLifecycleWorkPending } from "../domain/policy.js"; +import { lifecycleFailureMessage } from "./failure.js"; /** Database leases protect each attempt across server processes. */ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDriver, reportFailure: (context: LifecycleFailure, error: unknown) => void, canRun = () => true) { @@ -20,10 +21,13 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr if (!agent || !isLifecycleWorkPending(agent.lifecycleState)) return; const context = { agentId: agent.id, companyId: agent.companyId, phase: agent.lifecycleState, operationId: agent.lifecycleOperation?.id, version: agent.lifecycleVersion }; - const runStep = (stage: LifecycleFailure["stage"], work: () => Promise, pluginId?: string) => - attempt({ ...context, stage, ...(pluginId ? { pluginId } : {}) }, work); + let failureContext: LifecycleFailure = { ...context, stage: "host" }; + const runStep = (stage: LifecycleFailure["stage"], work: () => Promise, pluginId?: string) => { + const stepContext = { ...context, stage, ...(pluginId ? { pluginId } : {}) }; + return attempt(stepContext, work).catch(error => { failureContext = stepContext; throw error; }); + }; const renewal = setInterval(() => { - void runStep("renew", () => store.renew(agent, owner, new Date())).catch(() => {}); + void attempt({ ...context, stage: "renew" }, () => store.renew(agent, owner, new Date())).catch(() => {}); }, 30_000); renewal.unref?.(); try { @@ -65,8 +69,9 @@ export function createLifecycleWorker(store: LifecycleStore, driver: LifecycleDr async () => { await driver.onReady?.(changed); }).catch(() => {}); } if (changed && !isLifecycleWorkPending(changed.lifecycleState)) return; - } catch { - if (!await runStep("defer", () => store.defer(agent, owner, "The lifecycle step failed. Retry the operation.", new Date()))) continue; + } catch (error) { + const message = lifecycleFailureMessage(failureContext, error); + if (!await runStep("defer", () => store.defer(agent, owner, message, new Date()))) continue; return; } finally { clearInterval(renewal); diff --git a/server/src/modules/agent-lifecycle/index.ts b/server/src/modules/agent-lifecycle/index.ts index ba8abce8c37..99fc03bb117 100644 --- a/server/src/modules/agent-lifecycle/index.ts +++ b/server/src/modules/agent-lifecycle/index.ts @@ -142,3 +142,5 @@ export function startAgentLifecycle(db: Db, effects: LifecycleEffects, driver: L }, }; } + +export { lifecycleFailureCode } from "./application/failure.js"; diff --git a/server/src/services/agent-harness-verification.ts b/server/src/services/agent-harness-verification.ts index 92e08a92cec..295397a445b 100644 --- a/server/src/services/agent-harness-verification.ts +++ b/server/src/services/agent-harness-verification.ts @@ -19,7 +19,17 @@ export function assertHarnessTestPassed(result: AdapterEnvironmentTestResult) { const authenticationFailed = result.checks.some(check => check.code === ADAPTER_AUTH_MISSING_CHECK_CODE || /_hello_probe_auth_required$/.test(check.code)); const incomplete = !verified && (result.status === "warn" || result.checks.some(check => check.code.includes("hello_probe"))); - if (result.status === "fail" || authenticationFailed || incomplete) throw Object.assign(new Error("The harness test failed"), { code: "harness_test_failed" }); + if (result.status === "fail" || authenticationFailed || incomplete) { + const has = (suffix: string) => result.checks.some(check => check.code.endsWith(suffix)); + const code = authenticationFailed ? "harness_auth_failed" + : has("_hello_probe_model_unavailable") || has("_model_invalid") ? "harness_model_unavailable" + : has("_command_unresolvable") ? "harness_command_unavailable" + : has("_cwd_invalid") ? "harness_cwd_invalid" + : has("_hello_probe_timed_out") ? "harness_probe_timed_out" + : has("_hello_probe_unexpected_output") ? "harness_probe_unexpected_output" + : "harness_test_failed"; + throw Object.assign(new Error("The harness test failed"), { code }); + } } export function agentHarnessVerificationService(db: Db, manager: PluginWorkerManager) { diff --git a/server/src/services/agent-lifecycle.ts b/server/src/services/agent-lifecycle.ts index 25ef5e58738..a69ae1d9120 100644 --- a/server/src/services/agent-lifecycle.ts +++ b/server/src/services/agent-lifecycle.ts @@ -16,11 +16,7 @@ export type { LifecycleDriver, LifecycleAgent } from "../modules/agent-lifecycle export function createAgentLifecycleEffects(hooks: BudgetServiceHooks = {}): lifecycle.LifecycleEffects { return { reportFailure(context, error) { - // Only known codes are safe; provider errors can contain credentials or private data. - const allowedCodes = new Set(["required_plugin_unavailable", "invalid_lifecycle_result", "harness_test_failed", - "test_environment_unavailable", "ECONNREFUSED", "ECONNRESET", "ETIMEDOUT", "40001", "40P01", "53300", "57P01"]); - const candidate = error as { code?: unknown; cause?: { code?: unknown } } | null; - const code = [candidate?.code, candidate?.cause?.code].find(value => typeof value === "string" && allowedCodes.has(value)); + const code = lifecycle.lifecycleFailureCode(error); logger.warn({ ...context, failureCode: code ?? "unclassified" }, "Agent lifecycle operation failed"); }, transaction: withAccountingTransaction, diff --git a/ui/src/components/AgentLifecycleStatus.test.tsx b/ui/src/components/AgentLifecycleStatus.test.tsx index 8e4467f2f46..e2fe7baf497 100644 --- a/ui/src/components/AgentLifecycleStatus.test.tsx +++ b/ui/src/components/AgentLifecycleStatus.test.tsx @@ -34,7 +34,9 @@ it("retries a failed step once while pending and removes the failure after recov try { await act(async () => root.render()); expect(container.textContent).toContain(agent.lifecycleError); - expect(container.textContent).toContain("Termination is not complete"); + expect(container.textContent).toContain("Agent cleanup failed"); + expect(container.textContent).toContain("Paperclip retries automatically"); + expect(container.textContent).not.toContain("Required resources are being removed"); await act(async () => container.querySelector("button")!.click()); expect(retry).toHaveBeenCalledOnce(); await act(async () => root.render()); diff --git a/ui/src/components/AgentLifecycleStatus.tsx b/ui/src/components/AgentLifecycleStatus.tsx index 29a6984da77..689a54b8314 100644 --- a/ui/src/components/AgentLifecycleStatus.tsx +++ b/ui/src/components/AgentLifecycleStatus.tsx @@ -15,6 +15,12 @@ const phases = { cleaning_up: ["Cleaning up agent", "Required resources are being removed. Termination is not complete."], } satisfies Partial>; +const failedPhases = { + preparing: "Agent preparation failed", verifying: "Agent verification failed", + pausing: "Agent pause failed", resuming: "Agent resume failed", + terminating: "Agent termination failed", cleaning_up: "Agent cleanup failed", +}; + function pendingPhase(state?: AgentLifecycleState) { return state && state in phases ? phases[state as keyof typeof phases] : undefined; } @@ -48,15 +54,16 @@ export function AgentLifecycleStatus({ agent, onRetry, retryPending, refreshErro {phase &&
{!agent.lifecycleError && - {phase[1]} + {!agent.lifecycleError && {phase[1]}}
} {phase && agent.lifecycleError &&
{agent.lifecycleError}
} + {phase && agent.lifecycleError &&

Paperclip retries automatically; Retry now checks again immediately.

} ; } From efcc8a7d89729bfc82c8af00b87dfbdc656a30ab Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 06:57:38 -0500 Subject: [PATCH 2/4] test: bound command snapshot forwarding checks (#16421) Keep the command snapshot option test focused on forwarding and real file copyback. Use inline delegates in that test, and retain the separate real-worker suites and original deadlines. Validation: 124 affected tests, full typecheck/build, exact-head CI/E2E/Superagent and Apex 5/5. Negative controls reject lost command and restore forwarding. Failed earlier merge-readiness evidence remains in the PR. Co-Authored-By: Paperclip --- .../adapter-utils/src/sandbox-managed-runtime.test.ts | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts index d9331d310d2..b2135d1b279 100644 --- a/packages/adapter-utils/src/sandbox-managed-runtime.test.ts +++ b/packages/adapter-utils/src/sandbox-managed-runtime.test.ts @@ -427,7 +427,11 @@ describe("sandbox managed runtime", () => { const local = path.join(root, "local"), remote = path.join(root, "remote"); await mkdir(local); await writeFile(path.join(local, "file.txt"), "before"); - const capture = vi.spyOn(restoreMerge, "captureDirectorySnapshotOffThread"); + const mergeDirectoryWithBaseline = restoreMerge.mergeDirectoryWithBaseline; + const capture = vi.spyOn(restoreMerge, "captureDirectorySnapshotOffThread") + .mockImplementation(restoreMerge.captureDirectorySnapshot); + const merge = vi.spyOn(restoreMerge, "mergeDirectoryWithBaseline") + .mockImplementation(input => mergeDirectoryWithBaseline({ ...input, workspaceSnapshotIsolation: undefined })); try { const prepared = await prepareCommandManagedRuntime({ runner: makeInlineSpawnRunner(), @@ -442,8 +446,10 @@ describe("sandbox managed runtime", () => { expect(await readFile(path.join(remote, "file.txt"), "utf8")).toBe("before"); await writeFile(path.join(remote, "file.txt"), "after"); await prepared.restoreWorkspace(); + expect(merge).toHaveBeenCalled(); + expect(merge.mock.calls.every(([input]) => input.workspaceSnapshotIsolation === isolation)).toBe(true); expect(await readFile(path.join(local, "file.txt"), "utf8")).toBe("after"); - } finally { capture.mockRestore(); } + } finally { capture.mockRestore(); merge.mockRestore(); } }); it.each(["host_current", "adopt_remote", "durable_seed"] as const)("stages and restores both project repositories with independent Git histories (%s)", async (mode) => { From 6acec4266e17f0a31242d4ed0900fdd6e3b677f1 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 06:59:50 -0500 Subject: [PATCH 3/4] feat(ci): profile full CODEOWNER PR suites in Honeycomb (#16378) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its PR workflow runs dozens of parallel build and test jobs. > - Individual job durations do not show the elapsed cost of the whole suite. > - Maintainers need one shared timeline to find slow work and repeated setup. > - This pull request adds full-suite profiles for selected CI CODEOWNER PRs. > - Honeycomb shows each job, step, test file, and supported test on that timeline. > - The benefit is evidence for changes that reduce total CI time. ## Linked Issues or Issue Description **What existing behavior does this improve?** The main PR workflow has about 48 jobs. Maintainers need to compare their overlapping work and find which jobs delay completion. **Current behavior** GitHub shows job and step durations separately. Test files and tests are not joined to a full workflow trace. **Proposed behavior** A CI CODEOWNER adds `profile-ci` to a PR. The tested code stays on that PR branch. All main PR lanes run, including lanes normally skipped for stack position or release notes. A trusted post-run exporter sends one wall-clock trace to Honeycomb. **Reason and benefit** The trace supports decisions about which CI work to speed up or remove. Ordinary PRs do not export traces. The roadmap and related-PR search found no duplicate CI profiling work. ## What Changed - Add label selection and a full-suite override. Verify the PR author, original actor, and rerun actors against the trusted `.github/**` CODEOWNERS rule. - Record Vitest, Playwright, and Node test file/test timings in local artifacts. Keep ordinary reporter behavior and failing exit codes. - Add a trusted `workflow_run` exporter. Use actual GitHub timestamps for all jobs, setup, steps, waiting, and teardown. Mark missing test data and partial reruns. - Keep the ingest key out of PR jobs. Store it only in the `ci-profiling` environment, restricted to `master`. The exporter reads bounded JSON and never executes artifact code. - Record expected test invocations outside reporters so a missing invocation cannot appear complete. Preserve `node --test` on older PR checkouts. Flush large shard inventories before exit. - Add security, timing, routing, and real-reporter checks. Document operation and activation. Protect the exporter with CODEOWNERS. ## Verification - 509 focused workflow/reporter/shard checks passed. They include outside authors, outside rerun actors, bots, stale revisions, malicious ZIP paths, incomplete reports, and partial ingestion failures. - Final revision `3bf5d9017d` adds multi-file qualification and fixes short-file step parenting and duplicate Node file-wrapper events. Greptile reviewed this exact revision at 5/5 with no unresolved threads. [Main PR CI run](https://github.com/paperclipai/paperclip/actions/runs/38728040302): the initial attempt had 54 successful checks, 3 intentional skips, and one serialized-shard failure from an unchanged workspace-authorization test exceeding its 15-second timeout. The single failed-shard retry passed: the final result is 55 successful checks and 3 intentional skips. The same timeout reproduces locally with profiling disabled; both assertions pass with a diagnostic 60-second timeout override (about 23 seconds total). No source or CI timeout was changed. - A dry run against real GitHub attempt 2 of run `38677445871` produced exactly 3 executed jobs, 40 spans, and 510 seconds, excluding 45 carried-over jobs. The partial trace remains explicitly incomplete; this validation did not upload data. - `actionlint` passed for all four changed workflows. `git diff --check` passed. - Full Node 24 `pnpm -r typecheck` and `pnpm build` passed. The local full `pnpm test:run` was stopped after it reported a setup-token timeout and a distribution-plugin-catalog fixture failure. The 30 setup-token tests passed on focused rerun. The plugin fixture fails at `rmSync` of a directory symlink on this macOS/Node 24.11.0 host; the same failure reproduces with the unchanged base Vitest configuration and unchanged test/service files (10 other tests pass). The local full test suite is therefore not claimed green. - A real root-config Vitest probe passed 13 tests with profiling both off and on. Multi-file Vitest, Playwright, and Node probes each verify two distinct filename rows and four correctly parented tests, including repeated test names across files. Separate passing probes exercise the full exporter hierarchy `workflow → job → step → filename → tests`. Intentional failing probes preserve failing exit status. - Verified all 48 measured jobs in latest-head run `38728040302` used AWS: one `paperclip-ci-routing-x64` gate and 47 `paperclip-public-pr-x64` jobs. Profiling preserves AWS authorization and its GitHub-hosted fallback. The separate privileged exporter runs on `ubuntu-latest` outside the measured suite. - [Live Honeycomb baseline](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG): 48 jobs, 731 spans, 479 seconds. It matches [GitHub run 38671274294](https://github.com/paperclipai/paperclip/actions/runs/38671274294). This historical trace has job/step data only; it is not proof of the future label-triggered test-reporting journey. - Verified the dedicated Honeycomb key is ingest-only with dataset creation disabled. Verified the GitHub environment allows only the `master` branch. No repository-level secret of that name exists. Both credential values were scanned absent from source and branch history. After merge, add `profile-ci` to a CI CODEOWNER PR, wait for the entire PR run and exporter, then follow the run-summary link. Collapse job children to compare parallel lanes; expand a job to inspect its steps and tests. ## Risks - Activation requires merging the trusted workflow definitions to `master`. This PR continues using the current trusted reusable workflows until then. - Selected runs produce many spans and artifacts. Oversized artifacts fail export instead of truncating data. Missing or cancelled reporters are marked incomplete. - GitHub timings have second precision. Node test times use reporter receipt time. Rust retains job/step timing but lacks per-test wall timestamps. - CI CODEOWNERS are individual users in the explicit `.github/**` rule. Unsupported team rules and identity lookup failures disable profiling. - An interrupted upload can leave partial data. Retrying uses the same span IDs, but the Events API does not promise deduplication. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, code execution, repository tools, and browser verification. The runtime does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- .github/CODEOWNERS | 1 + .github/scripts/tests/ci-profile.test.mjs | 240 ++++++++++++++++++ .github/scripts/tests/ci-routing.test.mjs | 12 + .github/workflows/ci-profile.yml | 41 +++ .github/workflows/ci-routing.yml | 40 ++- .github/workflows/pr-trusted.yml | 207 ++++++++++++++- .github/workflows/pr.yml | 4 + .gitignore | 3 + doc/DEVELOPING.md | 4 + doc/ci-profiling.md | 147 +++++++++++ .../2026-10-14-ci-honeycomb-profiling.md | 61 +++++ package.json | 4 +- packages/paperclip-eval-kernel/package.json | 2 +- packages/paperclip-runner/package.json | 2 +- .../scripts/run-pr-vitest-lane.mjs | 2 + packages/paperclip-runner/vitest.config.ts | 5 +- scripts/ci-profile/authorize.mjs | 74 ++++++ scripts/ci-profile/export.mjs | 120 +++++++++ scripts/ci-profile/initialize.mjs | 17 ++ scripts/ci-profile/node-reporter.mjs | 41 +++ scripts/ci-profile/node-test.mjs | 16 ++ scripts/ci-profile/playwright-reporter.mjs | 32 +++ scripts/ci-profile/read-artifact.py | 23 ++ scripts/ci-profile/record.mjs | 44 ++++ scripts/ci-profile/reporters.test.mjs | 73 ++++++ scripts/ci-profile/run.mjs | 9 + scripts/ci-profile/trace.mjs | 176 +++++++++++++ scripts/ci-profile/vitest-reporter.mjs | 31 +++ scripts/run-vitest-stable.mjs | 8 +- tests/e2e/playwright.config.ts | 7 +- vitest.config.ts | 5 +- 31 files changed, 1433 insertions(+), 18 deletions(-) create mode 100644 .github/scripts/tests/ci-profile.test.mjs create mode 100644 .github/workflows/ci-profile.yml create mode 100644 doc/ci-profiling.md create mode 100644 doc/plans/2026-10-14-ci-honeycomb-profiling.md create mode 100644 scripts/ci-profile/authorize.mjs create mode 100644 scripts/ci-profile/export.mjs create mode 100644 scripts/ci-profile/initialize.mjs create mode 100644 scripts/ci-profile/node-reporter.mjs create mode 100644 scripts/ci-profile/node-test.mjs create mode 100644 scripts/ci-profile/playwright-reporter.mjs create mode 100644 scripts/ci-profile/read-artifact.py create mode 100644 scripts/ci-profile/record.mjs create mode 100644 scripts/ci-profile/reporters.test.mjs create mode 100644 scripts/ci-profile/run.mjs create mode 100644 scripts/ci-profile/trace.mjs create mode 100644 scripts/ci-profile/vitest-reporter.mjs diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 8687af5aa1b..5deb0af4680 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,6 +1,7 @@ # Replace @cryppadotta if a different maintainer or team should own release infrastructure. .github/** @cryppadotta @devinfoley @nickyleach @forgottendev +scripts/ci-profile/** @cryppadotta @devinfoley @nickyleach @forgottendev scripts/release*.sh @cryppadotta @devinfoley @nickyleach @forgottendev scripts/release-*.mjs @cryppadotta @devinfoley @nickyleach @forgottendev scripts/create-github-release.sh @cryppadotta @devinfoley @nickyleach @forgottendev diff --git a/.github/scripts/tests/ci-profile.test.mjs b/.github/scripts/tests/ci-profile.test.mjs new file mode 100644 index 00000000000..9492dae5847 --- /dev/null +++ b/.github/scripts/tests/ci-profile.test.mjs @@ -0,0 +1,240 @@ +import assert from "node:assert/strict"; +import test from "node:test"; +import { mkdtempSync, mkdirSync, readFileSync, existsSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { execFileSync, spawnSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { buildTrace } from "../../../scripts/ci-profile/trace.mjs"; +import { sendEvents } from "../../../scripts/ci-profile/export.mjs"; +import { associatedProfilePr, ciOwners, trustedOwnerIds } from "../../../scripts/ci-profile/authorize.mjs"; + +const at = (seconds) => new Date(1_800_000_000_000 + seconds * 1000).toISOString(); +const run = { + id: 42, run_attempt: 1, run_number: 7, status: "completed", conclusion: "failure", + repository: { id: 1170821064, full_name: "paperclipai/paperclip" }, head_repository: { id: 99 }, + actor: { id: 123, type: "User" }, triggering_actor: { id: 123, type: "User" }, + head_sha: "abc", head_branch: "branch", created_at: at(0), run_started_at: at(0), + html_url: "https://github.com/paperclipai/paperclip/actions/runs/42", +}; +const job = (id, name, start, end) => ({ + id, run_id: 42, run_attempt: 1, name: `ci / ${name}`, status: "completed", + conclusion: "success", created_at: at(1), started_at: at(start), completed_at: at(end), + labels: ["ubuntu-latest"], steps: [ + { number: 1, name: "Initialize CI profile", started_at: at(start), completed_at: at(start + 1), conclusion: "success" }, + { number: 2, name: "Tests", started_at: at(start + 1), completed_at: at(end), conclusion: "success" }, + ], +}); +const jobs = [job(1, "first", 2, 12), job(2, "second", 3, 20)]; +const record = (fields) => ({ version: 1, runId: "42", attempt: "1", invocation: "invocation", job: "first", framework: "vitest", ...fields }); + +test("one wall-clock trace includes parallel jobs, wait, and every step", () => { + const trace = buildTrace(run, jobs); + assert.equal(trace.wallMs, 20000); + assert.equal(trace.events[0].data.duration_ms, 20000); + assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "job").length, 2); + assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "step").length, 4); + assert.equal(trace.events.filter((event) => event.data["ci.kind"] === "wait").length, 2); + assert.equal(new Set(trace.events.map((event) => event.data["trace.trace_id"])).size, 1); + assert.equal(trace.events.find((event) => event.data.name === "ci / second").data["ci.finishes_last"], true); + assert.equal(trace.events[0].data.error, true); +}); + +test("file/test hierarchy keeps runner timestamps and rejects unrelated artifact data", () => { + const records = [ + record({ kind: "job" }), record({ kind: "job", job: "second" }), + record({ kind: "start" }), record({ kind: "end" }), + record({ kind: "file", id: "file", name: "a.test.ts", file: "a.test.ts", start: Date.parse(at(4)), duration: 6000, status: "passed" }), + record({ kind: "test", id: "case", parent: "file", name: "runs", file: "a.test.ts", start: Date.parse(at(5)), duration: 100, status: "passed", secret: "must not leave artifact" }), + record({ kind: "test", id: "foreign", runId: "41", name: "wrong run" }), + ]; + const trace = buildTrace(run, jobs, records); + const file = trace.events.find((event) => event.data.name === "a.test.ts"); + const entry = trace.events.find((event) => event.data.name === "runs"); + assert.equal(entry.time, at(5)); + assert.equal(entry.data["trace.parent_id"], file.data["trace.span_id"]); + assert.equal(file.data["trace.parent_id"], trace.events.find((event) => event.data.name === "Tests").data["trace.span_id"]); + assert.equal(trace.events.some((event) => event.data.name === "wrong run"), false); + assert.doesNotMatch(JSON.stringify(trace), /must not leave artifact/); + assert.equal(trace.events[0].data["ci.profile_complete"], false); + assert.match(trace.warnings.join(), /1 invalid/); +}); + +test("short files belong to the latest containing step despite second-precision boundaries", () => { + const records = [record({ kind: "file", id: "file", name: "short.test.ts", file: "short.test.ts", + start: Date.parse(at(3)) + 100, duration: 50, status: "passed" })]; + const trace = buildTrace(run, jobs, records); + const file = trace.events.find((event) => event.data.name === "short.test.ts"); + const step = trace.events.find((event) => event.data.name === "Tests"); + assert.equal(file.data["trace.parent_id"], step.data["trace.span_id"]); +}); + +test("cancelled reporters and missing shards remain explicit", () => { + const trace = buildTrace({ ...run, conclusion: "cancelled" }, jobs, [record({ kind: "job" }), record({ kind: "start" })]); + assert.equal(trace.events[0].data["ci.profile_complete"], false); + assert.match(trace.warnings.join(), /Missing timing artifacts for 1 jobs/); + assert.match(trace.warnings.join(), /1 test invocations/); +}); + +test("a completely absent reporter invocation cannot hide behind a successful sibling", () => { + const records = [ + record({ kind: "job" }), record({ kind: "job", job: "second" }), + record({ kind: "expected" }), record({ kind: "start" }), record({ kind: "end" }), + record({ kind: "file", id: "file", name: "a.test.ts", file: "a.test.ts", start: Date.parse(at(4)), duration: 100, status: "passed" }), + record({ kind: "expected", invocation: "reporter-never-loaded" }), + ]; + const trace = buildTrace(run, jobs, records); + assert.equal(trace.events[0].data["ci.profile_complete"], false); + assert.equal(trace.events[0].data["ci.expected_invocations"], 2); + assert.equal(trace.events[0].data["ci.reported_invocations"], 1); + assert.match(trace.warnings.join(), /1 test invocations/); +}); + +test("coverage is conservative after failures and does not merge framework invocation IDs", () => { + const records = [ + record({ kind: "job" }), record({ kind: "job", job: "second" }), + record({ kind: "expected" }), record({ kind: "start" }), record({ kind: "end" }), + ]; + assert.equal(buildTrace({ ...run, conclusion: "success" }, jobs, records).events[0].data["ci.profile_complete"], true); + assert.equal(buildTrace(run, jobs, records).events[0].data["ci.profile_complete"], false); + const trace = buildTrace({ ...run, conclusion: "success" }, jobs, [ + ...records, record({ framework: "node", kind: "expected" }), + ]); + assert.equal(trace.events[0].data["ci.profile_complete"], false); + assert.equal(trace.events[0].data["ci.expected_invocations"], 2); + assert.equal(trace.events[0].data["ci.reported_invocations"], 1); +}); + +test("trusted policy commands work with both current and pre-profile checkouts", () => { + const source = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8"); + const block = source.split(" - name: Select compatible Node test command\n")[1].split("\n - name:")[0]; + const script = block.split(" run: |\n")[1].split("\n").map((line) => line.replace(/^ {10}/, "")).join("\n"); + const directory = mkdtempSync(path.join(tmpdir(), "ci-profile-legacy-")); + try { + writeFileSync(path.join(directory, "probe.test.cjs"), 'require("node:test")("old checkout works", () => {});'); + for (const current of [false, true]) { + if (current) { + mkdirSync(path.join(directory, "scripts")); + symlinkSync(fileURLToPath(new URL("../../../scripts/ci-profile", import.meta.url)), path.join(directory, "scripts/ci-profile")); + } + const env = { ...process.env, GITHUB_ENV: path.join(directory, "env"), PAPERCLIP_CI_PROFILE: "false" }; + delete env.NODE_TEST_CONTEXT; + const result = spawnSync("bash", ["-c", `${script}\nsource "$GITHUB_ENV"\nnode "$CI_NODE_TEST" probe.test.cjs`], { + cwd: directory, env, encoding: "utf8", + }); + assert.equal(result.status, 0, result.stdout + result.stderr); + assert.match(readFileSync(env.GITHUB_ENV, "utf8"), current ? /CI_NODE_TEST=scripts\/ci-profile\/node-test.mjs/ : /CI_NODE_TEST=--test/); + } + } finally { rmSync(directory, { recursive: true, force: true }); } +}); + +test("reruns use separate IDs and clocks and cannot mix job attempts", () => { + const rerun = { ...run, run_attempt: 2, run_started_at: at(2) }; + assert.throws(() => buildTrace(rerun, jobs), /another run or attempt/); + const trace = buildTrace(rerun, jobs.map((entry) => ({ ...entry, run_attempt: 2 }))); + assert.equal(trace.wallMs, 18000); + assert.notEqual(trace.traceId, buildTrace(run, jobs).traceId); + assert.equal(trace.events[0].data["ci.partial_rerun"], true); + assert.throws(() => buildTrace({ ...run, status: "in_progress" }, jobs), /complete/); +}); + +test("partial reruns exclude old jobs that GitHub relabels with the new attempt number", () => { + const rerun = { ...run, run_attempt: 2, run_started_at: at(30) }; + const inventory = [ + { ...job(1, "carried build", 2, 12), run_attempt: 2 }, + { ...job(3, "retried tests", 32, 50), run_attempt: 2, created_at: at(31) }, + ]; + const trace = buildTrace(rerun, inventory); + assert.equal(trace.wallMs, 20000); + assert.equal(trace.jobCount, 1); + assert.equal(trace.events[0].data["ci.carried_job_count"], 1); + assert.equal(trace.events.some((event) => event.data.name === "ci / carried build"), false); + assert.equal(trace.events[0].data["ci.profile_complete"], false); + assert.match(trace.warnings.join(), /1 completed jobs carried over/); + assert.throws(() => buildTrace(rerun, inventory.slice(0, 1)), /No jobs executed/); +}); + +test("label alone cannot authorize outsiders, bots, stale revisions or outsider reruns", () => { + const pr = { user: { id: 123, type: "User" }, base: { repo: run.repository }, head: { repo: { id: 99 }, sha: "abc" }, labels: [{ name: "profile-ci" }] }; + assert.equal(associatedProfilePr(run, [pr], [123]), pr); + for (const changed of [ + { ...pr, labels: [] }, + { ...pr, head: { ...pr.head, sha: "next" } }, + { ...pr, head: { ...pr.head, repo: { id: 100 } } }, + { ...pr, user: { id: 999, type: "User" } }, + { ...pr, user: { id: 123, type: "Bot" } }, + { ...pr, base: { repo: { ...run.repository, id: 100 } } }, + ]) assert.equal(associatedProfilePr(run, [changed], [123]), undefined); + assert.equal(associatedProfilePr({ ...run, actor: { id: 999, type: "User" } }, [pr], [123]), undefined); + assert.equal(associatedProfilePr({ ...run, triggering_actor: { id: 999, type: "User" } }, [pr], [123]), undefined); + assert.equal(associatedProfilePr({ ...run, repository: { ...run.repository, id: 100 } }, [pr], [123]), undefined); + assert.equal(associatedProfilePr(run, [pr], []), undefined); +}); + +test("profiling uses only infrastructure CODEOWNERS resolved at trusted master", () => { + const source = '.github/** @owner @second\ndoc/** @outsider\n'; + assert.deepEqual(ciOwners(source), ["owner", "second"]); + for (const invalid of ['', '.github/**', '.github/** @org/team', '.github/** @owner\n/.github/** @other']) { + assert.throws(() => ciOwners(invalid), /CODEOWNERS/); + } + const endpoints = []; + const sha = "a".repeat(40); + assert.deepEqual(trustedOwnerIds((endpoint) => { + endpoints.push(endpoint); + if (endpoint.endsWith('git/ref/heads/master')) return { object: { sha } }; + if (endpoint.endsWith(`CODEOWNERS?ref=${sha}`)) return { encoding: 'base64', content: Buffer.from(source).toString('base64') }; + return { id: endpoint === '/users/owner' ? 123 : 456, type: 'User' }; + }), [123, 456]); + assert.deepEqual(endpoints, [ + '/repos/paperclipai/paperclip/git/ref/heads/master', + `/repos/paperclipai/paperclip/contents/.github/CODEOWNERS?ref=${sha}`, '/users/owner', '/users/second', + ]); +}); + +test("export checks per-event ingestion results, including partial failures", async () => { + const events = Array.from({ length: 501 }, () => ({ data: {} })); + const sizes = []; + await sendEvents(events, "test-only", "paperclip-ci", async (url, options) => { + assert.equal(url, "https://api.honeycomb.io/1/batch/paperclip-ci"); + assert.equal(options.redirect, "error"); + const size = JSON.parse(options.body).length; + sizes.push(size); + return { ok: true, json: async () => Array.from({ length: size }, () => ({ status: 202 })) }; + }); + assert.deepEqual(sizes, [500, 1]); + await assert.rejects(sendEvents([events[0]], "test-only", "paperclip-ci", async () => ({ ok: true, json: async () => [{ status: 400 }] })), /every span/); +}); + +test("workflow exporter runs trusted code and keeps keys out of the PR jobs", () => { + const workflow = readFileSync(new URL("../../workflows/ci-profile.yml", import.meta.url), "utf8"); + assert.match(workflow, /ref: \$\{\{ github.sha \}\}/); + assert.match(workflow, /startsWith\(github.event.workflow_run.display_title/); + assert.match(workflow, /environment: ci-profiling/); + assert.doesNotMatch(workflow, /head_sha|secrets: inherit|contents: write|pull-requests: write|pnpm install/); + const pr = readFileSync(new URL("../../workflows/pr-trusted.yml", import.meta.url), "utf8"); + assert.doesNotMatch(pr, /HONEYCOMB.*KEY|secrets\./); + assert.match(pr, /if: always\(\) && env.PAPERCLIP_CI_PROFILE == 'true'/); + assert.doesNotMatch(pr, /PAPERCLIP_CI_PROFILE:.*labels/); + assert.equal(pr.match(/PAPERCLIP_CI_PROFILE: \$\{\{ needs.gate.outputs.profile_ci \}\}/g).length, 9); + // upload-artifact excludes files inside dot-directories by default. Keep + // this opt-in scoped to timing NDJSON, never the workspace or other secrets. + assert.equal(pr.match(/path: \.ci-profile\/\*\.ndjson\n {10}include-hidden-files: true/g).length, 9); + const routing = readFileSync(new URL("../../workflows/ci-routing.yml", import.meta.url), "utf8"); + assert.match(routing, /PROFILE_CI: \$\{\{ steps.profile.outputs.profile_ci \}\}/); + assert.match(routing, /\/git\/ref\/heads\/master/); + assert.match(routing, /authorize\.mjs\?ref=\$TRUSTED_PROFILE_SHA/); +}); + +test("artifact parsing never extracts paths or executes files and rejects oversized records", () => { + const directory = mkdtempSync(path.join(tmpdir(), "ci-artifact-test-")); + const archive = path.join(directory, "artifact.zip"); + const reader = fileURLToPath(new URL("../../../scripts/ci-profile/read-artifact.py", import.meta.url)); + try { + execFileSync("python3", ["-c", "import zipfile,sys; z=zipfile.ZipFile(sys.argv[1],'w'); z.writestr('../escape.ndjson', '{\"kind\":\"test\"}\\n'); z.close()", archive]); + const result = execFileSync("python3", [reader, archive], { encoding: "utf8" }); + assert.deepEqual(JSON.parse(result), { kind: "test" }); + assert.equal(existsSync(path.join(directory, "escape.ndjson")), false); + execFileSync("python3", ["-c", "import zipfile,sys; z=zipfile.ZipFile(sys.argv[1],'w'); z.writestr('large.ndjson', 'x'*33000); z.close()", archive]); + assert.notEqual(spawnSync("python3", [reader, archive]).status, 0); + } finally { rmSync(directory, { recursive: true, force: true }); } +}); diff --git a/.github/scripts/tests/ci-routing.test.mjs b/.github/scripts/tests/ci-routing.test.mjs index b6cf5ff2b3e..9d2a07437f6 100644 --- a/.github/scripts/tests/ci-routing.test.mjs +++ b/.github/scripts/tests/ci-routing.test.mjs @@ -97,6 +97,8 @@ const stackedBase = { const routeCases = [ ["allowlisted PR", {}, awsRunner], + ["allowlisted profile label", { EVENT_ACTION: "labeled", EVENT_LABEL_NAME: "profile-ci" }, awsRunner], + ["unrelated label", { EVENT_ACTION: "labeled", EVENT_LABEL_NAME: "documentation" }], ["validated stacked base", stackedBase, awsRunner], ["lagging PR API base snapshot", { TEST_PR_BASE_SHA: "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb" }, awsRunner], ["disabled AWS", { AWS_CI_ENABLED: "false" }], @@ -144,6 +146,16 @@ for (const [name, stack, base, expected] of scopeCases) { } const files = (...paths) => JSON.stringify(paths.map((filename) => ({ filename }))); + +test("profile-ci overrides both middle-stack and release-notes-only skips", () => { + for (const overrides of [ + { STACK_JSON: JSON.stringify({ position: 2, size: 3, base: { ref: "master" } }), PR_BASE_REF: "stack-1" }, + { STACK_JSON: "null", PR_BASE_REF: "master", TEST_COMPARE_FILES: files("releases/v2026.1015.0.md") }, + ]) { + assert.equal(executeStep("Select PR CI scope", { ...overrides, PROFILE_CI: "false" }).full_ci, "false"); + assert.equal(executeStep("Select PR CI scope", { ...overrides, PROFILE_CI: "true" }).full_ci, "true"); + } +}); const manyReleaseNotes = JSON.stringify(Array.from({ length: 300 }, (_, index) => ({ filename: `releases/beta/v2026.1012.${index}-beta.0.md`, }))); diff --git a/.github/workflows/ci-profile.yml b/.github/workflows/ci-profile.yml new file mode 100644 index 00000000000..9034d543c0e --- /dev/null +++ b/.github/workflows/ci-profile.yml @@ -0,0 +1,41 @@ +name: CI profile + +on: + workflow_run: + workflows: [PR] + types: [completed] + +permissions: + actions: read + contents: read + pull-requests: read + +jobs: + export: + if: >- + github.repository == 'paperclipai/paperclip' && + github.event.workflow_run.event == 'pull_request' && + startsWith(github.event.workflow_run.display_title, 'CI profile · PR #') + runs-on: ubuntu-latest + timeout-minutes: 10 + # This environment permits only the master branch. Do not use a repository + # secret: an in-repository PR must not be able to request this credential. + environment: ci-profiling + steps: + # workflow_run is privileged. Never check out its head SHA or run code + # from downloaded artifacts. This checkout is the trusted default branch. + - name: Checkout trusted exporter + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 + with: + ref: ${{ github.sha }} + persist-credentials: false + sparse-checkout: scripts/ci-profile + + - name: Export selected complete CI attempt + env: + GH_TOKEN: ${{ github.token }} + HONEYCOMB_CI_API_KEY: ${{ secrets.HONEYCOMB_CI_API_KEY }} + HONEYCOMB_CI_DATASET: paperclip-ci + HONEYCOMB_TEAM: paperclip + HONEYCOMB_ENVIRONMENT: test + run: node scripts/ci-profile/export.mjs diff --git a/.github/workflows/ci-routing.yml b/.github/workflows/ci-routing.yml index a146b834bd2..f5d1393f3af 100644 --- a/.github/workflows/ci-routing.yml +++ b/.github/workflows/ci-routing.yml @@ -8,6 +8,8 @@ on: value: ${{ jobs.gate.outputs.runner }} full_ci: value: ${{ jobs.gate.outputs.full_ci }} + profile_ci: + value: ${{ jobs.gate.outputs.profile_ci }} permissions: actions: read @@ -22,6 +24,7 @@ jobs: outputs: runner: ${{ steps.route.outputs.runner }} full_ci: ${{ steps.scope.outputs.full_ci }} + profile_ci: ${{ steps.profile.outputs.profile_ci }} steps: - name: Validate PR identity and select runner @@ -35,6 +38,7 @@ jobs: EVENT_REPOSITORY: ${{ github.repository }} EVENT_REPOSITORY_ID: ${{ github.repository_id }} EVENT_ACTION: ${{ github.event.action }} + EVENT_LABEL_NAME: ${{ github.event.label.name }} EVENT_PR_NUMBER: ${{ github.event.pull_request.number }} EVENT_PR_AUTHOR_ID: ${{ github.event.pull_request.user.id }} EVENT_SENDER_ID: ${{ github.event.sender.id }} @@ -76,7 +80,10 @@ jobs: [[ "$EVENT_REPOSITORY_ID" == '1170821064' ]] || fail_closed 'unexpected repository ID' [[ "$EVENT_BASE_REPOSITORY_ID" == '1170821064' ]] || fail_closed 'unexpected base repository ID' [[ -n "$EVENT_BASE_REF" ]] || fail_closed 'missing base branch' - [[ "$EVENT_ACTION" =~ ^(opened|reopened|synchronize)$ ]] || fail_closed 'unsupported pull_request action' + [[ "$EVENT_ACTION" =~ ^(opened|reopened|synchronize|labeled)$ ]] || fail_closed 'unsupported pull_request action' + if [[ "$EVENT_ACTION" == 'labeled' && "${EVENT_LABEL_NAME:-}" != 'profile-ci' ]]; then + fail_closed 'only the profile-ci label can request a profiling run' + fi jq -e ' type == "array" and @@ -215,6 +222,31 @@ jobs: echo "runner=$aws_runner" >> "$GITHUB_OUTPUT" echo '::notice title=AWS CI routing::Using an ephemeral RunsOn Fleet runner' + - name: Authorize CODEOWNERS CI profiling + id: profile + shell: bash + env: + GH_TOKEN: ${{ github.token }} + PROFILE_REQUESTED: ${{ contains(github.event.pull_request.labels.*.name, 'profile-ci') }} + run: | + set -euo pipefail + if [[ "$PROFILE_REQUESTED" != 'true' ]]; then + echo 'profile_ci=false' >> "$GITHUB_OUTPUT" + exit 0 + fi + # Fetch only from canonical master, never the PR ref. Pin both the + # authorization program and CODEOWNERS to the same trusted revision. + if ! TRUSTED_PROFILE_SHA="$(gh api /repos/paperclipai/paperclip/git/ref/heads/master --jq .object.sha)" || + [[ ! "$TRUSTED_PROFILE_SHA" =~ ^[0-9a-f]{40}$ ]] || + ! gh api -H 'Accept: application/vnd.github.raw+json' \ + "/repos/paperclipai/paperclip/contents/scripts/ci-profile/authorize.mjs?ref=$TRUSTED_PROFILE_SHA" \ + > "$RUNNER_TEMP/ci-profile-authorize.mjs"; then + echo 'profile_ci=false' >> "$GITHUB_OUTPUT" + exit 0 + fi + export TRUSTED_PROFILE_SHA + node "$RUNNER_TEMP/ci-profile-authorize.mjs" + - name: Select PR CI scope id: scope shell: bash @@ -223,6 +255,7 @@ jobs: EVENT_NAME: ${{ github.event_name }} EVENT_BASE_SHA: ${{ github.event.pull_request.base.sha }} EVENT_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + PROFILE_CI: ${{ steps.profile.outputs.profile_ci }} STACK_JSON: ${{ toJSON(github.event.pull_request.stack) }} PR_BASE_REF: ${{ github.event.pull_request.base.ref }} run: | @@ -293,6 +326,11 @@ jobs: reason='release notes only' fi + if [[ "${PROFILE_CI:-false}" == 'true' ]]; then + full_ci='true' + reason='profile-ci requests the complete PR suite' + fi + echo "full_ci=$full_ci" >> "$GITHUB_OUTPUT" echo "::notice title=PR CI scope::$reason; full_ci=$full_ci" diff --git a/.github/workflows/pr-trusted.yml b/.github/workflows/pr-trusted.yml index 2ff512638af..3a4db1c85ac 100644 --- a/.github/workflows/pr-trusted.yml +++ b/.github/workflows/pr-trusted.yml @@ -11,6 +11,11 @@ permissions: contents: read pull-requests: read +# Test reporters write local artifacts only. The workflow_run exporter owns +# the Honeycomb credential, so fork PRs cannot execute with it. +env: + PAPERCLIP_CI_PROFILE_DIR: ${{ github.workspace }}/.ci-profile + concurrency: group: pr-${{ github.event.pull_request.number }} cancel-in-progress: true @@ -24,6 +29,10 @@ jobs: runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 10 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: policy + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -31,6 +40,20 @@ jobs: persist-credentials: false fetch-depth: 0 + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + + - name: Select compatible Node test command + run: | + # A rerun can use this trusted workflow with a pre-feature checkout. + if [[ -f scripts/ci-profile/node-test.mjs ]]; then + echo 'CI_NODE_TEST=scripts/ci-profile/node-test.mjs' >> "$GITHUB_ENV" + else + echo 'CI_NODE_TEST=--test' >> "$GITHUB_ENV" + fi + - name: Block manual lockfile edits if: >- github.head_ref != 'chore/refresh-lockfile' && @@ -76,28 +99,28 @@ jobs: run: node ./scripts/check-no-git-push.mjs - name: Test no-git-push check - run: node --test ./scripts/check-no-git-push.test.mjs + run: node "$CI_NODE_TEST" ./scripts/check-no-git-push.test.mjs - name: Validate feature module boundaries run: pnpm check:module-boundaries - name: Test feature module boundary check - run: node --test ./scripts/check-module-boundaries.test.mjs + run: node "$CI_NODE_TEST" ./scripts/check-module-boundaries.test.mjs - name: Test PR quality-gate scripts - run: node --test '.github/scripts/tests/*.test.mjs' + run: node "$CI_NODE_TEST" '.github/scripts/tests/*.test.mjs' - name: Test general-server shard partition - run: node --test ./scripts/__tests__/run-vitest-stable-shard.test.mjs + run: node "$CI_NODE_TEST" ./scripts/__tests__/run-vitest-stable-shard.test.mjs - name: Test e2e shard partition - run: node --test ./scripts/__tests__/e2e-shard.test.mjs + run: node "$CI_NODE_TEST" ./scripts/__tests__/e2e-shard.test.mjs - name: Test release verify workflow wiring - run: node --test ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs + run: node "$CI_NODE_TEST" ./scripts/__tests__/release-verify-workflow.test.mjs ./scripts/cloud-source-verification.test.mjs ./scripts/standard-image-contract.test.mjs - name: Test standalone package build concurrency - run: node --test ./scripts/__tests__/build-standalone-concurrency.test.mjs + run: node "$CI_NODE_TEST" ./scripts/__tests__/build-standalone-concurrency.test.mjs - name: Validate release package manifest run: node ./scripts/release-package-map.mjs check @@ -113,6 +136,17 @@ jobs: - name: Validate dependency resolution run: pnpm install --resolution-only --ignore-scripts --no-frozen-lockfile + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + typecheck_release_registry: name: Typecheck + Release Registry needs: [gate] @@ -120,12 +154,21 @@ jobs: runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 20 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Typecheck + Release Registry + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -283,6 +326,17 @@ jobs: - name: Verify release registry test coverage run: pnpm run test:release-registry + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + general_tests: name: General tests (${{ matrix.group_label }}) needs: [gate] @@ -398,12 +452,21 @@ jobs: - group: general-workspaces-c group_label: workspaces-c + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: General tests (${{ matrix.group_label }}) + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -456,6 +519,17 @@ jobs: pnpm test:run:general -- --group '${{ matrix.group }}' fi + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + docker_context_integrity: name: Docker context integrity needs: gate @@ -463,6 +537,10 @@ jobs: runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 15 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Docker context integrity + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 @@ -472,6 +550,11 @@ jobs: # Not every runner the gate can select ships the Buildx plugin — # the image-build workflows set it up explicitly, so this lane does # too rather than failing before it checks anything. + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Set up Docker Buildx uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 @@ -485,6 +568,17 @@ jobs: - name: Run generated-file drift checks against the Docker build context run: docker buildx build --file .github/docker-context-checks.Dockerfile . + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + verify: # Preserve the legacy required-check name while the underlying work runs in parallel. name: verify @@ -558,12 +652,21 @@ jobs: command: test:typescript:vitest --shard=2/2 release_crate: true + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Verify Paperclip Runner (${{ matrix.lane_label }}) + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -732,6 +835,17 @@ jobs: # read the shard flag as a test filter. run: pnpm --filter @paperclipai/paperclip-runner ${{ matrix.command }} + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + build: name: Build needs: [gate] @@ -739,12 +853,21 @@ jobs: runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 20 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Build + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -948,6 +1071,17 @@ jobs: fi node scripts/verify-grok-npm-install.mjs + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + verify_serialized_server: name: Verify serialized server suites (${{ matrix.shard_label }}) needs: [gate] @@ -994,12 +1128,21 @@ jobs: shard_count: 9 shard_label: 9/9 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Verify serialized server suites (${{ matrix.shard_label }}) + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -1046,6 +1189,17 @@ jobs: - name: Run serialized server test shard run: pnpm test:run:serialized -- --shard-index ${{ matrix.shard_index }} --shard-count ${{ matrix.shard_count }} + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + canary_dry_run: name: Canary Dry Run needs: [gate] @@ -1053,12 +1207,21 @@ jobs: runs-on: ${{ needs.gate.outputs.runner }} timeout-minutes: 20 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: Canary Dry Run + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -1234,6 +1397,16 @@ jobs: # build` output, which the Build lane also produces, so it moved there: # see "Verify built-in Grok from a clean public npm install" in `build`. + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 e2e_shards: name: e2e shard (${{ matrix.shard_label }}) @@ -1281,12 +1454,21 @@ jobs: shard_count: 8 shard_label: 8/8 + env: + PAPERCLIP_CI_PROFILE: ${{ needs.gate.outputs.profile_ci }} + PAPERCLIP_CI_PROFILE_JOB: e2e shard (${{ matrix.shard_label }}) + steps: - name: Checkout repository uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7 with: persist-credentials: false + - name: Initialize CI profile + continue-on-error: true + if: env.PAPERCLIP_CI_PROFILE == 'true' + run: node scripts/ci-profile/initialize.mjs + - name: Setup Node.js for pnpm bootstrap uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7 with: @@ -1376,6 +1558,17 @@ jobs: tests/e2e/test-results/ retention-days: 14 + - name: Upload CI profile timings + continue-on-error: true + if: always() && env.PAPERCLIP_CI_PROFILE == 'true' + uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7 + with: + name: ci-profile-${{ github.run_attempt }}-${{ github.job }}-${{ strategy.job-index || 0 }} + path: .ci-profile/*.ndjson + include-hidden-files: true + if-no-files-found: warn + retention-days: 14 + e2e: # Preserve the legacy required-check name while the specs run sharded # across the matrix above (same pattern as the `verify` aggregate). diff --git a/.github/workflows/pr.yml b/.github/workflows/pr.yml index 388c13838c7..9b66bc04368 100644 --- a/.github/workflows/pr.yml +++ b/.github/workflows/pr.yml @@ -1,7 +1,10 @@ name: PR +run-name: >- + ${{ contains(github.event.pull_request.labels.*.name, 'profile-ci') && (github.event.action != 'labeled' || github.event.label.name == 'profile-ci') && format('CI profile · PR #{0}', github.event.pull_request.number) || '' }} on: pull_request: + types: [opened, synchronize, reopened, labeled] permissions: actions: read @@ -10,6 +13,7 @@ permissions: jobs: ci: + if: github.event.action != 'labeled' || github.event.label.name == 'profile-ci' # Master requires CODEOWNERS review for .github/**, including this workflow. # The AWS runner group permits pr-trusted.yml@refs/heads/master. uses: paperclipai/paperclip/.github/workflows/pr-trusted.yml@master diff --git a/.gitignore b/.gitignore index 8a43761535a..5907c86c94a 100644 --- a/.gitignore +++ b/.gitignore @@ -89,3 +89,6 @@ test-results/ # Local native Runner test/session state .paperclip-runner-prp/ + +# Opt-in CI timing artifacts +.ci-profile/ diff --git a/doc/DEVELOPING.md b/doc/DEVELOPING.md index 7ab7698cd0d..b82af99d5b5 100644 --- a/doc/DEVELOPING.md +++ b/doc/DEVELOPING.md @@ -75,6 +75,10 @@ When changing the workflow path or branch, authorize the new reference before updating the caller. Retain older authorized SHA references while queued runs or supported reruns still use them. +For a full wall-clock profile of all main PR CI jobs, add `profile-ci` to the PR. +See [CI profiling](ci-profiling.md) for the Honeycomb waterfall, coverage, +operator setup, and activation requirements. + ## Start Dev From repo root: diff --git a/doc/ci-profiling.md b/doc/ci-profiling.md new file mode 100644 index 00000000000..d2031d9f7f8 --- /dev/null +++ b/doc/ci-profiling.md @@ -0,0 +1,147 @@ +# Profile the complete PR suite + +Add the `profile-ci` label to a PR authored by a CI CODEOWNER. Adding the label starts a fresh `PR` workflow +run. Further pushes while the label remains also run with profiling enabled. +The run is named **CI profile · PR #number**. It runs all lanes of the main PR +suite, including middle-of-stack and release-notes-only PRs that normally skip +most lanes. + +After the run finishes, open its Summary and follow **Open the Honeycomb +waterfall**. The separate **CI profile** workflow exports the trace after all +jobs finish. Its summary has the exact time range, job/span counts, and coverage +warnings. Keep the label on the PR and its head unchanged until export finishes. +Remove the label to stop profiling subsequent pushes. To profile the same +revision again, remove and re-add the label or rerun all jobs. + +The PR author, original workflow actor, and rerun actor must all be individual +users in the trusted `master` branch's `.github/**` CODEOWNERS rule. GitHub user +IDs are resolved from that rule; PR edits to CODEOWNERS cannot grant access. +A label on an outside contributor's PR does not enable profiling, even when a +maintainer applies it. Failed identity lookups disable profiling. + +Ordinary PRs do not write timing artifacts or export traces. The exporter job is +skipped for ordinary run names. Adding another label does not run the CI matrix. + +Profiling preserves the existing trusted AWS runner routing. Eligible PR jobs +use `paperclip-public-pr-x64`; the routing gate uses `paperclip-ci-routing-x64`. +The existing fail-closed GitHub-hosted fallback still applies if AWS routing +cannot be authorized. Only the separate post-run Honeycomb exporter uses +`ubuntu-latest` directly; it is outside the measured PR suite. + +## Read the waterfall + +The workflow root measures elapsed time from run creation to the last job's +completion. For reruns, it starts at that attempt's `run_started_at` instead of +the original creation time. Every job is a child on the same wall clock, with +its steps and available test details beneath it. All spans use a sample rate of +one. There is no sampling within the selected attempt. For a compact Gantt view, +open a job row's menu and choose **Collapse spans at this depth**. Expand an +individual job to inspect its steps and test files. + +The expandable hierarchy is **workflow → job → step → test filename → tests**. +Vitest, Playwright, and Node file rows use repository-relative filenames, also +available in `test.file`. Separate files retain their own children even when +individual test names repeat across files. + +- Compare the jobs that finish last (`ci.finishes_last`) with other overlapping + work. A long job that finishes early is not necessarily delaying CI. +- Expand setup, dependency installation, cache restore, Rust compilation, builds, + and tests to find repeated or slow work. +- `ci.execution_ms` measures a job's execution. `ci.start_offset_ms` shows when + it started relative to the whole run. Do not add parallel job durations to + calculate elapsed time. +- Waiting spans cover job creation to runner start. Dependency waits can occur + before GitHub creates a job, so they appear as gaps from the workflow start. + GitHub does not expose a separate ready timestamp; do not interpret every gap + as runner queue time or claim this is a computed dependency critical path. + +GitHub job/step timestamps have second precision. Vitest and Playwright record +test timestamps at the runner. Node's test reporter records completion receipt +time and duration (`test.clock=reporter-receipt`); buffered IPC delivery can shift +those small test spans. Vitest file spans include its test/hook execution and +carry collection/setup durations as fields. Playwright file spans cover the +first through last test attempt in that file. Rust and standalone script checks +retain their full job/step timings; stable libtest does not expose per-test wall +timestamps. The trace is CI timing data, not application tracing. + +Scope includes all main PR lanes: policy, typecheck/release registry, general and +serialized tests, Runner checks, Docker context checks, build, canary dry run, +browser E2E, and aggregate checks. Separate Storybook, image, deployment, release, +and live-provider eval workflows are outside this trace. + +## Failure, coverage, and reruns + +The exporter uses GitHub's attempt-specific job API and paginates jobs and +artifacts. It includes failed and cancelled jobs and uploads available test +timings with `always()`. A force-cancelled or failed setup job may leave no +artifact. Coverage warnings and `ci.profile_complete=false` make missing +artifacts, interrupted reporters, rejected data, and absent test details visible. +Test launchers write an expected-invocation record before starting each Vitest, +Playwright, or Node process. Each reporter must write matching start/end records, +so a missing reporter cannot hide behind another successful invocation in the +same job. Compare `ci.expected_invocations` with `ci.reported_invocations`. +Failed or cancelled workflows are conservatively incomplete because a failed +command can prevent later tests or steps from running. + +Each attempt gets a different trace ID. Rerunning failed jobs alone produces an +attempt trace for those jobs; it does not replay completed jobs. Reruns are marked +`ci.partial_rerun=true` conservatively. Use a fresh label-triggered run for an +unambiguous full-suite baseline. GitHub may carry earlier successful jobs into +the new attempt's job inventory. Jobs completed before the new attempt started +are excluded and counted in `ci.carried_job_count`. + +If export fails, inspect **CI profile** and rerun that exporter after fixing the +cause. A partial HTTP export may already have sent some spans; retries use the +same IDs but the Events API does not promise deduplication. Timing artifacts +remain available for 14 days. Invalid/oversized artifacts stop export instead of +silently truncating the suite. No logs, environment dumps, error messages, stack +traces, or credentials are included in the exported field allowlist. + +## Operator setup and activation + +Use the existing Honeycomb `paperclip` team, `test` environment, and a separate +service dataset named `paperclip-ci`. Create GitHub environment `ci-profiling` with **Selected branches and tags**: +allow only the branch `master`, with no PR-ref or tag patterns. Store environment +secret `HONEYCOMB_CI_API_KEY` with a dedicated ingest-only key for the Honeycomb +`test` environment. Do not create a repository- or organization-level secret of +that name: the environment restriction prevents in-repository PR workflows from +requesting the key. Tests still execute the PR branch; only the separate exporter +runs from `master`. It needs event +ingest permission; dataset creation permission is needed only if the dataset +has not been created yet. Do not store the environment configuration key in CI. +Create the repository label `profile-ci` if it does not exist. + +`ci-profile.yml` runs from the trusted default-branch revision. It checks out only +the exporter, installs no dependencies, and parses PR artifacts as bounded JSON +without extracting or executing their contents. The PR jobs never receive the +Honeycomb key. The exporter independently verifies CI CODEOWNER identities, +the current label, repository, and exact tested PR revision before reading +artifacts or sending data. Its credential is sent only in an HTTPS header to +`api.honeycomb.io`; redirects are rejected. Exported fields are allowlisted. +`CODEOWNERS` also covers `scripts/ci-profile/**`, so changes to the privileged +exporter require infrastructure-owner review. + +The new trusted workflow and routing definitions must merge to master before +the label-triggered journey is active. A PR uses `pr-trusted.yml@master` and +`ci-routing.yml@master`; do not change those trust references just to test this +feature before merge. + +## Local qualification + +```sh +node --test .github/scripts/tests/ci-profile.test.mjs .github/scripts/tests/ci-routing.test.mjs +node --test scripts/ci-profile/reporters.test.mjs +actionlint .github/workflows/{pr,pr-trusted,ci-routing,ci-profile}.yml +node scripts/ci-profile/export.mjs --run-id=RUN_ID --attempt=1 --historical --dry-run --output=/tmp/ci-trace.json +``` + +`--historical` is local-only and permits a deliberate export of an existing +unlabeled run. It has complete GitHub job/step timing but cannot reconstruct +test details that were never recorded. Omit `--dry-run` and supply +`HONEYCOMB_CI_API_KEY` to send the selected baseline to Honeycomb. It is labeled +as historical/incomplete test coverage; it is not proof of the label-triggered +workflow. + +References: [Honeycomb trace fields](https://docs.honeycomb.io/get-started/best-practices/organizing-data/), +[Events API](https://docs.honeycomb.io/api/events/create-events), +[GitHub workflow_run](https://docs.github.com/en/actions/reference/workflows-and-actions/events-that-trigger-workflows#workflow_run). diff --git a/doc/plans/2026-10-14-ci-honeycomb-profiling.md b/doc/plans/2026-10-14-ci-honeycomb-profiling.md new file mode 100644 index 00000000000..2a9a9ed3ebd --- /dev/null +++ b/doc/plans/2026-10-14-ci-honeycomb-profiling.md @@ -0,0 +1,61 @@ +# Opt-in CI profiling + +Goal: label a PR `profile-ci`, run the complete main PR suite, and open one +Honeycomb waterfall showing the full elapsed timeline. Use that evidence to +decide which CI work to speed up or remove. CI optimizations themselves are a +follow-up, not part of this change. + +## Acceptance + +| Case | Required behavior | Evidence | +| --- | --- | --- | +| Unlabeled PR | Existing scope and commands; no profile export | Routing and exporter tests | +| Labeled CI CODEOWNER PR, including stacked/release-notes-only | All main PR lanes run | Routing tests and full-run job inventory | +| Completed profile | One trace per run attempt; every job and step uses actual timestamps | Honeycomb waterfall reconciled against GitHub job API | +| Parallel work | Shared wall clock, with waiting and execution distinguishable | Overlapping lanes and root duration checked against GitHub | +| Tests | Vitest and Playwright file/test timings under their job/step; Node test timings where supported | Real reporter smoke runs and hierarchy checks | +| Failure/cancellation/rerun | Preserve completed timing data; mark missing coverage; attempts stay separate | Exporter regression cases | +| Credentials | Master-only environment owns ingest secret; trusted exporter verifies CI CODEOWNER author and actors; PR jobs only write timing artifacts | Workflow review, hostile-identity cases, and environment settings | + +Scope: the main `PR` workflow and every lane of `pr-trusted.yml`, including +policy, build, typecheck/release-registry, general/serialized test shards, +Runner checks, canary, and browser E2E. Separate release/deploy/live-provider +workflows remain outside this first trace. Rust checks are visible as complete +job/step timings; stable libtest does not provide per-test wall-clock reports. +No sampling inside a selected run. + +## State + +- Branch: `codex/ci-honeycomb-profiling` in the existing `76aa` worktree. +- PR: https://github.com/paperclipai/paperclip/pull/16378 +- Baseline: GitHub run `38671274294`, 48 jobs, 479 seconds from creation to last + job completion. Raw GitHub evidence is in `/tmp/paperclip-ci-profile-evidence`. +- Honeycomb: existing `paperclip` team / `test` environment; dedicated `paperclip-ci` + dataset. Authorized local configuration key ingested the historical baseline; + verified 731 spans and 7m 59s in the [live Honeycomb waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace/oaPMzw8zGwG). +- Implemented label selection, full-suite override, test timing reporters, trusted + post-run exporter, direct waterfall link, and missing-data diagnostics. +- Verified: 48-job / 731-span historical reconstruction; 508 workflow/reporter/shard + checks; real root-config probe with 13 tests and one file span; full Node 24 + typecheck and build. The local full Vitest run was stopped after a setup-token + timeout and a distribution-plugin-catalog fixture failure. The setup-token + suite passed on focused rerun; the directory-symlink removal failure reproduces + with the unchanged base configuration and test/service files on this macOS + Node 24.11.0 host. The PR body records the latest remote CI and review results. +- Created and verified the repository `profile-ci` label. Dedicated Honeycomb + ingest key `paperclip-ci-profile` has no dataset-creation permission. It is + stored only as `HONEYCOMB_CI_API_KEY` in the `ci-profiling` GitHub environment, + restricted to the `master` branch. Temporary key file removed; both credentials + scanned absent from repository files and branch history. +- Added CODEOWNER author/actor authorization and rejection tests, plus CODEOWNERS + coverage of the privileged exporter. +- Review fixes: retain `node --test` for older checked-out revisions; record expected + invocations outside reporters to detect wholly missing reporters. The shard + inventory now writes synchronously before exit after reproducing truncation + on both the unchanged baseline and working branch. +- Real partial-rerun evidence showed GitHub carrying old successful jobs into + the new attempt's inventory with the new attempt number. Exclude jobs whose + completion precedes the attempt start, and record their count explicitly. +- Delivery: reviewable PR and live Honeycomb evidence. Activation of new trusted + workflow definitions requires merge to master; do not claim pre-merge evidence + proves the post-merge label-triggered journey. diff --git a/package.json b/package.json index 66cf20785c7..117c6439ecc 100644 --- a/package.json +++ b/package.json @@ -61,11 +61,11 @@ "smoke:posthog-live": "node scripts/smoke/posthog-live.mjs", "smoke:pipelines-tutorial": "./scripts/smoke/pipelines-tutorial-smoke.sh", "smoke:terminal-bench-loop-skill": "node scripts/smoke/terminal-bench-loop-skill-smoke.mjs", - "test:release-registry": "node --test scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", + "test:release-registry": "node scripts/ci-profile/node-test.mjs scripts/ci-profile/reporters.test.mjs scripts/__tests__/canary-startup-playwright.test.mjs scripts/__tests__/cursor-public-install-sandbox.test.mjs scripts/__tests__/grok-public-install-sandbox.test.mjs scripts/__tests__/public-install-closure.test.mjs scripts/__tests__/prefetch-child.test.mjs scripts/__tests__/stage-bundled-packages.test.mjs scripts/__tests__/pack-staged-packages.test.mjs scripts/__tests__/retain-runner-qualification-packages.test.mjs scripts/__tests__/runner-release-crate-cache.test.mjs scripts/verify-release-registry-state.test.mjs scripts/release-package-map.test.mjs scripts/check-release-package-bootstrap.test.mjs scripts/check-no-git-push.test.mjs scripts/release-lib.test.mjs scripts/release-registry-versions.test.mjs scripts/link-plugin-dev-sdk.test.js scripts/acpx-patch-packaging.test.mjs scripts/service-onboard-smoke.test.mjs scripts/docker-onboard-smoke.test.mjs scripts/preview-artifacts.test.mjs scripts/cloud-migrator-artifacts.test.mjs", "storybook-visual:baseline": "node scripts/storybook-visual-baseline.mjs", "test:storybook-visual": "node scripts/storybook-visual-baseline.mjs download && node scripts/storybook-visual-baseline.mjs verify && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts", "test:storybook-visual:update": "node scripts/storybook-visual-baseline.mjs download && pnpm build-storybook && npx playwright test --config tests/storybook-visual/playwright.config.ts --update-snapshots && node scripts/storybook-visual-baseline.mjs pack", - "test:e2e": "npx playwright test --config tests/e2e/playwright.config.ts", + "test:e2e": "node scripts/ci-profile/run.mjs playwright npx playwright test --config tests/e2e/playwright.config.ts", "test:e2e:runner": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/launch.ts", "test:e2e:runner:image-id": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/daytona-image-content.ts", "test:e2e:runner:judge-first-task": "node cli/node_modules/tsx/dist/cli.mjs tests/runner-e2e/first-task-judge.ts", diff --git a/packages/paperclip-eval-kernel/package.json b/packages/paperclip-eval-kernel/package.json index 676efbf12b4..86297bb0541 100644 --- a/packages/paperclip-eval-kernel/package.json +++ b/packages/paperclip-eval-kernel/package.json @@ -17,7 +17,7 @@ "scripts": { "build": "tsc -p tsconfig.json", "typecheck": "tsc -p tsconfig.json --noEmit", - "test": "pnpm run build && node --test test/*.test.mjs" + "test": "pnpm run build && node ../../scripts/ci-profile/node-test.mjs test/*.test.mjs" }, "engines": { "node": ">=24.11.0" diff --git a/packages/paperclip-runner/package.json b/packages/paperclip-runner/package.json index 66218496966..738b9926fdd 100644 --- a/packages/paperclip-runner/package.json +++ b/packages/paperclip-runner/package.json @@ -81,7 +81,7 @@ "typecheck:browser": "tsc -p tsconfig.browser.json --noEmit", "test": "pnpm run test:typescript && pnpm run test:rust", "test:typescript": "pnpm run test:typescript:prep && vitest run", - "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node --test test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs", + "test:typescript:prep": "pnpm run ensure:eval-build-deps && pnpm run build:rust && node ../../scripts/ci-profile/node-test.mjs test/protocol-contract.test.mjs test/acpx-sidecar-contract.test.mjs test/acpx-codex-package-contract.test.mjs test/acpx-model-selection-package-contract.test.mjs test/acpx-message-boundaries-package-contract.test.mjs test/acpx-rich-extensions-package-contract.test.mjs test/acpx-pi-receipt-package-contract.test.mjs test/acpx-response-delivery-package-contract.test.mjs scripts/candidate-provider-pack.test.mjs scripts/cursor-provisioner-layout.test.mjs scripts/release-provider-pack.test.mjs scripts/aws-agentcore-provisioning.test.mjs scripts/build-verified-provider-entrypoints.test.mjs scripts/local-provider-smoke-environment.test.mjs scripts/materialize-opencode-binary.test.mjs scripts/installed-package-pack.test.mjs scripts/runner-package-executable.test.mjs scripts/stage-runner-binary.test.mjs scripts/build-inputs.test.mjs", "test:typescript:vitest": "node ./scripts/run-pr-vitest-lane.mjs", "test:rust": "cargo test --release --manifest-path runner/Cargo.toml --locked --workspace", "test:codex": "cargo test --manifest-path runner/Cargo.toml --locked -p paperclip-runner-core --test codex_provider", diff --git a/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs b/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs index cc49d2a94e2..51e980aee91 100644 --- a/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs +++ b/packages/paperclip-runner/scripts/run-pr-vitest-lane.mjs @@ -24,6 +24,7 @@ import { spawnSync } from "node:child_process"; import { dirname, resolve } from "node:path"; import { fileURLToPath } from "node:url"; +import { invocationEnv } from "../../../scripts/ci-profile/record.mjs"; const packageRoot = resolve(dirname(fileURLToPath(import.meta.url)), ".."); const workspaceRoot = resolve(packageRoot, "../.."); @@ -62,6 +63,7 @@ for (const planned of plannedCommands) { const result = spawnSync(planned.command, planned.args, { cwd: planned.cwd, stdio: "inherit", + env: planned.args[0] === "exec" && planned.args[1] === "vitest" ? invocationEnv("vitest") : process.env, }); if (result.status !== 0) { process.exit(result.status ?? 1); diff --git a/packages/paperclip-runner/vitest.config.ts b/packages/paperclip-runner/vitest.config.ts index 3b8076fa180..3514fa1ea01 100644 --- a/packages/paperclip-runner/vitest.config.ts +++ b/packages/paperclip-runner/vitest.config.ts @@ -1,7 +1,7 @@ import { resolve } from "node:path"; import { fileURLToPath } from "node:url"; -import { defineConfig } from "vitest/config"; +import { configDefaults, defineConfig } from "vitest/config"; const packageRoot = fileURLToPath(new URL(".", import.meta.url)); @@ -13,6 +13,9 @@ export default defineConfig({ ], }, test: { + ...(process.env.PAPERCLIP_CI_PROFILE === "true" + ? { reporters: [...configDefaults.reporters, resolve(packageRoot, "../../scripts/ci-profile/vitest-reporter.mjs")] } + : {}), environment: "node", include: [ "src/**/*.test.ts", diff --git a/scripts/ci-profile/authorize.mjs b/scripts/ci-profile/authorize.mjs new file mode 100644 index 00000000000..f863e22d65e --- /dev/null +++ b/scripts/ci-profile/authorize.mjs @@ -0,0 +1,74 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync, readFileSync } from "node:fs"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; + +const repository = "paperclipai/paperclip"; +const repositoryId = 1170821064; +export const getGithub = (endpoint) => JSON.parse(execFileSync("gh", ["api", endpoint], { + encoding: "utf8", maxBuffer: 4 * 1024 * 1024, timeout: 30_000, +})); + +// This is deliberately the CI infrastructure owner list, not the union of +// unrelated documentation owners. Unsupported teams/ambiguous rules deny access. +export function ciOwners(contents) { + const rules = contents.split(/\r?\n/).map((line) => line.split("#")[0].trim().split(/\s+/)) + .filter(([pattern]) => pattern === ".github/**" || pattern === "/.github/**"); + if (rules.length !== 1 || rules[0].length < 2 + || rules[0].slice(1).some((owner) => !/^@[a-z\d](?:[a-z\d-]*[a-z\d])?$/i.test(owner))) { + throw new Error("CI CODEOWNERS must have one explicit .github/** rule with individual GitHub users"); + } + return rules[0].slice(1).map((owner) => owner.slice(1)); +} + +export function trustedOwnerIds(get = getGithub, trustedSha) { + const sha = trustedSha || get(`/repos/${repository}/git/ref/heads/master`).object.sha; + if (!/^[a-f\d]{40}$/.test(sha)) throw new Error("Invalid trusted CODEOWNERS revision"); + const file = get(`/repos/${repository}/contents/.github/CODEOWNERS?ref=${sha}`); + if (file.encoding !== "base64") throw new Error("Cannot read trusted CODEOWNERS"); + return ciOwners(Buffer.from(file.content, "base64").toString("utf8")).map((login) => { + const user = get(`/users/${login}`); + if (user.type !== "User" || !Number.isSafeInteger(user.id) || user.id <= 0) { + throw new Error("Cannot resolve a CI CODEOWNER identity"); + } + return user.id; + }); +} + +export function authorizedActors(run, ownerIds) { + const allowed = (user) => user?.type === "User" && ownerIds.includes(user.id); + return run.repository?.id === repositoryId && run.repository?.full_name === repository + && allowed(run.actor) && allowed(run.triggering_actor); +} + +export function associatedProfilePr(run, pulls, ownerIds) { + if (!authorizedActors(run, ownerIds)) return undefined; + return pulls.find((pr) => pr.base.repo.id === repositoryId && pr.base.repo.full_name === repository + && pr.head.repo?.id === run.head_repository?.id && pr.head.sha === run.head_sha + && pr.user?.type === "User" && ownerIds.includes(pr.user.id) + && pr.labels.some((label) => label.name === "profile-ci")); +} + +async function main() { + let enabled = false; + try { + const event = JSON.parse(readFileSync(process.env.GITHUB_EVENT_PATH, "utf8")); + if (event.pull_request?.labels?.some((label) => label.name === "profile-ci")) { + const runId = process.env.GITHUB_RUN_ID; + const attempt = process.env.GITHUB_RUN_ATTEMPT; + if (!/^\d+$/.test(runId) || !/^[1-9]\d*$/.test(attempt)) throw new Error("Invalid CI run identity"); + const run = getGithub(`/repos/${repository}/actions/runs/${runId}/attempts/${attempt}`); + const pr = getGithub(`/repos/${repository}/pulls/${event.pull_request.number}`); + enabled = run.event === "pull_request" && run.path === ".github/workflows/pr.yml" + && Boolean(associatedProfilePr(run, [pr], trustedOwnerIds(getGithub, process.env.TRUSTED_PROFILE_SHA))); + } + } catch { + console.log("::notice::CI profiling authorization could not be verified; profiling disabled."); + } + appendFileSync(process.env.GITHUB_OUTPUT, `profile_ci=${enabled}\n`); + console.log(`::notice::CI CODEOWNERS profiling authorization: ${enabled}`); +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch(() => { console.error("CI profiling authorization failed"); process.exitCode = 1; }); +} diff --git a/scripts/ci-profile/export.mjs b/scripts/ci-profile/export.mjs new file mode 100644 index 00000000000..48e726b42ed --- /dev/null +++ b/scripts/ci-profile/export.mjs @@ -0,0 +1,120 @@ +import { execFileSync } from "node:child_process"; +import { appendFileSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import { buildTrace, traceUrl } from "./trace.mjs"; +import { associatedProfilePr, authorizedActors, trustedOwnerIds } from "./authorize.mjs"; + +const repository = "paperclipai/paperclip"; +const repoPath = `/repos/${repository}`; +const maxBytes = 32 * 1024 * 1024; +const gh = (args) => execFileSync("gh", ["api", ...args], { maxBuffer: maxBytes, timeout: 60_000 }); +const get = (endpoint) => JSON.parse(gh([endpoint]).toString()); +const pages = (endpoint, field) => JSON.parse(gh(["--paginate", "--slurp", endpoint]).toString()) + .flatMap((page) => field ? page[field] : page); + +export async function sendEvents(events, apiKey, dataset, fetcher = fetch) { + if (!apiKey) throw new Error("HONEYCOMB_CI_API_KEY is not configured"); + for (let offset = 0; offset < events.length; offset += 500) { + const batch = events.slice(offset, offset + 500); + const response = await fetcher(`https://api.honeycomb.io/1/batch/${encodeURIComponent(dataset)}`, { + method: "POST", headers: { "X-Honeycomb-Team": apiKey, "Content-Type": "application/json" }, + body: JSON.stringify(batch), redirect: "error", signal: AbortSignal.timeout(30_000), + }); + if (!response.ok) throw new Error(`Honeycomb export failed (HTTP ${response.status})`); + const results = await response.json(); + if (!Array.isArray(results) || results.length !== batch.length || results.some((result) => result.status !== 202)) { + throw new Error("Honeycomb did not accept every span in the batch"); + } + } +} + +async function main() { + const args = process.argv.slice(2); + const arg = (name) => args.find((value) => value.startsWith(`${name}=`))?.slice(name.length + 1); + const event = process.env.GITHUB_EVENT_PATH ? JSON.parse(readFileSync(process.env.GITHUB_EVENT_PATH, "utf8")) : null; + const runId = event?.workflow_run?.id ?? arg("--run-id"); + const attempt = event?.workflow_run?.run_attempt ?? arg("--attempt"); + if (!/^\d+$/.test(String(runId)) || !/^[1-9]\d*$/.test(String(attempt))) { + throw new Error("Provide --run-id and --attempt, or a workflow_run event"); + } + const run = get(`${repoPath}/actions/runs/${runId}/attempts/${attempt}`); + if (run.repository?.full_name !== repository || run.event !== "pull_request" + || run.path !== ".github/workflows/pr.yml" || run.status !== "completed") { + throw new Error("Only completed main PR workflow attempts can be profiled"); + } + // GitHub often leaves workflow_run.pull_requests empty for fork PRs. Resolve + // association through the head commit, then validate repo and exact head. + const pulls = pages(`${repoPath}/commits/${run.head_sha}/pulls?per_page=100`); + const historical = args.includes("--historical"); + if (historical && process.env.GITHUB_ACTIONS) throw new Error("Historical export is a local-only operation"); + const ownerIds = historical ? [] : trustedOwnerIds(get); + if (process.env.GITHUB_ACTIONS) { + const exporterId = process.env.GITHUB_RUN_ID; + const exporterAttempt = process.env.GITHUB_RUN_ATTEMPT; + if (!/^\d+$/.test(exporterId) || !/^[1-9]\d*$/.test(exporterAttempt)) throw new Error("Invalid exporter run identity"); + const exporter = get(`${repoPath}/actions/runs/${exporterId}/attempts/${exporterAttempt}`); + if (!authorizedActors(exporter, ownerIds)) { + console.log("Exporter actor is not a CI CODEOWNER; no data exported."); + return; + } + } + const pr = historical ? undefined : associatedProfilePr(run, pulls, ownerIds); + if (!pr && !historical) { + console.log("No CODEOWNERS-authorized profile-ci PR at this revision; no data exported."); + return; + } + const jobs = pages(`${repoPath}/actions/runs/${runId}/attempts/${attempt}/jobs?per_page=100`, "jobs"); + const artifacts = pages(`${repoPath}/actions/runs/${runId}/artifacts?per_page=100`, "artifacts") + .filter((artifact) => artifact.name.startsWith(`ci-profile-${attempt}-`)); + // The run name snapshots selection at dispatch. A label added while an + // ordinary run finishes must not retroactively select that earlier run. + if (!historical && !run.display_title.startsWith("CI profile · PR #")) { + console.log("This attempt was not selected for profiling; no data exported."); + return; + } + const directory = mkdtempSync(path.join(tmpdir(), "paperclip-ci-export-")); + const records = []; + let bytes = 0; + try { + for (const artifact of artifacts) { + if (artifact.expired || artifact.size_in_bytes > maxBytes) throw new Error("Timing artifact is expired or too large"); + const archive = path.join(directory, `${artifact.id}.zip`); + writeFileSync(archive, gh([`${repoPath}/actions/artifacts/${artifact.id}/zip`])); + const decoded = execFileSync("python3", [fileURLToPath(new URL("./read-artifact.py", import.meta.url)), archive], { + encoding: "utf8", maxBuffer: maxBytes, timeout: 30_000, + }); + bytes += Buffer.byteLength(decoded); + if (bytes > 128 * 1024 * 1024) throw new Error("Run timing artifacts exceed 128 MiB; refusing to truncate the trace"); + for (const line of decoded.trim().split("\n").filter(Boolean)) records.push(JSON.parse(line)); + } + } finally { + rmSync(directory, { recursive: true, force: true }); + } + const trace = buildTrace(run, jobs, records); + trace.events[0].data["ci.profile_source"] = historical ? "historical" : "label"; + const dataset = process.env.HONEYCOMB_CI_DATASET || "paperclip-ci"; + const url = traceUrl(process.env.HONEYCOMB_TEAM || "paperclip", process.env.HONEYCOMB_ENVIRONMENT || "test", dataset, trace); + if (arg("--output")) writeFileSync(arg("--output"), JSON.stringify(trace, null, 2)); + if (!args.includes("--dry-run")) { + // Publish the completed root only after every child has been accepted. + // A failed partial export must not advertise complete coverage. + await sendEvents(trace.events.slice(1), process.env.HONEYCOMB_CI_API_KEY, dataset); + await sendEvents([trace.events[0]], process.env.HONEYCOMB_CI_API_KEY, dataset); + } + const summary = [ + `### CI profile${pr ? ` for PR #${pr.number}` : " (historical job/step baseline)"}`, + "", `[Open the full CI waterfall in Honeycomb](${url})`, "", + `${trace.jobCount} jobs · ${(trace.wallMs / 1000).toFixed(1)} seconds elapsed · ${trace.events.length} spans`, + "", `GitHub run: ${run.html_url}/attempts/${attempt}`, "", + ...trace.warnings.map((warning) => `- ${warning}`), + "", "The workflow root measures elapsed time. Parallel job durations must not be summed as elapsed time.", "", + ].join("\n"); + console.log(summary); + if (process.env.GITHUB_STEP_SUMMARY) appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary); +} + +if (process.argv[1] && path.resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + main().catch((error) => { console.error(error.message); process.exitCode = 1; }); +} diff --git a/scripts/ci-profile/initialize.mjs b/scripts/ci-profile/initialize.mjs new file mode 100644 index 00000000000..f1895eb43b2 --- /dev/null +++ b/scripts/ci-profile/initialize.mjs @@ -0,0 +1,17 @@ +import { recorder } from "./record.mjs"; +import { appendFileSync } from "node:fs"; +import { createHash } from "node:crypto"; + +recorder("profile")({ kind: "job", start: Date.now() }); + +if (process.env.PAPERCLIP_CI_PROFILE_JOB === "policy" && process.env.GITHUB_STEP_SUMMARY) { + const key = `${process.env.GITHUB_REPOSITORY}:${process.env.GITHUB_RUN_ID}:${process.env.GITHUB_RUN_ATTEMPT}`; + const traceId = createHash("sha256").update(key).digest("hex").slice(0, 32); + // This job does not know when the last parallel lane will finish. A bounded + // search window includes queued jobs; the exporter writes the exact range. + const now = Math.floor(Date.now() / 1000); + const params = new URLSearchParams({ trace_id: traceId, trace_start_ts: String(now - 86400), trace_end_ts: String(now + 86400) }); + appendFileSync(process.env.GITHUB_STEP_SUMMARY, + `### Full CI profile\n\n[Open the Honeycomb waterfall](https://ui.honeycomb.io/paperclip/environments/test/datasets/paperclip-ci/trace?${params})\n\n` + + "The trace becomes available after this attempt completes and the CI profile exporter finishes. Keep the profile-ci label on the PR until then.\n"); +} diff --git a/scripts/ci-profile/node-reporter.mjs b/scripts/ci-profile/node-reporter.mjs new file mode 100644 index 00000000000..7612c25300d --- /dev/null +++ b/scripts/ci-profile/node-reporter.mjs @@ -0,0 +1,41 @@ +import path from "node:path"; +import { recorder, relativeFile } from "./record.mjs"; + +export default async function* reporter(source) { + const record = recorder("node"); + record({ kind: "start" }); + let index = 0; + const files = new Map(); + for await (const event of source) { + // complete is emitted as execution finishes; pass/fail are buffered in + // declaration order and cannot be used to place parallel tests in time. + if (event.type !== "test:complete") continue; + const data = event.data; + const duration = data.details?.duration_ms; + if (!data.file || !Number.isFinite(duration)) continue; + const file = relativeFile(data.file); + const end = Date.now(); + const start = end - duration; + const previous = files.get(file); + files.set(file, { + start: Math.min(previous?.start ?? start, start), + end: Math.max(previous?.end ?? end, end), + failed: previous?.failed || (!data.skip && !data.details.passed), + }); + // File completion events may use the CLI's relative argument as their + // name while data.file is absolute. They are containers, not test cases. + if (path.resolve(data.name) === path.resolve(data.file) || data.details.type === "suite") continue; + record({ + kind: "test", parent: file, + id: String(index++), name: data.name, file, start, duration, + status: data.skip ? "skipped" : data.details.passed ? "passed" : "failed", + clock: "reporter-receipt", + }); + } + for (const [file, timing] of files) { + record({ kind: "file", id: file, name: file, file, + start: timing.start, duration: timing.end - timing.start, + status: timing.failed ? "failed" : "passed", clock: "reporter-receipt" }); + } + record({ kind: "end" }); +} diff --git a/scripts/ci-profile/node-test.mjs b/scripts/ci-profile/node-test.mjs new file mode 100644 index 00000000000..5b542e33469 --- /dev/null +++ b/scripts/ci-profile/node-test.mjs @@ -0,0 +1,16 @@ +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; +import { invocationEnv } from "./record.mjs"; + +const args = ["--test"]; +if (process.env.PAPERCLIP_CI_PROFILE === "true") { + args.push("--test-reporter=spec", "--test-reporter-destination=stdout", + `--test-reporter=${fileURLToPath(new URL("./node-reporter.mjs", import.meta.url))}`, + "--test-reporter-destination=stdout"); +} +const result = spawnSync(process.execPath, [...args, ...process.argv.slice(2)], { + stdio: "inherit", env: invocationEnv("node"), +}); +if (result.error) console.error(result.error.message); +if (result.signal) process.kill(process.pid, result.signal); +else process.exitCode = result.status ?? 1; diff --git a/scripts/ci-profile/playwright-reporter.mjs b/scripts/ci-profile/playwright-reporter.mjs new file mode 100644 index 00000000000..148a1b5c2e9 --- /dev/null +++ b/scripts/ci-profile/playwright-reporter.mjs @@ -0,0 +1,32 @@ +import { recorder, relativeFile } from "./record.mjs"; + +export default class CiProfileReporter { + record = recorder("playwright"); + files = new Map(); + onBegin() { this.record({ kind: "start" }); } + onTestEnd(test, result) { + const file = relativeFile(test.location.file); + const start = result.startTime.getTime(); + const end = start + result.duration; + const parent = `${test.parent.project()?.name || ""}:${file}`; + const previous = this.files.get(parent); + this.files.set(parent, { + kind: "file", id: parent, name: file, file, + start: Math.min(previous?.start ?? start, start), + end: Math.max(previous?.end ?? end, end), + status: previous?.status === "failed" || ["failed", "timedOut"].includes(result.status) + ? "failed" : "passed", + }); + this.record({ + kind: "test", id: `${test.id}:${result.retry}`, parent, + name: test.titlePath().slice(1).join(" > "), file, + start, duration: result.duration, status: result.status, retries: result.retry, + }); + } + onEnd() { + for (const { end, ...file } of this.files.values()) { + this.record({ ...file, duration: end - file.start }); + } + this.record({ kind: "end" }); + } +} diff --git a/scripts/ci-profile/read-artifact.py b/scripts/ci-profile/read-artifact.py new file mode 100644 index 00000000000..7887d6720c8 --- /dev/null +++ b/scripts/ci-profile/read-artifact.py @@ -0,0 +1,23 @@ +"""Read timing JSON from an untrusted PR artifact without extracting any files.""" +import json +import sys +import zipfile + +MAX_BYTES = 32 * 1024 * 1024 +with zipfile.ZipFile(sys.argv[1]) as archive: + entries = archive.infolist() + if len(entries) > 5000 or sum(item.file_size for item in entries) > MAX_BYTES: + raise ValueError("CI timing artifact exceeds the supported size") + for item in entries: + if item.is_dir(): + continue + if not item.filename.endswith(".ndjson"): + raise ValueError("CI timing artifact contains an unexpected file") + with archive.open(item) as stream: + for line in stream: + if len(line) > 32 * 1024: + raise ValueError("CI timing record exceeds the supported size") + value = json.loads(line) + if not isinstance(value, dict): + raise ValueError("CI timing record must be an object") + print(json.dumps(value, separators=(",", ":"))) diff --git a/scripts/ci-profile/record.mjs b/scripts/ci-profile/record.mjs new file mode 100644 index 00000000000..0b8f9ee68f3 --- /dev/null +++ b/scripts/ci-profile/record.mjs @@ -0,0 +1,44 @@ +import { appendFileSync, mkdirSync } from "node:fs"; +import path from "node:path"; +import { randomUUID } from "node:crypto"; + +// PR processes write timing data only. They never receive a Honeycomb key. +export function recorder(framework, environment = process.env) { + const enabled = environment.PAPERCLIP_CI_PROFILE === "true"; + const directory = environment.PAPERCLIP_CI_PROFILE_DIR; + const invocation = environment.PAPERCLIP_CI_PROFILE_INVOCATION || randomUUID(); + const file = directory && path.join(directory, `${framework}-${invocation}.ndjson`); + let warned = false; + return (record) => { + if (!enabled || !file) return; + try { + mkdirSync(directory, { recursive: true }); + appendFileSync(file, JSON.stringify({ + version: 1, + framework, + invocation, + job: environment.PAPERCLIP_CI_PROFILE_JOB, + runId: environment.GITHUB_RUN_ID, + attempt: environment.GITHUB_RUN_ATTEMPT, + ...record, + }) + "\n"); + } catch { + // Profiling must not change a test's result. The exporter reports gaps. + if (!warned) console.error("CI profile: unable to write test timing data"); + warned = true; + } + }; +} + +// The launcher records intent before starting the test runner. A reporter that +// never loads or never writes anything must still leave a detectable gap. +export function invocationEnv(framework, environment = process.env) { + if (environment.PAPERCLIP_CI_PROFILE !== "true") return environment; + const env = { ...environment, PAPERCLIP_CI_PROFILE_INVOCATION: randomUUID() }; + recorder(framework, env)({ kind: "expected" }); + return env; +} + +export function relativeFile(file) { + return path.relative(process.env.GITHUB_WORKSPACE || process.cwd(), file).replaceAll("\\", "/"); +} diff --git a/scripts/ci-profile/reporters.test.mjs b/scripts/ci-profile/reporters.test.mjs new file mode 100644 index 00000000000..15bf6f95da6 --- /dev/null +++ b/scripts/ci-profile/reporters.test.mjs @@ -0,0 +1,73 @@ +import assert from "node:assert/strict"; +import { mkdtempSync, readFileSync, readdirSync, realpathSync, rmSync, symlinkSync, writeFileSync } from "node:fs"; +import { spawnSync } from "node:child_process"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { fileURLToPath } from "node:url"; +import test from "node:test"; + +const root = fileURLToPath(new URL("../../", import.meta.url)); +const scripts = path.join(root, "scripts/ci-profile"); + +for (const framework of ["vitest", "playwright", "node"]) { + test(`${framework} reports every file with its own child tests and preserves failing exit codes`, () => { + const directory = realpathSync(mkdtempSync(path.join(tmpdir(), "ci-profile-smoke-"))); + try { + symlinkSync(path.join(root, "node_modules"), path.join(directory, "node_modules"), "dir"); + const output = path.join(directory, "timings"); + const fixture = framework === "node" + ? 'import { test } from "node:test";' + : framework === "vitest" ? 'import { test } from "vitest";' : 'import { test } from "@playwright/test";'; + const filenames = ["probe.test.mjs", "second.test.mjs"]; + for (const filename of filenames) writeFileSync(path.join(directory, filename), `${fixture} +test("passing", async () => { await new Promise(resolve => setTimeout(resolve, 20)); }); +test("failing", async () => { throw new Error("expected probe failure"); }); +`); + let args; + if (framework === "node") { + args = [path.join(scripts, "node-test.mjs"), ...filenames]; + } else { + const config = framework === "vitest" + ? { test: { include: ["*.test.mjs"], reporters: [path.join(scripts, "vitest-reporter.mjs")] } } + : { testDir: directory, testMatch: "*.test.mjs", workers: 2, retries: 0, reporter: [[path.join(scripts, "playwright-reporter.mjs")]] }; + const configFile = path.join(directory, `${framework}.config.mjs`); + writeFileSync(configFile, `export default ${JSON.stringify(config)};`); + args = framework === "vitest" + ? [path.join(root, "node_modules/vitest/vitest.mjs"), "run", "--config", configFile, "--root", directory] + : [path.join(root, "node_modules/@playwright/test/cli.js"), "test", "--config", configFile]; + args = [path.join(scripts, "run.mjs"), framework, process.execPath, ...args]; + } + const start = Date.now(); + const env = { ...process.env }; + delete env.NODE_TEST_CONTEXT; + const result = spawnSync(process.execPath, args, { + cwd: directory, encoding: "utf8", timeout: 30_000, + env: { ...env, PAPERCLIP_CI_PROFILE: "true", PAPERCLIP_CI_PROFILE_DIR: output, + PAPERCLIP_CI_PROFILE_JOB: "probe", GITHUB_RUN_ID: "1", GITHUB_RUN_ATTEMPT: "1", GITHUB_WORKSPACE: directory }, + }); + const end = Date.now(); + assert.equal(result.status, 1, result.stdout + result.stderr); + const records = readdirSync(output).flatMap((file) => readFileSync(path.join(output, file), "utf8").trim().split("\n").map(JSON.parse)); + const cases = records.filter((record) => record.kind === "test"); + assert.equal(cases.length, 4, JSON.stringify(records)); + assert.ok(cases.some((record) => record.status === "passed")); + assert.ok(cases.some((record) => record.status === "failed")); + for (const record of cases) { + assert.ok(record.start >= start && record.start <= end, JSON.stringify(record)); + assert.ok(record.duration >= 0 && record.start + record.duration <= end + 10); + assert.ok(filenames.includes(record.file)); + } + assert.ok(records.some((record) => record.kind === "end")); + const expected = records.filter((record) => record.kind === "expected"); + assert.equal(expected.length, 1); + assert.ok(records.every((record) => record.invocation === expected[0].invocation)); + const files = records.filter((record) => record.kind === "file"); + assert.deepEqual(files.map((record) => record.name).sort(), filenames); + for (const file of files) { + const children = cases.filter((record) => record.parent === file.id); + assert.equal(children.length, 2); + assert.ok(children.every((record) => record.file === file.file)); + } + } finally { rmSync(directory, { recursive: true, force: true }); } + }); +} diff --git a/scripts/ci-profile/run.mjs b/scripts/ci-profile/run.mjs new file mode 100644 index 00000000000..c5272b69419 --- /dev/null +++ b/scripts/ci-profile/run.mjs @@ -0,0 +1,9 @@ +import { spawnSync } from "node:child_process"; +import { invocationEnv } from "./record.mjs"; + +const [framework, command, ...args] = process.argv.slice(2); +if (!["vitest", "playwright"].includes(framework) || !command) throw new Error("Provide a framework and test command"); +const result = spawnSync(command, args, { stdio: "inherit", env: invocationEnv(framework) }); +if (result.error) console.error(result.error.message); +if (result.signal) process.kill(process.pid, result.signal); +else process.exitCode = result.status ?? 1; diff --git a/scripts/ci-profile/trace.mjs b/scripts/ci-profile/trace.mjs new file mode 100644 index 00000000000..99beeabad04 --- /dev/null +++ b/scripts/ci-profile/trace.mjs @@ -0,0 +1,176 @@ +import { createHash } from "node:crypto"; + +const millis = (value) => Date.parse(value); +const id = (value, size = 16) => createHash("sha256").update(value).digest("hex").slice(0, size); +const failed = (status) => ["failure", "failed", "timed_out", "timedOut", "cancelled", "interrupted"].includes(status); +const text = (value) => typeof value === "string" && value.length <= 4096; + +export function buildTrace(run, jobs, records = []) { + if (run.status !== "completed" || !jobs.length || jobs.some((job) => job.status !== "completed")) { + throw new Error("A complete workflow attempt and its full job inventory are required"); + } + if (jobs.some((job) => job.run_id !== run.id || job.run_attempt !== run.run_attempt)) { + throw new Error("Job inventory contains another run or attempt"); + } + const key = `${run.repository.full_name}:${run.id}:${run.run_attempt}`; + const traceId = id(key, 32); + const rootId = id(`${key}:root`); + const start = millis(run.run_attempt === 1 ? run.created_at : run.run_started_at); + // The attempt jobs API can relabel carried-over successful jobs with the + // new attempt number. Their completion clocks still precede this attempt. + const carriedJobs = run.run_attempt > 1 ? jobs.filter((job) => millis(job.completed_at) < start) : []; + jobs = jobs.filter((job) => !carriedJobs.includes(job)); + if (!jobs.length) throw new Error("No jobs executed in this workflow attempt"); + const end = Math.max(...jobs.map((job) => millis(job.completed_at)).filter(Number.isFinite)); + if (!Number.isFinite(start) || !Number.isFinite(end) || end < start) throw new Error("Invalid run timing"); + const events = []; + const warnings = []; + if (carriedJobs.length) warnings.push(`${carriedJobs.length} completed jobs carried over from earlier attempts were excluded`); + const shared = { + "service.name": "paperclip-ci", + "trace.trace_id": traceId, + "ci.repository": run.repository.full_name, + "ci.run_id": String(run.id), + "ci.attempt": run.run_attempt, + "ci.commit": run.head_sha, + "ci.branch": run.head_branch, + "ci.url": run.html_url, + }; + const span = (spanId, parent, name, from, to, fields) => { + if (!Number.isFinite(from) || !Number.isFinite(to) || to < from) return; + events.push({ + time: new Date(from).toISOString(), samplerate: 1, + data: { ...shared, "trace.span_id": spanId, ...(parent ? { "trace.parent_id": parent } : {}), + name, duration_ms: to - from, ...fields }, + }); + }; + span(rootId, null, `PR CI #${run.run_number || run.id} · attempt ${run.run_attempt}`, start, end, { + "ci.kind": "workflow", "ci.status": run.conclusion, error: failed(run.conclusion), + "ci.job_count": jobs.length, + "ci.carried_job_count": carriedJobs.length, + "ci.wall_ms": end - start, + // A partial rerun intentionally does not re-execute successful jobs. + "ci.partial_rerun": run.run_attempt > 1, + }); + const roots = new Map(); + const byName = new Map(); + for (const job of jobs) { + const jobId = id(`${key}:job:${job.id}`); + roots.set(job.id, jobId); + byName.set(job.name.replace(/^ci \/ /, ""), job); + const created = Math.max(start, millis(job.created_at) || start); + const started = Math.max(created, millis(job.started_at) || created); + const completed = Math.max(started, millis(job.completed_at) || started); + // Include allocation wait in the job container, then distinguish it from + // execution. Dependency wait may precede job creation; it stays visible as + // a gap from the workflow start. GitHub does not expose a separate readyAt. + span(jobId, rootId, job.name, Math.min(created, started), completed, { + "ci.kind": "job", "ci.job": job.name, "ci.job_id": String(job.id), + "ci.status": job.conclusion, error: failed(job.conclusion), + "ci.runner": job.runner_name, "ci.runner_labels": (job.labels || []).join(","), + "ci.start_offset_ms": started - start, + "ci.execution_ms": Math.max(0, completed - started), + "ci.finishes_last": completed === end, + "ci.job_url": job.html_url, + }); + if (started > created) span(id(`${jobId}:wait`), jobId, "Waiting for runner / scheduling", created, started, { + "ci.kind": "wait", "ci.job": job.name, + }); + for (const step of job.steps || []) { + const from = millis(step.started_at); + const to = millis(step.completed_at); + if (from < start || to > end) continue; // skipped steps may use epoch timestamps + span(id(`${jobId}:step:${step.number}`), jobId, step.name, from, to, { + "ci.kind": "step", "ci.job": job.name, "ci.step": step.name, + "ci.status": step.conclusion, error: failed(step.conclusion), + }); + } + } + + const receipts = new Set(); + const invocations = new Map(); + const valid = []; + let rejected = 0; + for (const record of records) { + const job = byName.get(record?.job); + if (!job || record.version !== 1 || String(record.runId) !== String(run.id) + || String(record.attempt) !== String(run.run_attempt) || !text(record.invocation)) { + rejected++; continue; + } + if (record.kind === "job") { receipts.add(job.id); continue; } + const invocation = `${job.id}:${record.framework}:${record.invocation}`; + if (["expected", "start", "end"].includes(record.kind)) { + const state = invocations.get(invocation) || {}; + state[record.kind] = true; + invocations.set(invocation, state); + continue; + } + if (!["file", "test"].includes(record.kind) || !text(record.id) || !text(record.name) + || !text(record.file) || !["vitest", "playwright", "node"].includes(record.framework) + || !Number.isFinite(record.start) || !Number.isFinite(record.duration) || record.duration < 0 + || record.start < millis(job.started_at) - 1000 + || record.start + record.duration > millis(job.completed_at) + 1000) { + rejected++; continue; + } + valid.push({ record, job, invocation }); + } + const recordId = ({ record, invocation }) => id(`${key}:${invocation}:${record.kind}:${record.id}`); + const files = new Map(valid.filter(({ record }) => record.kind === "file") + .map((entry) => [`${entry.invocation}:${entry.record.id}`, recordId(entry)])); + for (const entry of valid) { + const { record, job, invocation } = entry; + const jobId = roots.get(job.id); + // GitHub steps are ordered and have second-precision timestamps. Prefer + // the latest containing step: rounding may also make a preceding step fit + // a short file/test. Preserve the runner's own clock. + const step = job.steps?.findLast((candidate) => record.start >= millis(candidate.started_at) + && record.start + record.duration <= millis(candidate.completed_at) + 1000 + && candidate.conclusion !== "skipped"); + const parent = files.get(`${invocation}:${record.parent}`) + || (step ? id(`${jobId}:step:${step.number}`) : jobId); + span(recordId(entry), parent, record.name, record.start, record.start + record.duration, { + "ci.kind": record.kind, "ci.job": job.name, "test.framework": record.framework, + "test.file": record.file, "test.status": typeof record.status === "string" ? record.status.slice(0, 32) : "unknown", error: failed(record.status), + "test.retries": Number.isFinite(record.retries) ? record.retries : 0, + "test.clock": record.clock === "reporter-receipt" ? "reporter-receipt" : "runner", + ...(Number.isFinite(record.collectMs) ? { "test.collect_ms": record.collectMs } : {}), + ...(Number.isFinite(record.setupMs) ? { "test.setup_ms": record.setupMs } : {}), + }); + } + const expectedReceipts = jobs.filter((job) => job.steps?.some((step) => step.name === "Initialize CI profile")); + const missing = expectedReceipts.filter((job) => !receipts.has(job.id)); + if (missing.length) warnings.push(`Missing timing artifacts for ${missing.length} jobs`); + const incomplete = [...invocations.values()].filter((value) => !value.expected || !value.start || !value.end).length; + if (incomplete) warnings.push(`${incomplete} test invocations did not finish reporting`); + if (rejected) warnings.push(`${rejected} invalid timing records ignored`); + if (run.conclusion !== "success") warnings.push("Workflow did not succeed; later tests or steps may not have run"); + if (!expectedReceipts.length) warnings.push("Job/step timings only; no test reporter collection steps were present"); + const root = events[0].data; + const expectedFramework = (name) => { + if (/General tests|Verify serialized server suites|Verify Paperclip Runner \(vitest/.test(name)) return "vitest"; + if (/e2e shard/.test(name)) return "playwright"; + if (/\/ policy$|Typecheck \+ Release Registry|Verify Paperclip Runner \(static checks/.test(name)) return "node"; + return null; + }; + const missingTests = jobs.filter((job) => { + const framework = expectedFramework(job.name); + return framework && !valid.some((entry) => entry.job.id === job.id && entry.record.framework === framework); + }); + if (expectedReceipts.length && missingTests.length) warnings.push(`Missing test detail for ${missingTests.length} test jobs (including jobs that failed before testing)`); + root["ci.test_count"] = valid.filter(({ record }) => record.kind === "test").length; + root["ci.file_count"] = valid.filter(({ record }) => record.kind === "file").length; + root["ci.expected_invocations"] = [...invocations.values()].filter((value) => value.expected).length; + root["ci.reported_invocations"] = [...invocations.values()].filter((value) => value.expected && value.start && value.end).length; + root["ci.profile_complete"] = warnings.length === 0 && run.run_attempt === 1; + root["ci.coverage_warnings"] = warnings.join("; "); + return { traceId, events, warnings, wallMs: end - start, jobCount: jobs.length }; +} + +export function traceUrl(team, environment, dataset, trace) { + const params = new URLSearchParams({ + trace_id: trace.traceId, + trace_start_ts: String(Math.floor(Date.parse(trace.events[0].time) / 1000)), + trace_end_ts: String(Math.ceil((Date.parse(trace.events[0].time) + trace.wallMs) / 1000)), + }); + return `https://ui.honeycomb.io/${encodeURIComponent(team)}/environments/${encodeURIComponent(environment)}/datasets/${encodeURIComponent(dataset)}/trace?${params}`; +} diff --git a/scripts/ci-profile/vitest-reporter.mjs b/scripts/ci-profile/vitest-reporter.mjs new file mode 100644 index 00000000000..b4a2fb54461 --- /dev/null +++ b/scripts/ci-profile/vitest-reporter.mjs @@ -0,0 +1,31 @@ +import { recorder, relativeFile } from "./record.mjs"; +import { experimental_getRunnerTask } from "vitest/node"; + +export default class CiProfileReporter { + record = recorder("vitest"); + onTestRunStart() { this.record({ kind: "start" }); } + onTestRunEnd() { this.record({ kind: "end" }); } + onTestCaseResult(test) { + const timing = test.diagnostic(); + if (!timing?.startTime) return; + this.record({ + kind: "test", id: test.id, parent: test.module.id, + name: test.fullName, file: relativeFile(test.module.moduleId), + start: timing.startTime, duration: timing.duration, + status: test.result().state, retries: timing.retryCount, + }); + } + onTestModuleEnd(module) { + const timing = module.diagnostic(); + // ModuleDiagnostic has no startTime in Vitest 5. Use the worker's + // timestamp, not the time the controller receives this reporter event. + const start = experimental_getRunnerTask(module).result?.startTime; + if (!start) return; + this.record({ + kind: "file", id: module.id, name: relativeFile(module.moduleId), + file: relativeFile(module.moduleId), start, + duration: timing.duration, status: module.state(), + collectMs: timing.collectDuration, setupMs: timing.setupDuration, + }); + } +} diff --git a/scripts/run-vitest-stable.mjs b/scripts/run-vitest-stable.mjs index a62f90094e1..56cb08b3fbb 100644 --- a/scripts/run-vitest-stable.mjs +++ b/scripts/run-vitest-stable.mjs @@ -1,10 +1,11 @@ #!/usr/bin/env node import { spawnSync } from "node:child_process"; -import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync } from "node:fs"; +import { mkdirSync, mkdtempSync, readdirSync, readFileSync, realpathSync, statSync, writeFileSync } from "node:fs"; import os from "node:os"; import path from "node:path"; import { fileURLToPath } from "node:url"; import { loadShardDurations, selectGeneralServerShard } from "./general-server-shard.mjs"; +import { invocationEnv } from "./ci-profile/record.mjs"; import { assertSelectedTests, partitionTestLines } from "./test-line-shard.mjs"; @@ -435,7 +436,7 @@ function runVitest(args, label, testShard = null) { } const result = spawnSync("pnpm", ["exec", "vitest", "run", ...sourceOnlyVitestArgs, ...args], { cwd: repoRoot, - env, + env: invocationEnv("vitest", env), stdio: "inherit", }); if (result.error) { @@ -598,7 +599,8 @@ if (options.dryRun) { options.mode === serializedModeName ? selectSerializedSuites(routeTests, options.shardIndex, options.shardCount) : routeTests; - console.log( + // The inventory exceeds pipe buffers; flush it before the immediate exit. + writeFileSync(1, JSON.stringify( { mode: options.mode, diff --git a/tests/e2e/playwright.config.ts b/tests/e2e/playwright.config.ts index 079c9166df9..ff52aaa342d 100644 --- a/tests/e2e/playwright.config.ts +++ b/tests/e2e/playwright.config.ts @@ -95,5 +95,10 @@ export default defineConfig({ }, }, outputDir: "./test-results", - reporter: [["list"], ["html", { open: "never", outputFolder: "./playwright-report" }]], + reporter: [ + ["list"], ["html", { open: "never", outputFolder: "./playwright-report" }], + ...(process.env.PAPERCLIP_CI_PROFILE === "true" + ? [[path.resolve(import.meta.dirname, "../../scripts/ci-profile/playwright-reporter.mjs")] as [string]] + : []), + ], }); diff --git a/vitest.config.ts b/vitest.config.ts index d2d9e52e12e..d91485c5935 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -1,7 +1,10 @@ -import { defineConfig } from "vitest/config"; +import { configDefaults, defineConfig } from "vitest/config"; export default defineConfig({ test: { + ...(process.env.PAPERCLIP_CI_PROFILE === "true" + ? { reporters: [...configDefaults.reporters, "./scripts/ci-profile/vitest-reporter.mjs"] } + : {}), projects: [ "packages/shared", "packages/skills-catalog", From 7f9d3d1929ce08a494a31472185ebc85d7219085 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Thu, 15 Oct 2026 07:10:35 -0500 Subject: [PATCH 4/4] fix(grok): avoid routine wakes for proactive admission (#16422) ## Thinking Path > - Paperclip lets people manage AI agents and their work. > - Grok Bot can request governed work from its own conversation. > - Managed Runner admission can take longer than the initial MCP response. > - The dispatcher treated a delayed offer as a reason to start another routine. > - The original conversation was already waiting for that same work. > - This pull request keeps admission in its original request and cancels old prepared echoes. > - Users get one task interaction while normal assigned-task wakes remain available. ## Linked Issues or Issue Description Refs #16219 and #16394. The duplicate search found no other proactive Grok correction. **What happened?** A real idle Grok conversation created one Paperclip task and reached accepted completion. During admission, the dispatcher also sent a routine webhook after the offer was ten seconds old. Two acceptance requests and two title writes were recorded. The extra routine can compete with the original conversation. **Expected behavior** Immediate and delayed Bot-initiated admission stay with the original request. The Bot checks the same request ID and arguments until admission succeeds or fails. A new routine must not echo that work back to the Bot. **Steps to reproduce** 1. Ask an idle Grok Bot to create and complete a Paperclip task. 2. Let managed admission return pending. 3. Leave its offered assignment unaccepted for more than ten seconds. 4. Run the dispatcher. It sends another routine webhook for that proactive offer. **Paperclip version or commit** Observed through the actual Grok client on Core 11d69d04c7531cf7eb37f7994b7112797e145837. Both new wake regressions also fail on current master before this correction. ## What Changed - Remove the delayed proactive-admission wake scan. - Fence prepared proactive wakes using the current binding and generation. A dispatcher restart cancels those old intents without sending them. - Tell Grok to keep checking its original queued request. Preserve Dot event instructions and normal assigned-task dispatch. - Add delayed-offer and recovered-intent regressions. Update the agent guide and release evidence plan. ## Verification - Two new network regressions fail before the fix. The corrected Grok and Dot suites pass all 80 checks. - The full focused Grok, Dot, MCP authorization/shutdown and question-delivery selection passes 214 checks across six files. - Repository typecheck and build pass. Token gates, the 2,731-story structural catalog and Storybook build pass. All these checks also pass after the branch was rebased onto master 250429942e27d568f20d3bc7dddcfc84a666480a. - The real-client investigation created one task and one managed run, saved a 100-word report, reached canonical Done and recorded successful lease cleanup. Its exact public source marker was absent from the operator prompt, and no Paperclip proxy operation fetched it. The no-echo condition failed on that source; this PR still requires a deployed real-client repeat. - Public synthetic marker for the deployed repeat: `grok-native-read-8731cd8120b140829fbe476f6974fde8`. - Current-head CI passes with 55 successful checks and two optional Storybook skips. Greptile reviewed e6b828af28e9202469c0d00f9f4dceac791411a2 at 5/5 with no findings. - The first workspace CI attempt hit a 5,000ms timeout in the unchanged snapshot-worker command-preparation test. All four variants passed locally; one targeted remote retry passed. The initial failure is retained and its root cause is not established. ## Risks Grok must keep checking the original request while admission is pending. If that conversation ends, the durable request remains available for reconciliation; Paperclip cannot infer provider idleness from a delay. Prepared intents from older dispatchers are cancelled by the authority guard. The real client must verify the polling path after deployment. No schema migration is needed. ## Model Used OpenAI GPT-6 in Codex, with reasoning, repository editing, shell tools and computer use. The exact serving model ID, snapshot and context window are not exposed by this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip --- doc/grok-bot-agent.md | 7 ++++-- ...026-10-13-grok-bot-production-readiness.md | 14 +++++++----- server/src/__tests__/dot-runner.test.ts | 3 +++ server/src/__tests__/grok-bot-routine.test.ts | 22 +++++++++++++++++++ server/src/services/dot-runner-broker.ts | 19 +++++++++++++--- server/src/services/grok-bot/dispatcher.ts | 18 +++++---------- 6 files changed, 60 insertions(+), 23 deletions(-) diff --git a/doc/grok-bot-agent.md b/doc/grok-bot-agent.md index 952fbace5a0..ebea2c20524 100644 --- a/doc/grok-bot-agent.md +++ b/doc/grok-bot-agent.md @@ -29,8 +29,11 @@ paste it into Paperclip's authorization page. Keep it out of Bot chat. Assign ordinary tasks to the agent. Read its progress, answer its question cards, and review its documents and files in the task thread. The Bot can initiate a new task from its Grok conversation through the admitted work-session tools. -An immediate proactive claim does not send the same assignment back through the -routine. +The originating conversation checks the same request while admission is queued. +Neither immediate nor delayed Bot-initiated admission sends that assignment back +through the routine. Grok keeps the request ID stable until admission succeeds +or fails. Human-assigned work and continuations after an answered question still +use their normal assignment wakes. ## Files and completion diff --git a/doc/plans/2026-10-13-grok-bot-production-readiness.md b/doc/plans/2026-10-13-grok-bot-production-readiness.md index e5c3697dfaf..05de9675f3e 100644 --- a/doc/plans/2026-10-13-grok-bot-production-readiness.md +++ b/doc/plans/2026-10-13-grok-bot-production-readiness.md @@ -1,6 +1,6 @@ # Grok Bot production readiness -Status as of October 15: a real assigned task now reaches canonical completion. Qualification found an active-session wake hazard; its correction is implemented and passing local checks. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested. +Status as of October 15: real assigned and Bot-initiated tasks reach canonical completion. The active-session wake correction is merged. A real proactive test found a delayed-admission echo; its correction is under verification. Production acceptance is incomplete. The current release goal and matrix below supersede the earlier dated campaign entries. Historical successes and failures remain evidence for the revisions actually tested. ## Current production finish line — October 14 @@ -12,12 +12,12 @@ There is no testing call, case-count, token or spending cap. The former six-case | --- | --- | --- | | Ordinary onboarding works | Actual Grok UI, fixed agent consent, saved setup, wrong-key behavior, authenticated challenge, refresh, reconnect and revocation | Resumed real Grok authorization, saved setup Done, authenticated routine challenge and automatic refresh passed on the October 15 serving revision. Runtime reconnect copy fix is merged; verify its published UI and fresh production setup next. | | Assigned work reaches a usable result | One actual managed workflow: progress, question and answer, document, downloaded file/hash, canonical completion or review and cleanup | A preliminary comparison task on serving Core ff0 completed progress → question → answer → report/CSV → canonical Done, with both cleanup receipts. The cited CSV downloads with the exact publication hash. Two distinct CSV uploads accompanied an active-session result wake; correct that wake behavior and requalify the final candidate. | -| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | Unverified. Run after the assigned-work gate. | +| Bot initiates governed work | Real-client task creation/claim/completion with stable receipts and no outbound echo or duplicate effects | A real idle-conversation request created one task and one managed run, saved a 100-word report, reached accepted Done and cleaned up its lease. A delayed-admission wake and two acceptance attempts were also recorded. Functional completion passed; no-echo acceptance failed. Requalify the correction. | | Recovery and authority are reliable | Healthy work past the prior five-minute failure window; actual controller restart preserving assignment/session/process and no duplicate wake; Stop, pause, reassignment, revocation, stale calls, queues and cleanup | The old hold was inspected and cleared. The successful comparison continuation ended below five minutes. Healthy work beyond that window, an actual restart and active Stop remain unverified; protocol tests are separate evidence. | -| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | The active-session wake correction follows origin/master. Initial local checks used d6e4b36c8c049b85813caf3e3cbf1f8d87e4e068; CI and review must verify the final PR head. Three regressions failed before the fix. Focused tests, repository typecheck/build, token gates and Storybook checks pass. Full CI, review and publication of this correction remain required; prior candidate receipts are historical. | +| One identifiable reviewed candidate passes source gates | Focused protocol/auth/Runner/UI tests, full CI/typecheck/tests/build/token/Storybook evidence and exact artifacts | Active-session wake correction #16394 merged with 55 passing checks, two optional skips and review 5/5. The proactive echo correction started from master 7fc2dc8b3236e1149ae32756b6153a503623404b and was rebased onto freshly fetched master 250429942e27d568f20d3bc7dddcfc84a666480a before submission. Two regression cases reproduce the real delayed-admission wake before the fix. All 214 focused protocol/auth/question checks pass, including 80 Grok and Dot checks. Repository typecheck/build, token gates, the 2,731-story structure and Storybook build pass. Fresh CI, review, publication and the real no-echo repeat remain required. | | Production opt-in is delivered and smoke-tested | Staging acceptance, correct served Core/distribution/Runner, normal editable opt-in controls/defaults, actual production Bot workflow and inspectable task/output links | Cloud opt-in policy and release-guide corrections are merged. Staging Cloud provenance and minimal feature values were verified, with Dot disabled and defaults unchanged. The corrected wake candidate, remaining real-client cases, ordinary production opt-in controls and production smoke are outstanding. | -The critical path is to review and publish the active-session wake correction, verify its served identity, then complete the remaining real-client acceptance cases. The selected staging workspace remains the test destination. Preserve one source manifest per case; do not combine incompatible runs into one green release. +The critical path is to review and publish the proactive-admission echo correction, verify its served identity, then complete the remaining real-client acceptance cases. The selected staging workspace remains the test destination. Preserve one source manifest per case; do not combine incompatible runs into one green release. Real Grok reconnect restored the same agent, plugin and webhook-only routine. Saved setup reached Done after both delivery and authenticated MCP acknowledgement. After the initial access token expired, Grok refreshed it and acknowledged another challenge without reauthorization. Grok/MCP/Runner were enabled with Dot disabled; setup allocated no managed Runner. The operator used the existing saved-invitation path to get the secure pairing-code copy control before the Runtime copy correction published. Record that as assisted recovery, not proof of unassisted fresh onboarding. The copy correction is merged in #16373 after 55 passing CI checks, two optional skips and review 5/5. @@ -27,7 +27,11 @@ An unread asynchronous tool result also caused a webhook during accepted work. T Core d73d7a8651db0897cb50adbca42731bcb8aed550 and its canonical private Cloud composition passed publication. One single-workspace request passed preparation but was blocked by an existing shared rollout; it created no campaign. The existing rollout later released the reservation. That prepared source is now superseded by the wake correction. Keep its receipts as history and do not resume its older deployment request. Select a new immutable reviewed source after this fix passes CI and review. Production opt-in policy and release-guide changes are merged, but production rollout and smoke remain outstanding. -Desktop control stopped responding after the successful exchanges. Resetting the tool did not restore it, and the user has been asked to reopen Codex. Browser control and webhook/MCP work remain usable; native Bot-initiated prompts and fresh production setup require restored desktop control or ordinary human setup actions. The dedicated production workspace has no qualification agent grant or smoke task. +Desktop control recovered on October 15. One ordinary message in the actual Grok conversation initiated the proactive source investigation; the operator created no preliminary task and sent no extra nudge. Serving Core 11d69d04c7531cf7eb37f7994b7112797e145837 includes #16394. The Bot created one task and one run, saved a 100-word report, submitted an accepted canonical result and reached Done. Its managed lease has a successful cleanup receipt. The exact public PR marker matches the saved control value, was absent from the operator prompt and appears in the saved report with its source. No Paperclip proxy operation occurred. The report records native GitHub API use; a complete provider tool transcript is unavailable. + +The proactive run also received an authenticated delayed-admission webhook while its original conversation was waiting. Two acceptance requests and two title writes were recorded. This fails the no-echo release condition despite successful task completion. Two local regressions reproduce the extra webhook for a delayed offer and a prepared intent recovered after restart. The correction keeps admission in the original caller's polling path and rejects old prepared proactive wakes. Human assignments and question continuations retain their ordinary dispatch paths. Source and real-client requalification are required. + +Normal staging automation has advanced beyond the older prepared candidate. No shared CI routing switch was changed. The earlier temporary fallback request is no longer needed for the already served successor. The dedicated production workspace still has no qualification agent grant or smoke task. Keep native providers and person-purpose MCP connections unchanged. No browser automation belongs in normal operation. Defer localhost relays, internal Grok APIs, marketplace distribution and Paperclip-connected third-party tools. Completion of a Paperclip work session does not assert remote Grok termination. diff --git a/server/src/__tests__/dot-runner.test.ts b/server/src/__tests__/dot-runner.test.ts index 80ce58575ee..5e4db08a3b5 100644 --- a/server/src/__tests__/dot-runner.test.ts +++ b/server/src/__tests__/dot-runner.test.ts @@ -243,6 +243,9 @@ describe("durable Dot Runner integration", () => { try { const requests = await Promise.all([1, 2, 3].map(() => f.broker.requestTurn(f.principal, "Create hello and assign it to my owner", requestId))); expect(requests[1]).toEqual(requests[0]); expect(requests[2]).toEqual(requests[0]); + expect(requests[0]!.instruction).toContain(provider === "grok_bot" + ? "Queued admission does not start another Grok routine" + : "Events also notify queued work"); const intake = await db.select().from(issues).where(eq(issues.id, requests[0]!.issueId)); expect(intake).toHaveLength(1); expect(intake[0]).toMatchObject({ assigneeAgentId: f.agent.id, createdByAgentId: f.agent.id, responsibleUserId: f.userId }); diff --git a/server/src/__tests__/grok-bot-routine.test.ts b/server/src/__tests__/grok-bot-routine.test.ts index 54ed20e5473..e54a3f286f2 100644 --- a/server/src/__tests__/grok-bot-routine.test.ts +++ b/server/src/__tests__/grok-bot-routine.test.ts @@ -289,6 +289,28 @@ describe("durable Grok routine qualification", () => { expect((await dotRunnerBroker(db).read(f.principal, work.id)).toolUsageInstructions).toContain("paperclip_grok_operation_status"); }); + it.each(["new offer", "prepared before restart"] as const)("does not echo delayed Bot-initiated work: %s", async scenario => { + const f = await fixture(); const { work, run } = await admittedWork(f, "offered"); + await db.update(heartbeatRuns).set({ contextSnapshot: { grokProactiveRequest: { + requestId: randomUUID(), bindingId: f.pairing.bindingId, bindingGeneration: 1, + } } }).where(eq(heartbeatRuns.id, run.id)); + await db.update(dotRunnerAssignments).set({ createdAt: new Date(Date.now() - 60_000) }).where(eq(dotRunnerAssignments.id, work.id)); + const legacy = scenario === "prepared before restart" ? await f.service.reserve({ + companyId: f.company.id, agentId: f.agent.id, bindingId: f.pairing.bindingId, + assignmentId: work.id, kind: "mailbox_updated", sourceEventId: `proactive-admission:${work.id}`, + }) : null; + f.delivery.mockClear(); const dispatcher = createGrokRoutineDispatcher(db, () => true, f.delivery); + await dispatcher.tick(); await dispatcher.tick(); + expect(f.delivery).not.toHaveBeenCalled(); + const wakes = await db.select().from(grokBotDispatches).where(eq(grokBotDispatches.assignmentId, work.id)); + if (legacy) expect(wakes).toEqual([expect.objectContaining({ id: legacy.id, status: "cancelled" })]); + else expect(wakes).toEqual([]); + // The original conversation still owns and can accept the polled offer. + await expect(dotRunnerBroker(db).read(f.principal, work.id)).resolves.toBeDefined(); + await (await acceptance(f, work))(); + expect((await db.select().from(dotRunnerAssignments).where(eq(dotRunnerAssignments.id, work.id)))[0]?.status).toBe("accepted"); + }); + it("cancels a queued wake when its assignment was already accepted", async () => { const f = await fixture(); const { work } = await admittedWork(f, "offered"); const queued = await f.service.reserve({ companyId: f.company.id, agentId: f.agent.id, diff --git a/server/src/services/dot-runner-broker.ts b/server/src/services/dot-runner-broker.ts index 0a41d9d78cd..c602ee37a3e 100644 --- a/server/src/services/dot-runner-broker.ts +++ b/server/src/services/dot-runner-broker.ts @@ -249,6 +249,16 @@ function createBroker(db: Db) { if (options.requireOffered && authority.assignment.status !== "offered") { throw fail("This work was already accepted. Read its feedback and operation receipts without another routine wake."); } + if (options.requireOffered && authority.binding.providerKind === "grok_bot") { + // A prepared intent from an older dispatcher must not start a second + // interaction while the original caller is waiting for admission. + const [run] = await db.select({ contextSnapshot: heartbeatRuns.contextSnapshot }).from(heartbeatRuns) + .where(and(eq(heartbeatRuns.id, authority.assignment.runId), eq(heartbeatRuns.companyId, companyId), eq(heartbeatRuns.agentId, agentId))); + const proactive = run?.contextSnapshot?.grokProactiveRequest as Record | undefined; + if (proactive?.bindingId === authority.binding.id && proactive.bindingGeneration === authority.binding.generation) { + throw fail("This Bot requested the work. Continue polling its original request instead of starting another routine."); + } + } return authority; }, async authorizeBinding(principal: McpPrincipal, companyId: string, bindingId: string) { @@ -390,13 +400,16 @@ function createBroker(db: Db) { if (!assignment) await new Promise(resolve => setTimeout(resolve, 50)); } while (!assignment && Date.now() < deadline); return { ...wake, issueId: task.id, identifier: task.identifier, assignment: assignment ?? null, - instruction: wake.status === "admission_failed" ? wake.message : "Drain the inbox now. If assignment is present, read and accept it, then use its catalog. If admission is still pending, retry this provider’s request_turn tool with exactly the same requestId and prompt to check this intake; never create a second request. Events also notify queued work. This intake supplies normal run authority; no human needs to create a preliminary task." }; + instruction: wake.status === "admission_failed" ? wake.message : `Drain the inbox now. If assignment is present, read and accept it, then use its catalog. If admission is still pending, retry this provider’s request_turn tool with exactly the same requestId and prompt to check this intake; never create a second request. ${b.providerKind === "grok_bot" ? "Queued admission does not start another Grok routine. Keep checking this request from the current conversation until it is admitted or fails." : "Events also notify queued work."} This intake supplies normal run authority; no human needs to create a preliminary task.` }; }, async requestWork(principal: McpPrincipal, issueId: string, requestId: string) { const b = await principalBinding(principal); if (!z.uuid().safeParse(requestId).success) throw fail("Use a stable UUID requestId."); if (!await enabled(b.providerKind) || !(await this.bindingForAgent(b.companyId, b.agentId))?.wakeVerified) throw fail("External agent admission is unavailable."); const key = `dot-work:${b.id}:${b.generation}:${requestId}`; + const admissionMessage = b.providerKind === "grok_bot" + ? "Normal admission determines when this task can run. Keep checking the original request with the same request ID and arguments, then read and accept its inbox assignment. Queued admission does not start another Grok routine." + : "Normal admission determines when this task can run. Read the mailbox after its event."; const receipt = async () => (await db.select().from(agentWakeupRequests).where(and(eq(agentWakeupRequests.companyId, b.companyId), eq(agentWakeupRequests.agentId, b.agentId), eq(agentWakeupRequests.idempotencyKey, key))).limit(1))[0]; const replay = async (row: typeof agentWakeupRequests.$inferSelect) => { if (row.payload?.issueId !== issueId) throw fail("requestId was reused for another task."); @@ -407,7 +420,7 @@ function createBroker(db: Db) { return { status: "admission_failed", runId: row.runId, admissionStatus: run.status, message: "The previous admission ended without an active assignment. Stop waiting for its mailbox event. Inspect the task's run and use normal task recovery after resolving the failure. This request ID retains its original receipt." }; } - return { status: "requested", runId: row.runId, message: "Normal admission determines when this task can run. Read the mailbox after its event." }; + return { status: "requested", runId: row.runId, message: admissionMessage }; }; // A retry is a receipt read, including after the task yielded or finished. const old = await receipt(); @@ -424,7 +437,7 @@ function createBroker(db: Db) { payload: { issueId, dotRequestId: requestId }, contextSnapshot: { issueId, ...(b.providerKind === "grok_bot" ? { grokProactiveRequest: { requestId, bindingId: b.id, bindingGeneration: b.generation } } : {}) }, idempotencyKey: key, requestedByActorType: "agent", requestedByActorId: b.agentId, allowRunCoalescing: false, durableDotRequest: { id: receiptId, companyId: b.companyId, agentId: b.agentId, issueId, requestId, idempotencyKey: key, requestedAt: new Date() } }); const reserved = await receipt(); - return reserved ? replay(reserved) : { status: "requested", runId: run?.id ?? null, message: "Normal admission determines when this task can run. Read the mailbox after its event." }; + return reserved ? replay(reserved) : { status: "requested", runId: run?.id ?? null, message: admissionMessage }; } catch (error) { // A concurrent reservation can lose the PK race on a different task. // Replay only that durable receipt; do not mask unrelated failures. diff --git a/server/src/services/grok-bot/dispatcher.ts b/server/src/services/grok-bot/dispatcher.ts index e9f08cb2e8b..864e3707114 100644 --- a/server/src/services/grok-bot/dispatcher.ts +++ b/server/src/services/grok-bot/dispatcher.ts @@ -1,5 +1,5 @@ -import { and, asc, eq, isNull, lt, sql } from "drizzle-orm"; -import { grokBotDispatches, grokBotRoutines, dotAgentBindings, dotRunnerAssignments, heartbeatRuns, type Db } from "@paperclipai/db"; +import { and, asc, eq } from "drizzle-orm"; +import { grokBotDispatches, grokBotRoutines, type Db } from "@paperclipai/db"; import { grokRoutineService } from "./routine.js"; import type { deliverGrokRoutineWake } from "./webhook.js"; import { externalBotEnabled } from "../external-bot-authority.js"; @@ -14,17 +14,9 @@ export function createGrokRoutineDispatcher(db: Db, isBackgroundWorkEnabled: () running = true; try { const service = grokRoutineService(db, deliver); - // A proactive caller receives an immediate assignment without an echo. - // Queued admission gets one delayed wake only while still unaccepted. - const proactive = await db.select({ assignment: dotRunnerAssignments }).from(dotRunnerAssignments) - .innerJoin(dotAgentBindings, eq(dotAgentBindings.id, dotRunnerAssignments.bindingId)) - .innerJoin(heartbeatRuns, eq(heartbeatRuns.id, dotRunnerAssignments.runId)) - .where(and(eq(dotAgentBindings.providerKind, "grok_bot"), isNull(dotAgentBindings.revokedAt), eq(dotRunnerAssignments.status, "offered"), - lt(dotRunnerAssignments.createdAt, new Date(Date.now() - 10_000)), sql`${heartbeatRuns.contextSnapshot}->'grokProactiveRequest'->>'bindingId' = ${dotAgentBindings.id}::text`)).limit(10); - for (const { assignment } of proactive) { - try { await service.reserve({ companyId: assignment.companyId, agentId: assignment.agentId, bindingId: assignment.bindingId, - assignmentId: assignment.id, kind: "mailbox_updated", sourceEventId: `proactive-admission:${assignment.id}` }); } catch { /* Current authority is rechecked on reservation and send. */ } - } + // Bot-initiated callers poll their durable request while admission is + // pending. Delay does not prove that the originating conversation ended. + // Starting a webhook here would create a competing Grok interaction. // An accepted session already has a routine handling it. Quiet MCP // traffic does not prove that routine ended. Keep feedback and receipts // durable for the Bot to read instead of starting overlapping work.