fix(runner): prepare diagnostic proxy and reduce bundled skill queries (#17941)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native turns reconcile company skills and select verified remote
runtime packs.
> - Catalog imports repeat folder transactions for skills in the same
bundled category.
> - Diagnostic requests can also reuse an older image proxy that lacks
the phase collector.
> - This pull request reuses categories within one import and selects
the current proxy for fresh diagnostics.
> - Each import has its own category map, and recovery keeps its
admitted runtime identity.
> - The result is less preparation work and a runtime that can provide
the requested measurements.

## Linked Issues or Issue Description

Refs #17918.

**What happened?**

A hosted run retained `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` but
emitted no provider phase batch. Its saved pack authority selected an
older image proxy. The deployed server proxy contained the collector.
The image proxy did not. Bundled inventory reconciliation also repeats
folder transactions for skills in the same category. The paired local
benchmark exposes that preparation cost.

**Expected behavior**

A fresh diagnostic request must use a verified proxy that supports the
current collector. Bundled skill reconciliation should resolve each
category once per import.

**Steps to reproduce**

1. Use a protected image pack with the same OpenCode interface revision
and an older proxy hash.
2. Enable `PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS=1` for a fresh
remote run.
3. Observe that the current backend accepts the older proxy and cannot
emit the requested phase batch.

## What Changed

- Reuse each bundled category during one import. Every skill still runs
its existing validation, access lookup, value comparison, and write
path.
- Compare the installed proxy hash with the current build when a fresh
run requests diagnostics.
- Keep compatible-pack reuse for ordinary runs and exact admitted
identity for recovery.
- Cover selection through the real backend, including recovery after
diagnostics are enabled.
- Document runtime preparation costs and admission limits for diagnostic
requests.

The skill importer owns category reuse. The provider verifier owns the
artifact comparison. The native backend passes the resolved diagnostic
requirement at the existing admission seam. The changes stay in those
existing owners and share one latency investigation and rollout.

## Verification

- Before the fix, the new verifier and backend cases failed because both
accepted a stale proxy.
- After the fix, 63 focused cases passed in 8.10 seconds. They cover
normal reuse, current diagnostic proxy selection, stale proxy rejection,
selected ACPX profiles, and admitted recovery identity.
- 110 skill and folder service cases passed against isolated real
PostgreSQL databases. Only cluster bootstrap used the existing local
PostgreSQL service. Queries, migrations, authorization, file handling,
and transactions ran on their real paths.
- A paired benchmark used the same immutable skill snapshot,
authenticated actor, 170-skill inventory, and 3.5 ms delay in each
network direction. It checked every delivered file and executable bit.
Warm median fell from 3.319 s to 2.482 s (25.2%). SQL calls fell by 81
in each matched sample, from 241–243 to 160–162. The bundled phase
median fell from 2.596 s to 1.721 s. The final measurement uses commit
`37eac8d8de`; only the skill importer differs from baseline
`7951455073`. These local results are not hosted acceptance.
- `pnpm -r --workspace-concurrency=1 typecheck` passed with the pinned
pnpm 9.15.4 and a 6 GiB Node heap. Earlier attempts exposed a subprocess
toolchain mismatch, a local IPC restriction, and a fixture type error.
Each was corrected before the passing run.
- Module boundary and whitespace checks passed.
- `pnpm build` passed for all workspaces.
- `pnpm test:run` did not complete. Embedded PostgreSQL initialization
failed after repeated startup attempts. The owned test subtree was
stopped after preserving the failure log. This is not a full local test
pass. On final commit `37eac8d8de`, all 53 successful check contexts
passed; two optional Storybook checks were skipped. Greptile is 5/5 with
no unresolved findings. The draft awaits merge approval.
- The catalog change has a measured local gain. The proxy change fixes
missing phase capture. Real hosted response latency remains above the
target and needs a new matched measurement after delivery.

## Risks

The category map belongs to one import and one company. The importer
still resolves current skill data on every invocation.

Fresh diagnostic runs can prepare a current verified pack, which can add
startup time. Admission can fail when matching runtime assets are
unavailable. Ordinary runs keep the existing fast compatibility path.
Recovery keeps the already admitted identity, even if diagnostics are
enabled later. The operator endpoint gate, collector limits, and closed
span fields are unchanged.

## Model Used

OpenAI GPT-6 through Codex, with repository inspection, shell execution,
browser tools, and Honeycomb queries. The session does not expose a
verifiable backend deployment ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/observability.md

@@ -354,6 +354,14 @@ phase names, wall timestamps, and success/failure, with a 1 MiB request limit,
 128 retained measurements, and 16 measurements per phase. No operator OTLP endpoint
 or ingestion credential is passed to OpenCode.
 
+For fresh remote diagnostic runs, the selected pack must contain the current
+build's OpenCode proxy. An otherwise compatible image pack can predate the
+collector. Such a pack cannot satisfy the diagnostic request. The existing
+verified pack preparation path selects or stages the current proxy; it can add
+startup time or fail admission when matching runtime assets are unavailable.
+Ordinary runs retain compatible image-pack reuse. Recovery retains the pack
+identity already admitted for that session.
+
 Available phases cover context/instruction loading, skill discovery, tool and model
 resolution, model streaming (`ai.streamText`, including stream consumption), Git
 discovery/default branch, snapshots, dependency installation, and plugin startup.

server/src/services/company-skills.ts

@@ -6385,6 +6385,7 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
     database: DbOrTransaction = db,
   ): Promise<CompanySkill[]> {
     const out: CompanySkill[] = [];
+    const bundledFolders = new Map<string, Awaited<ReturnType<typeof folderSvc.ensureBundledCategory>>>();
     for (const skill of imported) {
       assertImportedSkillKeyAllowed(skill);
       assertImportedSkillSourceAllowed(skill);
@@ -6412,9 +6413,11 @@ export function companySkillService(db: Db, options: { actor?: AuthorizationActo
       const parsed = parseFrontmatterMarkdown(skill.markdown);
       const storeMetadata = readSkillStoreMetadata(parsed.frontmatter, metadata);
       const bundledCategory = paperclipBundledFolderCategory(skill.key, incomingMeta);
-      const bundledFolder = bundledCategory
-        ? await folderSvc.ensureBundledCategory(companyId, bundledFolderLabel(bundledCategory))
-        : null;
+      let bundledFolder = bundledCategory ? bundledFolders.get(bundledCategory) : undefined;
+      if (bundledCategory && !bundledFolder) {
+        bundledFolder = await folderSvc.ensureBundledCategory(companyId, bundledFolderLabel(bundledCategory));
+        bundledFolders.set(bundledCategory, bundledFolder);
+      }
       const projectId = asString(incomingMeta.projectId);
       const projectName = asString(incomingMeta.projectName);
       const projectFolder = !existing && incomingKind === "project_scan" && projectId && projectName

server/src/services/native-runtime/native-session-executor.test.ts

@@ -1659,9 +1659,11 @@ describe("remote provider pack manifest", () => {
     return JSON.stringify(value);
   };
 
-  it.each(["opencode", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => {
+  it.each(["opencode", "opencode-diagnostics", "opencode-diagnostics-stale", "cursor", "claude-x64", "claude-arm64"] as const)("uses the bundled Linux image authority for remote %s without controller overrides", async provider => {
     const architecture = provider === "claude-arm64" ? "arm64" : "x64";
     const claude = provider.startsWith("claude-");
+    const opencode = provider.startsWith("opencode");
+    const diagnostics = provider.startsWith("opencode-diagnostics");
     const root = await mkdtemp(join(tmpdir(), "paperclip-bundled-image-"));
     const digest = "sha256:" + "a".repeat(64);
     const artifacts = Object.fromEntries(Object.entries({
@@ -1669,7 +1671,7 @@ describe("remote provider pack manifest", () => {
       productionLock: "pnpm-lock.yaml", opencodeCommand: "node_modules/.bin/opencode",
       opencodeExecutable: "node_modules/opencode-ai/bin/opencode.exe",
       opencodeProxy: "dist/cli/opencode-app-server-proxy.cjs", acpxSidecar: "dist/cli/acpx-runtime-sidecar.cjs",
-    }).map(([key, path]) => [key, { path, sha256: digest }])) as Record<"nodeCommand" | "opencodeCommand", { path: string; sha256: string }>;
+    }).map(([key, path]) => [key, { path, sha256: digest }])) as Record<"nodeCommand" | "opencodeCommand" | "opencodeProxy", { path: string; sha256: string }>;
     const payload = {
       compatibility: { opencode: 1, acpx: 1 },
       target: { platform: "linux", architecture }, runnerSourceRevision: "a".repeat(40),
@@ -1695,16 +1697,17 @@ describe("remote provider pack manifest", () => {
     daemon.writeUInt16LE(architecture === "arm64" ? 183 : 62, 18);
     await writeFile(state.bundledRunnerBinary, daemon);
     if (claude) state.resolveRemoteRunnerBinary.mockResolvedValue(state.bundledRunnerBinary);
-    const remoteExecution = { ...execution, provider: provider === "opencode"
+    const remoteExecution = { ...execution, provider: opencode
       ? { kind: "opencode", model: null } : { kind: "acpx", agent: claude ? "claude" : "cursor", model: null },
       workspace: { ...execution.workspace, cwd: root },
-      session: { ...execution.session, normalizedSessionId: `bundled-${provider}-${randomUUID()}`, driverKind: provider === "opencode" ? "opencode_server" : "acpx_runtime" },
+      session: { ...execution.session, normalizedSessionId: `bundled-${provider}-${randomUUID()}`, driverKind: opencode ? "opencode_server" : "acpx_runtime" },
     } as unknown as NativeExecutionInputV1;
     const syncIn = vi.fn(async (_operations: { files: { kind: string }[] }[]) => undefined);
     const installed = structuredClone(manifest);
-    if (provider === "opencode") {
+    if (opencode) {
       installed.payload.runnerSourceRevision = "c".repeat(40);
       installed.payload.artifacts.nodeCommand.sha256 = "sha256:" + "d".repeat(64);
+      if (provider !== "opencode-diagnostics") installed.payload.artifacts.opencodeProxy.sha256 = "sha256:" + "e".repeat(64);
       installed.digest = "sha256:" + createHash("sha256").update(canonical(installed.payload)).digest("hex");
     }
     const execute = vi.fn(async (command: { command: string; args?: string[] }) => ({
@@ -1717,12 +1720,18 @@ describe("remote provider pack manifest", () => {
       }) : "",
     }));
     try {
-      await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution,
+      const prepare = createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution,
         runnerInstanceId: "bundled-image-runner", runnerIngressAuthorized: true,
-        ...(provider === "opencode" ? { runnerEnvironment: { PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}),
+        ...(opencode ? { runnerEnvironment: { ...configuredEnvironmentProjection(diagnostics ? { PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" } : {}), PAPERCLIP_NATIVE_MCP_NAME: "assigned-fixture", PAPERCLIP_NATIVE_MCP_URL: "http://127.0.0.1/mcp/gateways/assigned-fixture", PAPERCLIP_NATIVE_MCP_TOKEN: "private-fixture" } } : {}),
         runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
           environmentId: "environment", leaseId: "lease", providerKey: "daytona",
           effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute, syncIn } } as never });
+      if (provider === "opencode-diagnostics-stale") {
+        await expect(prepare).rejects.toThrow("image provider pack did not match the bundled release identity");
+        expect(state.createBackend).not.toHaveBeenCalled();
+        return;
+      }
+      await prepare;
       state.createTransport.mockClear();
       state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
       const transport = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & {
@@ -1730,9 +1739,10 @@ describe("remote provider pack manifest", () => {
         environment?: NodeJS.ProcessEnv;
       };
       expect(transport.runnerBinary).toBe(state.bundledRunnerBinary);
-      if (provider === "opencode") {
+      if (opencode) {
         expect(state.createAssignedMcpTools).toHaveBeenCalledWith(expect.objectContaining({ gatewayPublicId: "assigned-fixture", bearerToken: "private-fixture", delivery: "on_demand" }));
         for (const name of ["PAPERCLIP_NATIVE_MCP_NAME", "PAPERCLIP_NATIVE_MCP_URL", "PAPERCLIP_NATIVE_MCP_TOKEN"]) expect(transport.environment?.[name]).toBeUndefined();
+        expect(transport.environment?.PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS).toBe(diagnostics ? "1" : undefined);
       }
       if (claude) {
         expect(state.resolveRemoteRunnerBinary).toHaveBeenCalledWith(expect.objectContaining({ remoteCwd: "/workspace", configuredPath: undefined }));
@@ -1750,7 +1760,8 @@ describe("remote provider pack manifest", () => {
       expect(execute.mock.calls.filter(([call]) => call.args?.[0] === "--build-metadata")).toHaveLength(1);
       expect(execute.mock.calls.some(([call]) => call.args?.[0] === "-e" || call.args?.[0] === "--version")).toBe(false);
       expect(syncIn.mock.calls.every(([operations]) => operations.every(operation => operation.files.every(file => file.kind === "file")))).toBe(true);
-      if (provider === "claude-x64") {
+      if (provider === "claude-x64" || provider === "opencode") {
+        const admitted = opencode ? installed : manifest;
         const identity = transport.prpIdentity!;
         const artifact = { version: "0.0.0", digest: "sha256:" + createHash("sha256").update(daemon).digest("hex") };
         resolveRemoteProviderPackAuthority({ stateDirectory: transport.stateDirectory!,
@@ -1758,7 +1769,7 @@ describe("remote provider pack manifest", () => {
             workspaceId: remoteExecution.binding.executionWorkspaceId, sessionId: remoteExecution.session.normalizedSessionId!,
             runnerInstanceId: identity.runnerInstanceId, environmentLeaseId: identity.environmentLeaseId,
             remoteCwd: "/workspace", providerDigest: nativeSha256(remoteExecution.provider) },
-          reattach: false, current: () => manifest, runnerArtifact: () => artifact }).verified();
+          reattach: false, current: () => admitted, runnerArtifact: () => artifact }).verified();
         await mkdir(join(transport.stateDirectory!, "control-plane"), { recursive: true });
         await writeFile(join(transport.stateDirectory!, "control-plane", "control-plane-state.json"), JSON.stringify(durableControlPlaneState(identity)));
         const next = structuredClone(manifest);
@@ -1771,7 +1782,7 @@ describe("remote provider pack manifest", () => {
             schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
             packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
             durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
-            providerPacks: [{ root: "/opt/paperclip-runner/provider-pack", manifest, immutable: true }],
+            providerPacks: [{ root: "/opt/paperclip-runner/provider-pack", manifest: admitted, immutable: true }],
           }) };
           if (request.args[0] === "-e" || request.args[0] === "--version") return { exitCode: 0, timedOut: false, stderr: "", stdout: request.args[0] === "--version" ? "1.18.34" : "" };
           const script = request.args[1];
@@ -1781,6 +1792,7 @@ describe("remote provider pack manifest", () => {
         });
         await createRunnerdBackend({ db: leaseDb(remoteExecution), execution: remoteExecution,
           runnerInstanceId: identity.runnerInstanceId, runnerIngressAuthorized: true,
+          ...(opencode ? { runnerEnvironment: configuredEnvironmentProjection({ PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS: "1" }) } : {}),
           runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd: "/workspace",
             environmentId: "environment", leaseId: "lease", providerKey: "daytona", effectiveCapabilities: { runnerWebSocketIngress: true }, runner: { execute: recoveryExecute },
             sandboxLeaseAcquisition: { outcome: "resumed", providerLeaseId: "sandbox" } } as never,
@@ -1790,7 +1802,7 @@ describe("remote provider pack manifest", () => {
         });
         state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
         const recovered = state.createTransport.mock.calls.at(-1)![0];
-        expect(recovered).toMatchObject({ providerPackAuthorityDigest: manifest.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 } });
+        expect(recovered).toMatchObject({ providerPackAuthorityDigest: admitted.digest, runnerArtifact: artifact, adoptExistingRunner: { pid: 123 } });
       }
     } finally {
       state.bundledPackManifestPath = null; state.bundledRunnerBinary = null;

server/src/services/native-runtime/native-session-executor.ts

@@ -12444,6 +12444,8 @@ async function createRunnerdBackendWithinSessionClaim(
           manifest: expected, installation,
           provider: input.execution.provider as { kind: "opencode" } | { kind: "acpx"; agent: string },
           requireExactIdentity: reattachProviderPack || explicitProviderPackSelection,
+          requireCurrentOpenCodeProxy: !reattachProviderPack
+            && runnerEnvironment?.PAPERCLIP_OPENCODE_PERFORMANCE_DIAGNOSTICS === "1",
         }));
         if (!reattachProviderPack) providerPackAuthority!.select(manifest);
         expectedProviderPackManifest = manifest;

server/src/services/native-runtime/remote-provider-pack-verification.test.ts

@@ -61,11 +61,12 @@ describe("installed provider pack compatibility", () => {
     }
   });
   const reseal = (pack: ProviderPackIdentity) => ({ ...pack, digest: sha(canonicalProviderPackJson(pack.payload)) });
-  const check = (installed: ProviderPackIdentity, options: { immutable?: boolean; requireExactIdentity?: boolean } = {}) => {
+  const check = (installed: ProviderPackIdentity, options: { immutable?: boolean; requireExactIdentity?: boolean; requireCurrentOpenCodeProxy?: boolean } = {}) => {
     const execute = vi.fn();
     return verifyRemoteProviderPack({ runner: { execute }, cwd: "/workspace", packRoot: "/opt/paperclip-runner/provider-pack",
       manifest: manifest("b"), installation: { root: "/opt/paperclip-runner/provider-pack", manifest: installed, immutable: options.immutable ?? true },
-      provider: { kind: "opencode" }, requireExactIdentity: options.requireExactIdentity });
+      provider: { kind: "opencode" }, requireExactIdentity: options.requireExactIdentity,
+      requireCurrentOpenCodeProxy: options.requireCurrentOpenCodeProxy });
   };
   it.each(["interface", "platform", "schema", "artifact_path", "digest", "legacy"])("rejects an incompatible %s", async field => {
     let pack = manifest("a");
@@ -85,6 +86,15 @@ describe("installed provider pack compatibility", () => {
     await expect(check(manifest("a"), { requireExactIdentity: true })).rejects.toThrow("incompatible");
     await expect(check(manifest("b"), { requireExactIdentity: true })).resolves.toEqual(manifest("b"));
   });
+  it("requires the current proxy for requested OpenCode diagnostics", async () => {
+    const installed = manifest("a");
+    await expect(check(installed)).resolves.toEqual(installed);
+    await expect(check(installed, { requireCurrentOpenCodeProxy: true }))
+      .rejects.toThrow("requested OpenCode diagnostics");
+    installed.payload.artifacts.opencodeProxy = manifest("b").payload.artifacts.opencodeProxy;
+    await expect(check(reseal(installed), { requireCurrentOpenCodeProxy: true }))
+      .resolves.toEqual(reseal(installed));
+  });
   it("ignores unrelated provider and dependency changes for OpenCode", async () => {
     const pack = manifest("a");
     pack.payload.compatibility!.acpx += 1;

server/src/services/native-runtime/remote-provider-pack-verification.ts

@@ -34,7 +34,7 @@ export type PackProvider = { kind: "opencode" } | { kind: "acpx"; agent: string
  */
 function compatibleInstallation<T extends ProviderPackIdentity>(
   installation: ProviderPackInstallation, expected: T, packRoot: string,
-  provider: PackProvider, requireExactIdentity: boolean,
+  provider: PackProvider, requireExactIdentity: boolean, requireCurrentOpenCodeProxy: boolean,
 ): T {
   const actual = installation.manifest as T | null;
   const payload = actual?.payload;
@@ -56,6 +56,10 @@ function compatibleInstallation<T extends ProviderPackIdentity>(
       throw new Error("runner_remote_provider_artifact_incompatible: installed provider pack artifact identity is invalid");
     }
   }
+  if (provider.kind === "opencode" && requireCurrentOpenCodeProxy
+    && payload.artifacts.opencodeProxy?.sha256 !== expected.payload.artifacts.opencodeProxy?.sha256) {
+    throw new Error("runner_remote_provider_artifact_incompatible: installed proxy does not match the requested OpenCode diagnostics");
+  }
   if (provider.kind === "acpx") {
     const agent = provider.agent;
     const currentProfile = expected.payload.acpxProfileDigests?.[agent];
@@ -72,7 +76,7 @@ function compatibleInstallation<T extends ProviderPackIdentity>(
   return structuredClone(actual);
 }
 
-export async function verifyRemoteProviderPack<T extends ProviderPackIdentity>({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false }: {
+export async function verifyRemoteProviderPack<T extends ProviderPackIdentity>({ runner, cwd, packRoot, manifest, installation, provider, requireExactIdentity = false, requireCurrentOpenCodeProxy = false }: {
   runner: CommandManagedRuntimeRunner;
   cwd: string;
   packRoot: string;
@@ -80,10 +84,11 @@ export async function verifyRemoteProviderPack<T extends ProviderPackIdentity>({
   installation?: ProviderPackInstallation;
   provider?: PackProvider;
   requireExactIdentity?: boolean;
+  requireCurrentOpenCodeProxy?: boolean;
 }): Promise<T> {
   if (installation) {
     if (!provider) throw new Error("runner_remote_provider_artifact_incompatible: provider interface is required");
-    return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity);
+    return compatibleInstallation(installation, manifest, packRoot, provider, requireExactIdentity, requireCurrentOpenCodeProxy);
   }
   const expected = Buffer.from(
     canonicalProviderPackJson(manifest),