diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 77ece897599..ce7060a65b8 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -168,6 +168,12 @@ keeps `reuseLease:false` and `runnerLifecycleMode:"per_turn"`. The dedicated warm-continuity fixture uses `reuseLease:true` and `runnerLifecycleMode:"warm"`; its distinct `configurationKey` is part of the suite fingerprint even though both fixtures report `environmentId:"daytona"`. +Suites may select an `environmentOverrides` entry by task ID and environment +ID when one case needs a different lifecycle. The matrix, setup payload and +suite fingerprint use that selected configuration. Kimi's native Write Stop +and provider-loss cases select the non-reused, per-turn Daytona fixture required +by their strict observer; its other Daytona cases retain warm reuse. Local Kimi +fixtures retain the instance-managed environment. Keep short provider cleanup backstops, a Daytona secret reference, and an immutable image digest. Teardown must delete the environment with reusable-lease destruction and must fail the @@ -380,6 +386,31 @@ the model's claimed result by reading the actual final bytes, and also exercises remote copy-back. Keep candidate admission scoped to the selected model and the isolated operator environment; ordinary agent configuration must not enable it. +## Kimi Code candidate qualification + +The explicit-only `kimi-qualification` suite reuses eight established user +journeys and adds native pending-permission Stop and native worker-loss journeys +in each local/Daytona environment, with one attempt per cell. The +`runner-acpx-kimi` profile selects the exact candidate/model pair in the isolated +server, and receives only an encrypted `KIMI_API_KEY` secret reference. No native question/plan or usage/cost support is inferred from these fixtures. Semantic Paperclip interaction cards retain their +existing browser, durable revision, and continuation assertions. + +The file cell creates its project/workspace through the public fixture registry +path, then the agent creates, edits, validates and registers the file itself. +Independent workspace byte and downloaded-artifact byte matchers both remain +required. Stop/reset and chat restart reuse their existing active-work and +continuation observations. They do not replace native pending-permission, +provider-loss, or complete runtime-cleanup acceptance. The original +`extended-harnesses` matrix remains unchanged. + +The two Kimi lifecycle tasks require a native Write request with an exact +workspace target. The browser must show its permission card before the action. +One task uses public Stop; the other kills only the attributed native child. +Both require a rejected stale Allow response, expired pending work, no completed +mutation, continuous target absence, and physical process retirement. The Linux +fault helper verifies process birth, ancestry, pinned code, held descriptors and +pidfd identity before it can signal a process. + ## Persistent agent files The `instruction_persistence` flow uses production managed storage and public file diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 22388682167..7521b58fd31 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1098,6 +1098,18 @@ published with permanent campaign dashboards; fixture authors must therefore keep credentials and other private data out of every captured UI state. SVG is active content and is rejected from the packaged evidence entirely. +For native matrix campaigns, set `PAPERCLIP_RUNNER_E2E_RESULTS_ROOT` to a +dedicated normalized absolute directory outside the checkout, for example +`/private/tmp/paperclip-product-evidence`. Final attempt packages, campaign JSON, +and dashboards are written there directly, so removing a checkout does not +remove their evidence. The default remains `tests/runner-e2e/results`. +Empty/relative paths, traversal, filesystem/home/checkout roots, and symbolic-link +paths are rejected before credentials load. The output root is owner-only. +Use a distinct `PAPERCLIP_E2E_CAMPAIGN_ID` for each invocation to preserve earlier +attempts. This changes storage only; required evidence, redaction, grades, and +temporary-instance cleanup retain their existing checks. Provider-connection, +GitHub-review, and product-journey launchers retain their separate output contracts. + The standard task and chat evidence collectors read durable run events through the paginated public API, including completion events beyond the first 1,000 rows. It rejects missing, repeated, or out-of-order sequence numbers and fails capture after @@ -1967,6 +1979,49 @@ an API-key pass must not be reported as subscription evidence. See [`doc/grok-native-runner.md`](../../doc/grok-native-runner.md) for installation, credential boundaries and repetition requirements. +### Kimi Code qualification + +`kimi-qualification` is a separate explicit-only candidate suite for maintained +Kimi Code `@moonshot-ai/kimi-code@2.3.3`. It registers ten journeys on both local +and Daytona: a completed reply, Paperclip semantic plan revision/approval, Ask +mode, semantic question continuation, question continuation after a controller +restart, verified file/artifact delivery, active Stop/reset/resume, and chat +continuity after restart, native pending-permission Stop, and native worker loss. +Its twenty cells do not change `extended-harnesses` +or the default `--all` selection. Each cell has one attempt with no automatic +retry. The exact `kimi-for-coding` model has authenticated request evidence; +Product journey qualification remains pending. Registration and deterministic +checks are not Product qualification evidence. The version-7 definition binds +the current pending native identity and trusted company-scoped startup authority on Mac ARM64 and +Linux x64. After company creation, the owned supervisor validates the selected +manual Kimi execution and restarts the same disposable server with its exact +model, profile, closure, target and expiry. Ambient grants, saved agent config +and public requests cannot stage this authority. Metadata Test uses a separately +bound operation/session identity and sends no model prompt. An ordinary qualified +native profile needs no candidate grant. Earlier version-4/5/6 results remain +historical evidence and do not qualify this merged runtime. + +Supply only the selected `KIMI_API_KEY`; the fixture writes its encrypted company +secret reference into the Kimi provider configuration. It does not fall back to +legacy `KIMI_MODEL_API_KEY` or import ambient login directories. Native Kimi +questions, native plan approval, child activity, per-turn usage/cost, and +same-turn steering are not claimed by these semantic journeys. Daytona cells +also require the configured immutable runtime image and the existing remote +prerequisites. Provider/process-loss and unanswered native-permission Stop +remain separate acceptance gates, not claims of the chat Stop cell. +The Kimi Stop/reset/resume cell retains all heartbeat rows. It still requires +exactly three native provider runs and one server-authored `/new` reset. A +separate pre-dispatch admission wait may be excluded from provider counts only +with the complete server receipt, no provider authority or events, the same +saved human `/new`, and exact recovered stopped-source evidence. The final +reset and fresh native continuation remain required; a cancelled legacy row +alone cannot qualify. Other chat suites keep their existing accounting. + +```sh +pnpm test:e2e:runner -- --list --suite kimi-qualification +pnpm test:e2e:runner -- --id kimi-qualification.runner-acpx-kimi.local.message-marker --max-automatic-retries 0 +``` + ### Grok branch qualification on EC2 The trusted default-branch workflow can run the explicit `grok-qualification` diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 906b6e1d1d5..07bfacf1f2b 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -156,6 +156,15 @@ rejects mutable tag or branch references. ## Secrets and protected environments +Kimi's explicit-only `kimi-qualification` cells use `KIMI_API_KEY`, delivered +through the launcher's normal encrypted company-secret fixture path. All ambient +`KIMI_*` values, including legacy model keys and login-home aliases, are removed +from the product server environment. Neither the catalog nor evidence stores +the key. The maintained Code profile does not borrow an ambient OAuth cache or +fall back to `KIMI_MODEL_API_KEY`. This registration does not add a hosted paid +workflow credential grant; hosted authorization and secret delivery remain a +separate change. + Explicit Grok subscription campaigns may select `GROK_AUTH_JSON` from the same protected paid environment. Deliver it only to that subscription profile's paid test step. Prefer a dedicated test-account login and refresh it before a campaign. diff --git a/tests/runner-e2e/api.test.ts b/tests/runner-e2e/api.test.ts index 0fc5d44b4c4..aca4cd2fbbd 100644 --- a/tests/runner-e2e/api.test.ts +++ b/tests/runner-e2e/api.test.ts @@ -67,3 +67,13 @@ it("keeps existing polling retries when the strict error policy is absent", asyn load, accept: Boolean, intervalMs: 1 })).resolves.toBe(true); expect(load).toHaveBeenCalledTimes(2); }); + +it.each([200, 204, 404, 409])("forwards the cleanup request timeout and preserves DELETE status %s", async status => { + vi.stubEnv("PAPERCLIP_RUNNER_E2E_PORT", "3100"); + const remove = vi.fn(async () => response(status)); + const api = new RunnerApi({ delete: remove } as unknown as APIRequestContext); + const result = api.delete("/api/environments/environment", { allowNotFound: true, timeout: 500 }); + if (status === 409) await expect(result).rejects.toThrow("409"); + else await expect(result).resolves.toBeUndefined(); + expect(remove).toHaveBeenCalledExactlyOnceWith("/api/environments/environment", { timeout: 500 }); +}); diff --git a/tests/runner-e2e/api.ts b/tests/runner-e2e/api.ts index 317401d7736..1e1dd6171bc 100644 --- a/tests/runner-e2e/api.ts +++ b/tests/runner-e2e/api.ts @@ -116,9 +116,9 @@ export class RunnerApi { async delete( path: string, - options?: { allowNotFound?: boolean }, + options?: { allowNotFound?: boolean; timeout?: number }, ): Promise { - const response = await this.request.delete(path, this.sessionOptions()); + const response = await this.request.delete(path, { ...this.sessionOptions(), timeout: options?.timeout }); if (response.ok() || (options?.allowNotFound && response.status() === 404)) return; throw new Error(await failureMessage(response, "DELETE")); diff --git a/tests/runner-e2e/budgeted-setup.test.ts b/tests/runner-e2e/budgeted-setup.test.ts index c1c7e4dacb7..0019cd73966 100644 --- a/tests/runner-e2e/budgeted-setup.test.ts +++ b/tests/runner-e2e/budgeted-setup.test.ts @@ -4,7 +4,12 @@ import path from "node:path"; import { expect, it, vi } from "vitest"; import type { RunnerApi } from "./api.js"; import { createEvalAttemptPhase, readEvalAttemptPhase } from "./attempt-phase.js"; -import { finishBudgetedEvalSetup, type EvalSetupObservation } from "./budgeted-setup.js"; +import { createKimiLifecycleTiming, finishBudgetedEvalSetup, type EvalSetupObservation } from "./budgeted-setup.js"; + +import { runKimiLifecycleFlow } from "./kimi-lifecycle.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import { runnerE2EPlaywrightCaseTimeoutMs, runnerE2EPlaywrightWatchdogMs } from "./web-server-command.js"; const identity = { executionId: "fixture-execution", attemptId: "fixture-attempt" }; const ready = { id: "agent", companyId: "company", status: "idle", lifecycleState: "ready", lifecycleError: null }; @@ -95,3 +100,95 @@ it.each(["local", "daytona"] as const)("uses the bounded attempt window for remo } } finally { now.mockRestore(); await rm(f.directory, { recursive: true, force: true }); } }); + +it("waits for the same agent to be ready before changing Kimi lifecycle policy without a gateway", async () => { + const execution = selectRunnerExecutions(parseRunnerSelectors(["--id", "kimi-qualification.runner-acpx-kimi.daytona.native-write-provider-loss"]))[0]!; + const order: string[] = [], observations: EvalSetupObservation[][] = []; + const configured = { ...ready, adapterConfig: { model: "kimi-for-coding" } }; + const get = vi.fn(async (url: string) => { + if (url === "/api/health") return { deploymentMode: "local_trusted" }; + if (url === "/api/auth/get-session") return { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } }; + if (url === "/api/agents/agent") { + order.push(order.length === 0 ? "verifying" : order.length === 1 ? "ready" : "policy-read"); + return order.length === 1 ? { ...configured, lifecycleState: "verifying" } : configured; + } + throw new Error(`Unexpected GET ${url}`); + }); + const patch = vi.fn(async () => { order.push("policy-patch"); throw new Error("stop-after-policy-patch"); }); + await expect(runKimiLifecycleFlow({ page: {} as any, api: { get, patch } as unknown as RunnerApi, + fixtures: { company: { id: "company" }, agent: { id: "agent" }, environment: { id: "environment" } } as LiveFixtureValues, + execution, nonce: "unit", workspacePath: "/unused", deadlineAt: Date.now() + 5_000, remoteBootstrap: {} as any, + registerCleanupAssertion: () => {}, registerBeforeEnvironmentTeardownAssertion: () => {}, observe: () => {}, capture: async () => {}, + evidence: async (name, value) => { if (name === "kimi-lifecycle-agent-ready.json") observations.push(structuredClone((value as { observations: EvalSetupObservation[] }).observations)); }, + })).rejects.toThrow("stop-after-policy-patch"); + expect(order).toEqual(["verifying", "ready", "policy-read", "policy-patch"]); + expect(observations.at(-1)?.map(row => row.lifecycleState)).toEqual(["verifying", "ready"]); + expect(observations.flat()).not.toContainEqual(expect.objectContaining({ adapterConfig: expect.anything() })); + expect(patch).toHaveBeenCalledOnce(); +}); + +it("does not change Kimi lifecycle policy after readiness fails", async () => { + const execution = selectRunnerExecutions(parseRunnerSelectors(["--id", "kimi-qualification.runner-acpx-kimi.daytona.native-write-provider-loss"]))[0]!; + const get = vi.fn(async (url: string) => url === "/api/health" ? { deploymentMode: "local_trusted" } + : url === "/api/auth/get-session" ? { session: { userId: "local-board", id: "paperclip:local_implicit:local-board" } } + : { ...ready, lifecycleError: "harness_command_unavailable" }); + const patch = vi.fn(); + await expect(runKimiLifecycleFlow({ page: {} as any, api: { get, patch } as unknown as RunnerApi, + fixtures: { company: { id: "company" }, agent: { id: "agent" }, environment: { id: "environment" } } as LiveFixtureValues, + execution, nonce: "unit", workspacePath: "/unused", deadlineAt: Date.now() + 5_000, remoteBootstrap: {} as any, + registerCleanupAssertion: () => {}, registerBeforeEnvironmentTeardownAssertion: () => {}, observe: () => {}, capture: async () => {}, evidence: async () => {}, + })).rejects.toThrow("setup failed"); + expect(patch).not.toHaveBeenCalled(); +}); + +it("keeps a finite setup cap and one full graded Kimi window after the recorded 242s readiness delay", () => { + const startedAt = 1_000_000, deadlineMs = 300_000; + const timing = createKimiLifecycleTiming({ startedAt, deadlineMs }); + expect(timing.setupDeadlineAt).toBe(startedAt + 300_000); + expect(timing.playwrightTimeoutMs).toBe(690_000); + expect(timing.cleanupDeadlineAt).toBe(startedAt + 660_000); + expect(startedAt + timing.playwrightTimeoutMs - timing.cleanupDeadlineAt).toBe(30_000); + const readyAt = startedAt + 242_000; + const workDeadlineAt = timing.beginWork(readyAt); + expect(workDeadlineAt - readyAt).toBe(300_000); + expect(workDeadlineAt + 90_000 - startedAt).toBeLessThanOrEqual(timing.playwrightTimeoutMs); + expect(() => timing.beginWork(readyAt + 1_000)).toThrow("already started"); + for (const elapsed of [300_000, 301_000]) { + expect(() => createKimiLifecycleTiming({ startedAt, deadlineMs }).beginWork(startedAt + elapsed)).toThrow("setup deadline expired"); + } +}); + +it.each([60_000, 300_000])("lets Kimi cleanup and its final report finish after %sms of server startup", async startupMs => { + vi.useFakeTimers(); + try { + const execution = selectRunnerExecutions(parseRunnerSelectors(["--id", "kimi-qualification.runner-acpx-kimi.daytona.native-write-provider-loss"]))[0]!; + const deadlineMs = execution.task.attemptTimeoutMs[execution.environment.id]; + const spawnedAt = Date.now(); + let killed = false, cleanupFinished = false, reportSaved = false; + const watchdog = setTimeout(() => { killed = true; }, runnerE2EPlaywrightWatchdogMs([execution])); + await vi.advanceTimersByTimeAsync(startupMs); + const timing = createKimiLifecycleTiming({ startedAt: Date.now(), deadlineMs }); + setTimeout(() => { if (!killed) cleanupFinished = true; }, timing.cleanupDeadlineAt - Date.now() - 1); + setTimeout(() => { if (!killed && cleanupFinished) reportSaved = true; }, timing.playwrightTimeoutMs - 1); + await vi.advanceTimersByTimeAsync(timing.playwrightTimeoutMs - 1); + expect({ killed, cleanupFinished, reportSaved }).toEqual({ killed: false, cleanupFinished: true, reportSaved: true }); + expect(runnerE2EPlaywrightCaseTimeoutMs(execution.task.flow, deadlineMs)).toBe(timing.playwrightTimeoutMs); + expect(Date.now() - spawnedAt).toBe(startupMs + 690_000 - 1); + await vi.advanceTimersByTimeAsync(300_000 - startupMs + 1); + expect(killed).toBe(true); + clearTimeout(watchdog); + } finally { vi.clearAllTimers(); vi.useRealTimers(); } +}); + +it("keeps other launcher case and startup allowances unchanged", () => { + const executions = selectRunnerExecutions(parseRunnerSelectors(["--suite", "kimi-qualification"])); + const others = executions.filter(execution => execution.task.flow !== "kimi_native_lifecycle"); + expect(others.length).toBeGreaterThan(0); + for (const execution of others) { + const deadlineMs = execution.task.attemptTimeoutMs[execution.environment.id]; + expect(runnerE2EPlaywrightCaseTimeoutMs(execution.task.flow, deadlineMs)).toBe(deadlineMs + 90_000); + expect(runnerE2EPlaywrightWatchdogMs([execution])).toBe(deadlineMs + 90_000 + 300_000); + } + expect(runnerE2EPlaywrightWatchdogMs(executions)).toBe(executions.reduce((total, execution) => total + + (execution.task.flow === "kimi_native_lifecycle" ? 2 : 1) * execution.task.attemptTimeoutMs[execution.environment.id] + 90_000, 300_000)); +}); diff --git a/tests/runner-e2e/budgeted-setup.ts b/tests/runner-e2e/budgeted-setup.ts index ee69773da95..942495f4e88 100644 --- a/tests/runner-e2e/budgeted-setup.ts +++ b/tests/runner-e2e/budgeted-setup.ts @@ -2,11 +2,12 @@ import type { RunnerEnvironmentId } from "./types.js"; import type { Agent } from "../../packages/shared/src/types/agent.js"; import { pollUntil, type RunnerApi } from "./api.js"; import { beginEvalAgentPhase } from "./attempt-phase.js"; +import { runnerE2EPlaywrightCaseTimeoutMs } from "./web-server-command.js"; type SetupState = Pick & { lifecycleHolds?: string[] }; export type EvalSetupObservation = SetupState & { observedAt: string }; -export async function finishBudgetedEvalSetup(input: { +type EvalAgentSetupInput = { executionId: string; environmentId: RunnerEnvironmentId; agentId: string; @@ -14,33 +15,51 @@ export async function finishBudgetedEvalSetup(input: { deadlineAt: number; api: Pick; observe: (observations: EvalSetupObservation[]) => Promise; -}, env: NodeJS.ProcessEnv = process.env) { - if (env.PAPERCLIP_E2E_GATEWAY_BASE_URL) { - const deadlineAt = input.environmentId === "daytona" ? input.deadlineAt : Math.min(input.deadlineAt, Date.now() + 45_000); - const observations: EvalSetupObservation[] = []; - let previous: string | undefined; - const scoped = (agent: SetupState) => agent.id === input.agentId && agent.companyId === input.companyId; - await pollUntil({ label: "budgeted eval agent setup", deadlineAt, intervalMs: 500, - load: async () => { - const agent = await input.api.get(`/api/agents/${input.agentId}`, { timeout: Math.max(1, Math.min(5_000, deadlineAt - Date.now())) }); - const state: SetupState = { id: agent.id, companyId: agent.companyId, status: agent.status, - lifecycleState: agent.lifecycleState, lifecycleError: agent.lifecycleError, lifecycleHolds: agent.lifecycleHolds }; - const current = JSON.stringify(state); - if (current !== previous) { - observations.push({ ...state, observedAt: new Date().toISOString() }); - await input.observe(observations); - previous = current; - } - return state; - }, - accept: agent => scoped(agent) && agent.lifecycleState === "ready" && ["idle", "active"].includes(agent.status) - && !agent.lifecycleError && !agent.lifecycleHolds?.length, - reject: agent => !scoped(agent) ? "Agent scope changed" - : agent.lifecycleError ? `Agent setup failed: ${agent.lifecycleError}` - : agent.lifecycleHolds?.length ? "Agent setup is held" - : ["paused", "rejected", "terminated"].includes(agent.lifecycleState ?? "") ? `Agent setup stopped: ${agent.lifecycleState}` : undefined, - }); - if (Date.now() >= deadlineAt) throw new Error("Timed out saving budgeted eval agent readiness"); - } +}; + +export async function waitForEvalAgentReady(input: Omit) { + const deadlineAt = input.environmentId === "daytona" ? input.deadlineAt : Math.min(input.deadlineAt, Date.now() + 45_000); + const observations: EvalSetupObservation[] = []; + let previous: string | undefined; + const scoped = (agent: SetupState) => agent.id === input.agentId && agent.companyId === input.companyId; + await pollUntil({ label: "budgeted eval agent setup", deadlineAt, intervalMs: 500, + load: async () => { + const agent = await input.api.get(`/api/agents/${input.agentId}`, { timeout: Math.max(1, Math.min(5_000, deadlineAt - Date.now())) }); + const state: SetupState = { id: agent.id, companyId: agent.companyId, status: agent.status, + lifecycleState: agent.lifecycleState, lifecycleError: agent.lifecycleError, lifecycleHolds: agent.lifecycleHolds }; + const current = JSON.stringify(state); + if (current !== previous) { + observations.push({ ...state, observedAt: new Date().toISOString() }); + await input.observe(observations); + previous = current; + } + return state; + }, + accept: agent => scoped(agent) && agent.lifecycleState === "ready" && ["idle", "active"].includes(agent.status) + && !agent.lifecycleError && !agent.lifecycleHolds?.length, + reject: agent => !scoped(agent) ? "Agent scope changed" + : agent.lifecycleError ? `Agent setup failed: ${agent.lifecycleError}` + : agent.lifecycleHolds?.length ? "Agent setup is held" + : ["paused", "rejected", "terminated"].includes(agent.lifecycleState ?? "") ? `Agent setup stopped: ${agent.lifecycleState}` : undefined, + }); + if (Date.now() >= deadlineAt) throw new Error("Timed out saving budgeted eval agent readiness"); +} + +export async function finishBudgetedEvalSetup(input: EvalAgentSetupInput, env: NodeJS.ProcessEnv = process.env) { + if (env.PAPERCLIP_E2E_GATEWAY_BASE_URL) await waitForEvalAgentReady(input); await beginEvalAgentPhase(input.executionId, env); } + +export function createKimiLifecycleTiming(input: { startedAt: number; deadlineMs: number }) { + const setupDeadlineAt = input.startedAt + input.deadlineMs; + let workStarted = false; + return { setupDeadlineAt, playwrightTimeoutMs: runnerE2EPlaywrightCaseTimeoutMs("kimi_native_lifecycle", input.deadlineMs), + cleanupDeadlineAt: input.startedAt + 2 * input.deadlineMs + 60_000, + beginWork(readyAt: number) { + if (workStarted) throw new Error("Kimi lifecycle work window already started"); + if (readyAt < input.startedAt || readyAt >= setupDeadlineAt) throw new Error("Kimi lifecycle setup deadline expired"); + workStarted = true; + return readyAt + input.deadlineMs; + }, + }; +} diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 0a0c80bdf39..c6e51d0537a 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -74,6 +74,24 @@ describe("runner E2E catalog", () => { retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1 }); }); + it("selects non-reused Kimi lifecycle leases without changing its warm or local cases", () => { + const suite = runnerSuites.find(row => row.id === "kimi-qualification")!; + const cells = runnerMatrix.filter(row => row.suite.id === suite.id); + const input = { secretRefs: { DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "fixture-secret", version: "latest" as const } }, + daytonaImage: `registry.example.test/runtime@sha256:${"a".repeat(64)}`, executionId: "fixture" }; + expect(cells).toHaveLength(20); + for (const cell of cells) { + if (cell.environment.id === "local") { + expect(cell.environment).toBe(suite.environments.find(environment => environment.id === "local")); + continue; + } + const lifecycle = cell.task.flow === "kimi_native_lifecycle"; + expect(cell.environment.buildEnvironment(input).config).toMatchObject({ reuseLease: !lifecycle, runnerLifecycleMode: lifecycle ? "per_turn" : "warm" }); + expect(cell.groups.includes("warm")).toBe(!lifecycle); + } + expect(suiteDefinitionHash(suite)).not.toBe(suiteDefinitionHash({ ...suite, environmentOverrides: undefined })); + }); + it.each([ ["runner-codex-subscription", "CODEX_AUTH_JSON"], ["runner-codex", "OPENAI_API_KEY"], ])("keeps %s receipt smoke separately selectable with exactly four bounded native turns", (profile, credential) => { @@ -260,10 +278,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(775); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(775); + expect(validateRunnerCatalog()).toHaveLength(795); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(795); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 7c1dca62b36..8b6fae6296c 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -8,6 +8,7 @@ import { TASK_WORKSPACES_SUITE, taskWorkspaceTask, taskWorkspaceProfile, taskWor import { repositoryProofTasks, repositoryProofProfile, repositoryProofDigest } from "./repository-verification.js"; import { automaticRepositoryTask, automaticRepositoryDigest } from "./automatic-repository.js"; import { nativeActiveStopTasks } from "./native-active-stop-tasks.js"; +import { kimiLifecycleTasks } from "./kimi-lifecycle-tasks.js"; import { piControlTasks } from "./pi-controls-cases.js"; import { cursorNativeTasks } from "./cursor-native-cases.js"; import { copilotProtectionTasks } from "./copilot-protection-tasks.js"; @@ -195,7 +196,7 @@ function nativeProfile(input: { provider: "codex" | "opencode" | "acpx"; model: string; credential: RunnerProfileFixture["credential"]; - acpxAgent?: "claude" | "codex" | "grok" | "cursor" | "copilot" | "pi"; + acpxAgent?: "claude" | "codex" | "grok" | "cursor" | "copilot" | "pi" | "kimi"; qualificationCandidate?: RunnerProfileFixture["qualificationCandidate"]; supportedEnvironments?: readonly (typeof ENVIRONMENT_IDS)[number][]; modelQualification?: RunnerProfileFixture["modelQualification"]; @@ -664,7 +665,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [ buildRevisionRequest: (nonce) => [ "Revise this same plan; do not implement it yet.", - `Remove PAPERCLIP_E2E_PLAN_DRAFT_${nonce} and include PAPERCLIP_E2E_PLAN_REVISED_${nonce}.`, + `Replace the first line with exactly PAPERCLIP_E2E_PLAN_REVISED_${nonce} and remove PAPERCLIP_E2E_PLAN_DRAFT_${nonce}. Use literal underscore characters with no backslashes or other text on that first line.`, "Change the plan from two steps to exactly three numbered steps, with verification as step 3.", "Publish the revised canonical Plan revision and request confirmation for that new revision.", "In a native runner, call write_document for key `plan`, then call request_human_input exactly once with interactionKind `confirmation`, targetRevisionId set to the returned latest Plan revision, and continuationPolicy `wake_assignee`; do not call paperclip_finish while waiting. After write_document succeeds, request_human_input must be your immediate next action using that call's returned latestRevisionId; do not emit assistant text, end the response or heartbeat, or stop after write_document alone.", @@ -674,7 +675,7 @@ export const runnerTasks: readonly RunnerTaskFixture[] = [ [ "This is a planning-mode lifecycle acceptance task.", "First, create a small canonical Plan with exactly two numbered steps and request approval; do not implement it.", - `The initial Plan body must contain PAPERCLIP_E2E_PLAN_DRAFT_${nonce}.`, + `The initial Plan body must have exactly PAPERCLIP_E2E_PLAN_DRAFT_${nonce} as its first line, followed by exactly two numbered steps. Use literal underscore characters with no backslashes or other text on that first line.`, "If the plan is sent back, revise that same Plan document according to the revision note, publish a new revision-bound confirmation, and still do not implement.", `Before the revised Plan is accepted, do not spell, quote, repeat, announce, or include PAPERCLIP_E2E_PLAN_DONE_${nonce} in any visible response, comment, or tool payload; refer to it only as “the terminal marker.”`, "Only after the revised plan is accepted, implement it by posting one final visible task-thread response containing exactly " + @@ -1447,6 +1448,38 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ], expectedMatrixSize: 16, definitionMetadata: { version: 2, binary: "1.0.13", model: "grok-4.7", scheduling: "explicit-only", repetitionsRequired: 3, artifactOracle: "independent-python-contract", stopBoundary: "provider-turn-started" }, }, + { + id: "kimi-qualification", label: "Kimi Code Qualification", manualOnly: true, + description: "Kimi Code replies, semantic planning and questions, verified file delivery, Stop and controller continuation through local and Daytona user journeys.", + groups: ["native"], + profiles: [nativeProfile({ + id: "runner-acpx-kimi", label: "Runner Kimi Code", + provider: "acpx", acpxAgent: "kimi", qualificationCandidate: "kimi", + credential: "KIMI_API_KEY", model: "kimi-for-coding", + modelQualification: { source: "qualified_runner_profile", qualificationId: "kimi:2.3.3:native-qualified-profile17" }, + })], + environments: [localEnvironment, daytonaWarmEnvironment], + environmentOverrides: Object.fromEntries(kimiLifecycleTasks.map(task => [task.id, { + daytona: runnerEnvironments.find(environment => environment.id === "daytona")!, + }])), + tasks: [ + ...runnerTasks, ...localIntegrityTasks, extendedHarnessFileTask, + ...chatHardeningTasks.filter(task => ["stop-new-resume", "continuity-restart"].includes(task.id)), + ...kimiLifecycleTasks, + ].map(task => ({ ...task, automaticRetryPolicy: "single_attempt" as const })), + expectedMatrixSize: 20, + definitionMetadata: { version: 7, binary: "2.3.3", model: "kimi-for-coding", + candidateAdmission: "ordinary-released-profile-without-candidate-grant-v1", profileVersion: QUALIFIED_ACPX_PROFILES.kimi.agentProfileVersion, + stopNewRunAccounting: "three-native-provider-runs-authored-reset-and-verified-predispatch-wait-v1", + planPromptContract: "literal-first-line-markers-v2", + modelDiscovery: "exact-model-authenticated-observed", qualification: "qualified", + scheduling: "explicit-only", maximumAttemptsPerCell: 1, repetitionsRequired: 1, + instructionAdmission: "native-effective-prompt-digest-on-create-load", + nativeQuestions: "not-claimed", nativePlans: "not-claimed", accounting: "usage-and-cost-unavailable", + artifactOracle: "independent-workspace-and-public-download-bytes", stopBoundary: "provider-turn-started", + remoteLifecycleBootstrap: "observer-bound-scoped-get-task-context-dual-turn-workspace-binding-full-semantic-command-ack-and-bounded-native-receipt", + nativeWriteLifecycle: "pending-permission-stop-and-exact-native-worker-loss", nativeWriteCleanup: "continuous-no-effects-through-owned-tree-retirement" }, + }, { id: "api-response-reading", label: "Bounded API response reading", manualOnly: true, description: "Retrieve evidence beyond a saved API preview through authorized bounded text windows.", @@ -1870,6 +1903,11 @@ export function suiteDefinitionHash(suite: RunnerSuiteFixture) { id: environment.id, configurationKey: environment.configurationKey ?? "default", })), + ...(suite.environmentOverrides ? { environmentOverrides: Object.entries(suite.environmentOverrides).map(([taskId, overrides]) => ({ + taskId, environments: Object.values(overrides).flatMap(environment => environment ? [{ + id: environment.id, configurationKey: environment.configurationKey ?? "default", + }] : []), + })) } : {}), tasks: suite.tasks.map((task) => ({ id: task.id, flow: task.flow, @@ -1898,32 +1936,36 @@ export function buildRunnerMatrix( ) .flatMap((environment) => suite.tasks - .map((task) => ({ - id: `${suite.id}.${profile.id}.${environment.id}.${task.id}`, - suite, - suiteDefinitionHash: suiteDefinitionHash(suite), - profile, - environment, - task: suite.id === "confirmation-replies" && profile.id === "runner-codex" && environment.id === "daytona" && task.id === "task-card-accept" - ? { ...task, expectedRunCount: 4, automaticRetryPolicy: "single_attempt" as const } - : suite.id === "extended-harnesses" && profile.qualificationCandidate === "pi" && task.id === "file-edit-validate" - ? { ...task, buildPrompt: piFilePrompt } - : suite.id === "extended-harnesses" && profile.qualificationCandidate === "copilot" && task.id === "file-edit-validate" - ? { ...task, buildPrompt: copilotFilePrompt } : task, - groups: [ - ...new Set([ - ...suite.groups, - ...profile.groups, - ...environment.groups, - ...task.groups, - ]), - ], - requiredCredentials: [ - ...(["provider_connection", "github_review_journey", "antigravity_agent_journey"].includes(task.flow) ? [] : [profile.credential]), - ...(!["provider_connection", "github_review_journey", "antigravity_agent_journey"].includes(task.flow) && environment.credential ? [environment.credential] : []), - ...(task.requiredCredentials ?? []), - ], - })) + .map((task) => { + const selectedEnvironment = suite.environmentOverrides?.[task.id]?.[environment.id] ?? environment; + if (selectedEnvironment.id !== environment.id) throw new Error(`Fixture environment override changed provider ID for ${suite.id}.${task.id}`); + return { + id: `${suite.id}.${profile.id}.${environment.id}.${task.id}`, + suite, + suiteDefinitionHash: suiteDefinitionHash(suite), + profile, + environment: selectedEnvironment, + task: suite.id === "confirmation-replies" && profile.id === "runner-codex" && environment.id === "daytona" && task.id === "task-card-accept" + ? { ...task, expectedRunCount: 4, automaticRetryPolicy: "single_attempt" as const } + : suite.id === "extended-harnesses" && profile.qualificationCandidate === "pi" && task.id === "file-edit-validate" + ? { ...task, buildPrompt: piFilePrompt } + : suite.id === "extended-harnesses" && profile.qualificationCandidate === "copilot" && task.id === "file-edit-validate" + ? { ...task, buildPrompt: copilotFilePrompt } : task, + groups: [ + ...new Set([ + ...suite.groups, + ...profile.groups, + ...selectedEnvironment.groups, + ...task.groups, + ]), + ], + requiredCredentials: [ + ...(["provider_connection", "github_review_journey", "antigravity_agent_journey"].includes(task.flow) ? [] : [profile.credential]), + ...(!["provider_connection", "github_review_journey", "antigravity_agent_journey"].includes(task.flow) && selectedEnvironment.credential ? [selectedEnvironment.credential] : []), + ...(task.requiredCredentials ?? []), + ], + }; + }) .filter((execution) => !excludedExecutionIds.has(execution.id)), ), ); diff --git a/tests/runner-e2e/chat-flow.ts b/tests/runner-e2e/chat-flow.ts index f529ca2f781..b772c27e89c 100644 --- a/tests/runner-e2e/chat-flow.ts +++ b/tests/runner-e2e/chat-flow.ts @@ -10,7 +10,7 @@ import type { } from "../../packages/shared/src/types/issue.js"; import type { LiveFixtureValues } from "./live-fixtures.js"; import type { MatrixExecution } from "./types.js"; -import { isBlockedUnstartedWake, isReassignedUnstartedWake } from "./non-execution-wake.js"; +import { chatProviderRuns, classifyPreDispatchExecutionWait, isBlockedUnstartedWake, isReassignedUnstartedWake, verifyChatAdmissionWaitCoverage, type ChatAdmissionWait } from "./non-execution-wake.js"; import { collectRunEvents } from "./run-observations.js"; import { chatMarker } from "./chat-cases.js"; import { assertChatRememberedAfterRestart, assertChatStartupStopped, isChatStopReady, runChatHardeningFlow } from "./chat-hardening.js"; @@ -292,7 +292,12 @@ export async function readRunningChatLog( export async function collectChatRunEvidence( api: Pick, run: ChatRun, + allowAdmissionWait = false, ) { + const admissionEvents = allowAdmissionWait ? await collectRunEvents((afterSeq, limit) => + api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`), + ) : null; + const admissionWait = admissionEvents ? classifyPreDispatchExecutionWait(run, admissionEvents) : null; const recovery = run.resultJson?.executionRecovery as Record | undefined; const providerNeverStarted = run.runtimeMode === "legacy" && !run.runtimeModeResolvedAt && run.logStore === null && run.logRef === null && ["failed", "interrupted"].includes(run.status) && @@ -301,11 +306,12 @@ export async function collectChatRunEvidence( return { runId: run.id, ...(providerNeverStarted ? { logOmissionReason: "provider_not_started" } : {}), + ...(admissionWait ? { logOmissionReason: "pre_dispatch_execution_wait" } : {}), ...(reassignedBeforeStart ? { logOmissionReason: "assignee_changed_before_start" } : {}), - log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted || reassignedBeforeStart + log: isResetRun(run) || isBlockedUnstartedWake({ ...run }) || providerNeverStarted || admissionWait || reassignedBeforeStart ? null : await api.get(`/api/heartbeat-runs/${run.id}/log?limitBytes=1048576`), - events: await collectRunEvents((afterSeq, limit) => + events: admissionEvents ?? await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${run.id}/events?afterSeq=${afterSeq}&limit=${limit}`), ), }; @@ -364,6 +370,7 @@ export interface ChatFlowInput { execution: MatrixExecution; nonce: string; workspacePath: string; + deadlineAt: number; restart: () => Promise; observe: (issue: ChatIssue, runs: ChatRun[]) => void; capture: (id: string, label: string, file: string) => Promise; @@ -382,6 +389,10 @@ export async function runChatFlow(input: ChatFlowInput) { const expectedStops = new Map(); let issue: ChatIssue; let runs: ChatRun[] = []; + const admissionAccounting = execution.suite.id === "kimi-qualification" && caseId === "stop-new-resume" && execution.profile.generation === "native"; + let admissionWaits: ChatAdmissionWait[] = []; + let stoppedForSessionReset: string | undefined; + const providerRuns = () => admissionAccounting ? chatProviderRuns(runs, admissionWaits) : runs.filter(run => !isResetRun(run)); const settings = await api.get>( "/api/instance/settings/experimental", ); @@ -404,6 +415,14 @@ export async function runChatFlow(input: ChatFlowInput) { (a, b) => a.createdAt.localeCompare(b.createdAt) || a.id.localeCompare(b.id), ); + const finalAdmissionEvidence = async () => ({ + comments: await comments(), + runEvidence: await Promise.all(runs.map(run => collectChatRunEvidence(api, run, true))), + sources: await Promise.all(admissionWaits.map(async wait => ({ + runId: stoppedForSessionReset!, recoveryActionId: wait.recoveryActionId, + response: await api.get(`/api/issues/${issue!.id}/recovery-actions?inspectSourceRunId=${stoppedForSessionReset}&inspectSourceActionId=${wait.recoveryActionId}`), + }))), + }); const idle = async (minimumProviderRuns: number) => { const inconsistentIdle = createChatIdleFailureDetector(minimumProviderRuns); await pollUntil({ @@ -414,12 +433,35 @@ export async function runChatFlow(input: ChatFlowInput) { const resolved = await api.get(chatPath); if (resolved) issue = resolved; runs = await allRuns(); + if (admissionAccounting) { + admissionWaits = []; + for (const run of runs.filter(row => row.errorCode === "execution_reconciliation_required")) { + const evidence = await collectChatRunEvidence(api, run, true); + const wait = classifyPreDispatchExecutionWait(run, evidence.events); + expect(wait?.companyId).toBe(f.company.id); + expect(wait?.agentId).toBe(f.agent.id); + expect(wait?.issueId).toBe(resolved?.id); + admissionWaits.push(wait!); + } + } if (resolved) input.observe(resolved, runs); + let admissionReady = true; + if (admissionAccounting && admissionWaits.length && minimumProviderRuns === 3 && + providerRuns().length === 3 && runs.every(run => ["succeeded", "cancelled"].includes(run.status))) { + const proof = await finalAdmissionEvidence(); + admissionReady = !proof.sources.some(source => { + const inspection = (source.response as { sourceInspection?: { status?: string; failureCode?: string } }).sourceInspection; + return inspection?.status === "blocked" && inspection.failureCode === "workspace_writer_active"; + }); + if (admissionReady) verifyChatAdmissionWaitCoverage({ runs, runEvidence: proof.runEvidence, comments: proof.comments, + issue: issue!, companyId: f.company.id, agentId: f.agent.id, stoppedRunId: stoppedForSessionReset, recoverySources: proof.sources }); + } return { + admissionReady, resolved: Boolean(resolved), status: resolved?.status, conversationState: resolved?.conversationState, - providerRunCount: runs.filter((run) => !isResetRun(run)).length, + providerRunCount: providerRuns().length, activeRuns: runs .filter((run) => ["queued", "running"].includes(run.status)) .map((run) => run.id), @@ -429,6 +471,7 @@ export async function runChatFlow(input: ChatFlowInput) { reject: (state) => state.failure ?? inconsistentIdle(state), accept: (state) => !state.failure && + state.admissionReady && state.resolved && state.providerRunCount >= minimumProviderRuns && state.activeRuns.length === 0 && @@ -491,7 +534,7 @@ export async function runChatFlow(input: ChatFlowInput) { 1, ); const initialId = issue!.id; - const before = runs.filter((run) => !isResetRun(run))[0]!; + const before = providerRuns()[0]!; await noTasks(); await page.reload({ waitUntil: "domcontentloaded", timeout: 60_000 }); if (caseId === "continuity-restart") { @@ -543,6 +586,7 @@ export async function runChatFlow(input: ChatFlowInput) { if (!isChatStopReady(events, caseId === "stop-startup-new-resume" ? "startup" : "active")) return false; } else if (!(events.length || (await readRunningChatLog(api, active.id))?.length)) return false; cancelledId = active.id; + stoppedForSessionReset = active.id; await input.evidence("chat-stop-boundary.json", { phase: caseId === "stop-startup-new-resume" ? "startup" : "active", runId: active.id, events, @@ -564,12 +608,14 @@ export async function runChatFlow(input: ChatFlowInput) { } const oldComments = await comments(); await sendChatMessage(page, "/new"); + // Stop can be acknowledged before workspace preservation retries. + // Observe the reset within the existing case deadline. await expect .poll( async () => (await api.get(chatPath)) .conversationSessionGeneration, - { timeout: 30_000 }, + { timeout: stopCase ? Math.max(1, input.deadlineAt - Date.now()) : 30_000 }, ) .toBe(1); await expect @@ -1016,13 +1062,19 @@ export async function runChatFlow(input: ChatFlowInput) { hiringEvidence = await refreshHiringEvidence!(); runs = hiringEvidence.runs; } + const finalAdmission = admissionAccounting ? await finalAdmissionEvidence() : undefined; + const finalComments = finalAdmission?.comments; + const admissionRunEvidence = finalAdmission?.runEvidence; + const admissionWaitSources = finalAdmission?.sources ?? []; + if (admissionAccounting) admissionWaits = verifyChatAdmissionWaitCoverage({ runs, runEvidence: admissionRunEvidence!, comments: finalComments!, issue: issue!, + companyId: f.company.id, agentId: f.agent.id, stoppedRunId: stoppedForSessionReset, recoverySources: admissionWaitSources }); assertChatFlowRunCount({ suiteId: execution.suite.id, task: execution.task, // Hiring must account for every actual company run, including unexpected resets. - runs: execution.suite.id === "hiring-templates" ? runs : runs.filter((run) => !isResetRun(run)), + runs: execution.suite.id === "hiring-templates" ? runs : providerRuns(), hiringEvidence, hiringApiState: execution.suite.id === "hiring-templates" ? hiringEvidence!.turnApiState : undefined, }); - for (const run of runs.filter((run) => !isResetRun(run))) { + for (const run of providerRuns()) { expect(run.runtimeMode).toBe(execution.profile.expectedRuntimeMode); expect(run.status).toBe( expectedStops.get(run.id) ?? (stopCase && run.status === "cancelled" @@ -1033,7 +1085,9 @@ export async function runChatFlow(input: ChatFlowInput) { await input.evidence("api-state.json", { issue: issue!, runs, + ...(admissionAccounting ? { admissionWaitSources } : {}), runGroups: { + ...(admissionAccounting ? { admissionWaits: admissionWaits.map(wait => wait.runId) } : {}), resets: runs.filter(isResetRun).map((run) => run.id), cancelled: runs .filter((run) => run.status === "cancelled") @@ -1041,18 +1095,16 @@ export async function runChatFlow(input: ChatFlowInput) { conversation: runs .filter( (run) => - !isResetRun(run) && run.contextSnapshot?.issueId === issue!.id, + providerRuns().some(provider => provider.id === run.id) && run.contextSnapshot?.issueId === issue!.id, ) .map((run) => run.id), handoff: runs .filter((run) => run.contextSnapshot?.issueId !== issue!.id) .map((run) => run.id), }, - comments: await comments(), + comments: finalComments ?? await comments(), activity: await api.get(`/api/issues/${issue!.id}/activity`), - runEvidence: await Promise.all( - runs.map((run) => collectChatRunEvidence(api, run)), - ), + runEvidence: admissionRunEvidence ?? await Promise.all(runs.map(run => collectChatRunEvidence(api, run))), }); await input.capture( "final-state", diff --git a/tests/runner-e2e/daytona-image-content.ts b/tests/runner-e2e/daytona-image-content.ts index 135bed034e0..9b69aa4f3be 100644 --- a/tests/runner-e2e/daytona-image-content.ts +++ b/tests/runner-e2e/daytona-image-content.ts @@ -42,7 +42,7 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [...new Set([ "packages/paperclip-runner/scripts/candidate-provider-pack.mjs", "packages/paperclip-runner/scripts/build-copilot-distribution.mjs", "packages/paperclip-runner/scripts/copilot-inner-distribution.mjs", - // ACPX generation recomputes the Copilot, Grok and Pi release attestations. + // ACPX generation recomputes the Copilot, Grok, Pi and Kimi release attestations. "packages/paperclip-runner/scripts/acpx-release-attestations.mjs", "packages/paperclip-runner/scripts/generate-acpx-profiles.mjs", "packages/paperclip-runner/scripts/materialize-copilot-binary.mjs", @@ -50,11 +50,22 @@ export const DAYTONA_IMAGE_INPUT_PATHS = [...new Set([ "packages/paperclip-runner/scripts/materialize-cursor-distribution.mjs", "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", "packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", + "packages/paperclip-runner/antigravity-distributions.json", + "packages/paperclip-runner/scripts/provision-antigravity.mjs", + "packages/paperclip-runner/scripts/antigravity-provisioner-layout.mjs", + "packages/paperclip-runner/scripts/materialize-antigravity-distribution.mjs", + "packages/paperclip-runner/scripts/patch-antigravity-runtime.mjs", + "packages/paperclip-runner/scripts/patch-antigravity-runtime.admission.py", "packages/paperclip-runner/scripts/provider-pack-runtime-tree.mjs", "packages/paperclip-runner/scripts/generate-acpx-sidecar-contract.mjs", "packages/paperclip-runner/scripts/generate-protocol-schema-module.mjs", "packages/paperclip-runner/scripts/materialize-pi-distribution.mjs", "packages/paperclip-runner/scripts/pi-distribution", + "packages/paperclip-runner/scripts/materialize-kimi-distribution.mjs", + "packages/paperclip-runner/scripts/patch-kimi-runtime.mjs", + "packages/paperclip-runner/scripts/kimi-instructions-runtime.mjs", + "packages/paperclip-runner/scripts/provision-kimi.mjs", + "packages/paperclip-runner/scripts/kimi-distribution", "packages/paperclip-runner/scripts/cursor-native-usage.mjs", "packages/paperclip-runner/scripts/cursor-native-usage-source.json", "packages/paperclip-runner/scripts/provision-cursor.mjs", @@ -95,7 +106,7 @@ export interface DaytonaImageContentOptions { export function normalizedDaytonaCandidateProviders(values: readonly string[]): string[] { const selected = [...values].sort(); - if (selected.some(value => !["cursor", "copilot", "pi"].includes(value)) + if (selected.some(value => !["cursor", "copilot", "pi", "kimi"].includes(value)) || new Set(selected).size !== selected.length) { throw new Error("Daytona candidate providers must be distinct known ACP profiles"); } @@ -311,7 +322,7 @@ if (invokedPath && import.meta.url === pathToFileURL(invokedPath).href) { const arguments_ = process.argv.slice(2); const candidateFlag = arguments_[0]; if (arguments_.length > 1 || (candidateFlag !== undefined && !candidateFlag.startsWith("--candidate-providers="))) { - throw new Error("Expected only --candidate-providers=cursor,copilot,pi"); + throw new Error("Expected only --candidate-providers=cursor,copilot,pi,kimi"); } const candidateProviders = candidateFlag?.slice("--candidate-providers=".length).split(",").filter(Boolean) ?? []; computeDaytonaImageContentId({ candidateProviders }) diff --git a/tests/runner-e2e/daytona-image.test.ts b/tests/runner-e2e/daytona-image.test.ts index 6eb230dd7fd..3d3e985d227 100644 --- a/tests/runner-e2e/daytona-image.test.ts +++ b/tests/runner-e2e/daytona-image.test.ts @@ -27,6 +27,7 @@ describe("runner E2E Daytona image contract", () => { "src/drivers/acpx/copilot-closure-pins.ts", "src/drivers/acpx/pi-closure-pins.ts", "src/drivers/acpx/pi-node-pins.ts", + "src/drivers/acpx/kimi-closure-pins.ts", "src/providers/grok/launcher.cjs", ].map(input => `packages/paperclip-runner/${input}`); @@ -65,6 +66,43 @@ describe("runner E2E Daytona image contract", () => { } }); + it("binds each Antigravity input consumed by the merged provider entrypoint builder", async () => { + const required = [ + "packages/paperclip-runner/antigravity-distributions.json", + "packages/paperclip-runner/scripts/provision-antigravity.mjs", + "packages/paperclip-runner/scripts/antigravity-provisioner-layout.mjs", + "packages/paperclip-runner/scripts/materialize-antigravity-distribution.mjs", + "packages/paperclip-runner/scripts/patch-antigravity-runtime.mjs", + "packages/paperclip-runner/scripts/patch-antigravity-runtime.admission.py", + ]; + const root = await mkdtemp(path.join(tmpdir(), "paperclip-daytona-builder-inputs-")); + const options = { + repositoryRoot: root, + inputPaths: DAYTONA_IMAGE_INPUT_PATHS.filter(input => required.includes(input)), + baseImages: [`example.test/base:1@sha256:${"a".repeat(64)}`], + frontendDigest: `sha256:${"c".repeat(64)}`, + } as const; + try { + for (const input of required) { + expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(input); + await mkdir(path.dirname(path.join(root, input)), { recursive: true }); + await writeFile(path.join(root, input), await readFile(path.join(repositoryRoot, input))); + } + const baseline = await computeDaytonaImageContentId(options); + for (const input of required) { + const original = await readFile(path.join(root, input)); + await writeFile(path.join(root, input), Buffer.concat([original, Buffer.from("\nchanged-build-input\n")])); + expect(await computeDaytonaImageContentId(options)).not.toBe(baseline); + await rm(path.join(root, input)); + await expect(computeDaytonaImageContentId(options)).rejects.toThrow(); + await writeFile(path.join(root, input), original); + expect(await computeDaytonaImageContentId(options)).toBe(baseline); + } + } finally { + await rm(root, { recursive: true, force: true }); + } + }); + it("keeps the qualified native Grok binary separate from the legacy command", async () => { const [dockerfile, packBuilder, runnerPackage] = await Promise.all([ readFile(path.join(repositoryRoot, "docker/daytona-runner/Dockerfile"), "utf8"), @@ -266,6 +304,11 @@ describe("runner E2E Daytona image contract", () => { "packages/paperclip-runner/runner/crates", "packages/paperclip-runner/scripts/materialize-pi-distribution.mjs", "packages/paperclip-runner/scripts/pi-distribution", + "packages/paperclip-runner/scripts/materialize-kimi-distribution.mjs", + "packages/paperclip-runner/scripts/patch-kimi-runtime.mjs", + "packages/paperclip-runner/scripts/kimi-instructions-runtime.mjs", + "packages/paperclip-runner/scripts/provision-kimi.mjs", + "packages/paperclip-runner/scripts/kimi-distribution", "packages/paperclip-runner/src", ]) { expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(requiredPath); @@ -351,6 +394,7 @@ describe("runner E2E Daytona image contract", () => { const baseline = await computeDaytonaImageContentId(options); const candidate = await computeDaytonaImageContentId({ ...options, candidateProviders: ["pi"] }); expect(candidate).not.toBe(baseline); + expect(await computeDaytonaImageContentId({ ...options, candidateProviders: ["kimi"] })).not.toBe(baseline); expect(await computeDaytonaImageContentId({ ...options, candidateProviders: ["copilot", "pi"] })) .toBe(await computeDaytonaImageContentId({ ...options, candidateProviders: ["pi", "copilot"] })); await expect(computeDaytonaImageContentId({ ...options, candidateProviders: ["pi", "pi"] })) @@ -408,9 +452,15 @@ describe("runner E2E Daytona image contract", () => { it.each([ "packages/paperclip-runner/scripts/cursor-runtime-patch.mjs", "packages/paperclip-runner/scripts/provider-pack-executable-shims.mjs", + "packages/paperclip-runner/scripts/materialize-kimi-distribution.mjs", + "packages/paperclip-runner/scripts/patch-kimi-runtime.mjs", + "packages/paperclip-runner/scripts/kimi-instructions-runtime.mjs", + "packages/paperclip-runner/scripts/provision-kimi.mjs", + "packages/paperclip-runner/scripts/kimi-distribution/package.json", + "packages/paperclip-runner/scripts/kimi-distribution/package-lock.json", ])("invalidates the image when %s changes", async (patchPath) => { const root = await mkdtemp(path.join(tmpdir(), "paperclip-provider-image-id-")); - expect(DAYTONA_IMAGE_INPUT_PATHS).toContain(patchPath); + expect(DAYTONA_IMAGE_INPUT_PATHS.some(inputPath => patchPath === inputPath || patchPath.startsWith(`${inputPath}/`))).toBe(true); const options = { repositoryRoot: root, inputPaths: [patchPath], diff --git a/tests/runner-e2e/evidence.ts b/tests/runner-e2e/evidence.ts index 97d3d26a94a..96157985355 100644 --- a/tests/runner-e2e/evidence.ts +++ b/tests/runner-e2e/evidence.ts @@ -9,6 +9,8 @@ import { lstat, } from "node:fs/promises"; import path from "node:path"; +import type { RunnerE2EResult } from "./types.js"; +import { assertRunnerE2EResultsPath } from "./results-root.js"; import { assertSecretFree, findSecretLeak, @@ -271,3 +273,33 @@ export async function packageEvidence(input: { files.push("evidence-manifest.json"); return { files, leaks, missing }; } + +export async function publishRunnerE2EAttemptEvidence(input: { + result: RunnerE2EResult; + privateDir: string; + uploadDir: string; + resultsRoot: string; + secrets: readonly string[]; +}): Promise { + let result = input.result; + const resultPath = path.join(input.privateDir, "result.json"); + const writeResult = () => writeFile(resultPath, + `${JSON.stringify(sanitizeJson(result, input.secrets), null, 2)}\n`, "utf8"); + await assertRunnerE2EResultsPath(input.resultsRoot, input.uploadDir); + await writeResult(); + const evidence = await packageEvidence({ ...input, expectPassScreenshot: result.status === "passed" }); + if (evidence.leaks.length > 0 || evidence.missing.length > 0) { + result = { + ...result, + status: "failed", + failureClass: evidence.leaks.length > 0 ? "secret_leak" : "permanent_infrastructure", + error: evidence.leaks.length > 0 + ? `Secret leak rejected from evidence: ${evidence.leaks.map(leak => leak.file).join(", ")}` + : `Required evidence missing: ${evidence.missing.join(", ")}`, + }; + await writeResult(); + await assertRunnerE2EResultsPath(input.resultsRoot, input.uploadDir); + await packageEvidence({ ...input, expectPassScreenshot: false }); + } + return result; +} diff --git a/tests/runner-e2e/fixtures/kimi-context-command-v1.json b/tests/runner-e2e/fixtures/kimi-context-command-v1.json new file mode 100644 index 00000000000..a44b46dd457 --- /dev/null +++ b/tests/runner-e2e/fixtures/kimi-context-command-v1.json @@ -0,0 +1,284 @@ +{ + "schema": "paperclip.e2e.kimi-context-real-producer-calibration.v1", + "identity": { + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "semantic-event-1", + "eventType": "harness.diagnostic", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 1, + "sourceEventId": "semantic-event-1", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "harness.diagnostic", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "code": "synthetic-prior-frame" + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "semantic-event-2", + "eventType": "harness.diagnostic", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 2, + "sourceEventId": "semantic-event-2", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "harness.diagnostic", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "code": "synthetic-prior-frame" + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "semantic-event-3", + "eventType": "harness.diagnostic", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 3, + "sourceEventId": "semantic-event-3", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "harness.diagnostic", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "code": "synthetic-prior-frame" + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "semantic-event-4", + "eventType": "semantic_tool.input", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 4, + "sourceEventId": "semantic-event-4", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "semantic_tool.input", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "semantic_tool": { + "schema": "paperclip.prp.semantic_tool.v1", + "schemaVersion": 1, + "phase": "input", + "callId": "native-mcp-context-1", + "operationId": "get_task_context", + "correlation": { + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "idempotencyKey": null, + "content": { + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "redactionDisposition": "digest_only", + "references": [] + }, + "input": {} + } + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ], + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 1, + "type": "semantic_tool.result", + "issuedAt": "2026-10-29T18:41:45.919Z", + "payload": { + "callId": "native-mcp-context-1", + "operationId": "get_task_context", + "sourceEventId": "semantic-event-4", + "sourceEventType": "semantic_tool.input", + "correlation": { + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "inputDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "result": { + "company": { + "id": "synthetic-company" + }, + "actor": { + "id": "synthetic-agent" + }, + "activeTask": { + "id": "synthetic-task", + "description": "synthetic operator continuation" + } + }, + "isError": false + }, + "status": "completed", + "result": { + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 1, + "commandType": "semantic_tool.result", + "status": "completed", + "result": { + "status": "delivered", + "callId": "native-mcp-context-1" + } + } + } + ], + "projected": { + "schema": "paperclip.e2e.kimi-context-authority.v1", + "companyId": "synthetic-company", + "issueId": "synthetic-task", + "agentId": "synthetic-agent", + "environmentId": "synthetic-env", + "runId": "00000000-0000-4000-8000-000000000001", + "sessionId": "session-test-1", + "sourceInstanceId": "runner-test-1", + "turnId": "turn-test-1", + "observerLeaseId": "synthetic-lease", + "actionSha256": "sha256:4dfc4faade4ee2bae6ac1793a1856d660a97401da6ea4d642d79f759f508e0b7", + "callId": "native-mcp-context-1", + "inputSourceSeq": 4, + "inputSourceEventId": "semantic-event-4", + "inputSha256": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "inputEventSha256": "sha256:fd6478ab28148b294051aaa5471a8adf043812b049f60958bbb8849793c3d104", + "resultSha256": "sha256:7a08899045e679406d58c8ceb4cd1c2bfbf59d83264494c819685a435127284e", + "correlationSha256": "sha256:e7b944a6f5b4b8ebca18e2125b06a9e76dc040cc007fe9bda4706fe46eb6d30d", + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 1, + "commandSha256": "sha256:b734d0d4f57a3945b6b00b50f51bf5c029f88282a4fbdef9c4160c45576f605e", + "deliverySha256": "sha256:656035050e8a0a5dbd5f0260d7b075c81c10cdbd810b26b60792013ce3ab1e5f", + "checkpointIdentitySha256": "sha256:449ca73c797d0a5c63ae11086d0cd64e83296ad8e6bfcce83ee21254c406b3ce", + "authenticatedConnectionCount": 1, + "highestRecordedSourceSeq": 4 + }, + "checkpoint": { + "runId": "00000000-0000-4000-8000-000000000001", + "recoveryEvidence": { + "runnerInstanceId": "runner-test-1", + "normalizedSessionId": "session-test-1", + "authenticatedConnectionCount": 1, + "highestRecordedSourceSeq": 4 + }, + "records": [ + [ + "command:1", + "{\"schema\":\"paperclip.prp.command.v1\",\"commandId\":\"command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3\",\"controllerSeq\":1,\"type\":\"semantic_tool.result\",\"issuedAt\":\"2026-10-29T18:41:45.919Z\",\"payload\":{\"callId\":\"native-mcp-context-1\",\"operationId\":\"get_task_context\",\"sourceEventId\":\"semantic-event-4\",\"sourceEventType\":\"semantic_tool.input\",\"correlation\":{\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\"},\"inputDigest\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"result\":{\"company\":{\"id\":\"synthetic-company\"},\"actor\":{\"id\":\"synthetic-agent\"},\"activeTask\":{\"id\":\"synthetic-task\",\"description\":\"synthetic operator continuation\"}},\"isError\":false},\"status\":\"completed\",\"result\":{\"commandId\":\"command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3\",\"controllerSeq\":1,\"commandType\":\"semantic_tool.result\",\"status\":\"completed\",\"result\":{\"status\":\"delivered\",\"callId\":\"native-mcp-context-1\"}}}" + ], + [ + "event:1", + "{\"sourceSeq\":1,\"sourceEventId\":\"semantic-event-1\",\"eventType\":\"harness.diagnostic\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":1,\"sourceEventId\":\"semantic-event-1\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"harness.diagnostic\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"code\":\"synthetic-prior-frame\"}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:2", + "{\"sourceSeq\":2,\"sourceEventId\":\"semantic-event-2\",\"eventType\":\"harness.diagnostic\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":2,\"sourceEventId\":\"semantic-event-2\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"harness.diagnostic\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"code\":\"synthetic-prior-frame\"}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:3", + "{\"sourceSeq\":3,\"sourceEventId\":\"semantic-event-3\",\"eventType\":\"harness.diagnostic\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":3,\"sourceEventId\":\"semantic-event-3\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"harness.diagnostic\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"code\":\"synthetic-prior-frame\"}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:4", + "{\"sourceSeq\":4,\"sourceEventId\":\"semantic-event-4\",\"eventType\":\"semantic_tool.input\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":4,\"sourceEventId\":\"semantic-event-4\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"semantic_tool.input\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"semantic_tool\":{\"schema\":\"paperclip.prp.semantic_tool.v1\",\"schemaVersion\":1,\"phase\":\"input\",\"callId\":\"native-mcp-context-1\",\"operationId\":\"get_task_context\",\"correlation\":{\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\"},\"idempotencyKey\":null,\"content\":{\"digest\":\"sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"redactionDisposition\":\"digest_only\",\"references\":[]},\"input\":{}}}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ] + ] + } +} diff --git a/tests/runner-e2e/fixtures/kimi-context-command-v2.json b/tests/runner-e2e/fixtures/kimi-context-command-v2.json new file mode 100644 index 00000000000..80b0967307e --- /dev/null +++ b/tests/runner-e2e/fixtures/kimi-context-command-v2.json @@ -0,0 +1,410 @@ +{ + "schema": "paperclip.e2e.kimi-context-real-producer-calibration.v1", + "identity": { + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "nativeTurnId": "native-turn-test", + "requestId": "context-request", + "toolCallId": "0:synthetic-context-call", + "action": "synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation ", + "publication": { + "observerLeaseId": "synthetic-lease", + "companyId": "synthetic-company", + "agentId": "synthetic-agent", + "issueId": "synthetic-task", + "environmentId": "synthetic-env", + "runId": "00000000-0000-4000-8000-000000000001", + "sessionId": "session-test-1", + "sourceInstanceId": "runner-test-1", + "turnId": "native-turn-test", + "requestId": "context-request", + "toolCallId": "0:synthetic-context-call", + "actionSha256": "sha256:d074470e2899007f3814a855423002b3fff1f3290254826adfcf076e02ee8e0a" + }, + "committedEvents": [ + { + "sourceSeq": 1, + "sourceEventId": "semantic-event-1", + "eventType": "turn.started", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 1, + "sourceEventId": "semantic-event-1", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "turn.started", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "provider": "acpx", + "providerTurnId": "native-turn-test", + "status": "running", + "turn": { + "id": "native-turn-test" + } + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 2, + "sourceEventId": "semantic-event-2", + "eventType": "runtime_request.created", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 2, + "sourceEventId": "semantic-event-2", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "runtime_request.created", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "request": { + "schema": "paperclip.runtime_request.v2", + "requestKind": "permission_approval", + "type": "permission", + "status": "pending", + "requestId": "context-request", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "prompt": "mcp__paperclip__get_task_context", + "origin": { + "adapter": "acpx-runtime-sidecar", + "provider": "kimi", + "method": "session/request_permission" + }, + "choices": [ + { + "key": "accept", + "label": "Allow once" + }, + { + "key": "decline", + "label": "Reject" + }, + { + "key": "cancel", + "label": "Cancel" + } + ], + "details": { + "toolCallId": "0:synthetic-context-call" + } + } + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 3, + "sourceEventId": "semantic-event-3", + "eventType": "runtime_request.resolved", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 3, + "sourceEventId": "semantic-event-3", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "runtime_request.resolved", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "requestId": "context-request", + "requestKind": "permission_approval", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "action": "accept", + "status": "resolved", + "provider": "acpx" + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + }, + { + "sourceSeq": 4, + "sourceEventId": "semantic-event-4", + "eventType": "semantic_tool.input", + "priority": 0, + "envelope": { + "protocol": "paperclip.runner", + "version": 1, + "kind": "event", + "runnerInstanceId": "runner-test-1", + "environmentLeaseId": "environment-test-1", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "payload": { + "sourceSeq": 4, + "sourceEventId": "semantic-event-4", + "sourceInstanceId": "runner-test-1", + "sourceKind": "runner", + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1", + "eventType": "semantic_tool.input", + "schema": "paperclip.prp.event.v1", + "schemaVersion": 1, + "priority": 0, + "emittedAt": "2026-08-25T18:00:00.000Z", + "payload": { + "semantic_tool": { + "schema": "paperclip.prp.semantic_tool.v1", + "schemaVersion": 1, + "phase": "input", + "callId": "native-mcp-context-1", + "operationId": "get_task_context", + "correlation": { + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "idempotencyKey": null, + "content": { + "digest": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "redactionDisposition": "digest_only", + "references": [] + }, + "input": {} + } + } + } + }, + "deliveryCount": 1, + "logicalEffectCount": 1 + } + ], + "commands": [ + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_prp_00000001", + "controllerSeq": 1, + "type": "turn.start", + "issuedAt": "2026-10-29T20:32:36.009Z", + "payload": { + "text": "synthetic scoped task", + "turnId": "native-turn-test" + }, + "status": "completed", + "result": { + "commandId": "command_prp_00000001", + "controllerSeq": 1, + "commandType": "turn.start", + "status": "completed", + "result": { + "status": "accepted", + "providerTurnId": "native-turn-test" + } + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_prp_00000002", + "controllerSeq": 2, + "type": "request.resolve", + "issuedAt": "2026-10-29T20:32:36.067Z", + "payload": { + "requestId": "context-request", + "turnId": "turn-test-1", + "resolution": { + "action": "accept" + } + }, + "status": "completed", + "result": { + "commandId": "command_prp_00000002", + "controllerSeq": 2, + "commandType": "request.resolve", + "status": "completed", + "result": { + "status": "delivered", + "requestId": "context-request" + } + } + }, + { + "schema": "paperclip.prp.command.v1", + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 3, + "type": "semantic_tool.result", + "issuedAt": "2026-10-29T20:32:36.177Z", + "payload": { + "callId": "native-mcp-context-1", + "operationId": "get_task_context", + "sourceEventId": "semantic-event-4", + "sourceEventType": "semantic_tool.input", + "correlation": { + "runId": "00000000-0000-4000-8000-000000000001", + "normalizedSessionId": "session-test-1", + "turnId": "turn-test-1", + "itemId": "item-test-1" + }, + "inputDigest": "44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "result": { + "company": { + "id": "synthetic-company" + }, + "actor": { + "id": "synthetic-agent" + }, + "activeTask": { + "id": "synthetic-task", + "description": "synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation " + } + }, + "isError": false + }, + "status": "completed", + "result": { + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 3, + "commandType": "semantic_tool.result", + "status": "completed", + "result": { + "status": "delivered", + "callId": "native-mcp-context-1" + } + } + } + ], + "projected": { + "schema": "paperclip.e2e.kimi-context-authority.v2", + "companyId": "synthetic-company", + "issueId": "synthetic-task", + "agentId": "synthetic-agent", + "environmentId": "synthetic-env", + "runId": "00000000-0000-4000-8000-000000000001", + "sessionId": "session-test-1", + "sourceInstanceId": "runner-test-1", + "turnId": "native-turn-test", + "observerLeaseId": "synthetic-lease", + "executionWorkspaceId": "environment-test-1", + "controllerTurnId": "turn-test-1", + "controllerItemId": "item-test-1", + "requestId": "context-request", + "toolCallId": "0:synthetic-context-call", + "requestCreatedSourceSeq": 2, + "requestResolvedSourceSeq": 3, + "requestCreatedSha256": "sha256:ace2ee97798265b4cdc5a6e99c070925494f32cc6bf0bfa5880e512d5fd42adc", + "requestResolvedSha256": "sha256:81ad2e894058d8aedf8159165d1a4ceac804119abd9cb39d9046161817fdf135", + "turnStartSha256": "sha256:197cfbf7b3672e3c3eba8a76f63cf3b6f9936d87eff4818a7861c7faeb9ae9b9", + "resolutionCommandSha256": "sha256:90b3828448dd987e9d514816d56c3ef968c5077e9a896f80c2d71542434930a0", + "actionSha256": "sha256:d074470e2899007f3814a855423002b3fff1f3290254826adfcf076e02ee8e0a", + "callId": "native-mcp-context-1", + "inputSourceSeq": 4, + "inputSourceEventId": "semantic-event-4", + "inputSha256": "sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a", + "inputEventSha256": "sha256:fd6478ab28148b294051aaa5471a8adf043812b049f60958bbb8849793c3d104", + "resultSha256": "sha256:2b8100a593ee7ad9e0cb2a155c4dd8aa996b6a111034bd1ef64bca0bdb056ca2", + "correlationSha256": "sha256:e7b944a6f5b4b8ebca18e2125b06a9e76dc040cc007fe9bda4706fe46eb6d30d", + "commandId": "command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3", + "controllerSeq": 3, + "commandSha256": "sha256:89dc8ed35644e6e991740d5e844208eb8265039cfd07607ae8cb65bd6b1d6e05", + "deliverySha256": "sha256:8c8d8c89917f99ca52e3982ed7ba36efbdffca1d09a516098f323bbac3e7e367", + "checkpointIdentitySha256": "sha256:449ca73c797d0a5c63ae11086d0cd64e83296ad8e6bfcce83ee21254c406b3ce", + "authenticatedConnectionCount": 1, + "highestRecordedSourceSeq": 4 + }, + "checkpoint": { + "runId": "00000000-0000-4000-8000-000000000001", + "recoveryEvidence": { + "runnerInstanceId": "runner-test-1", + "normalizedSessionId": "session-test-1", + "authenticatedConnectionCount": 1, + "highestRecordedSourceSeq": 4 + }, + "records": [ + [ + "command:1", + "{\"schema\":\"paperclip.prp.command.v1\",\"commandId\":\"command_prp_00000001\",\"controllerSeq\":1,\"type\":\"turn.start\",\"issuedAt\":\"2026-10-29T20:32:36.009Z\",\"payload\":{\"text\":\"synthetic scoped task\",\"turnId\":\"native-turn-test\"},\"status\":\"completed\",\"result\":{\"commandId\":\"command_prp_00000001\",\"controllerSeq\":1,\"commandType\":\"turn.start\",\"status\":\"completed\",\"result\":{\"status\":\"accepted\",\"providerTurnId\":\"native-turn-test\"}}}" + ], + [ + "command:2", + "{\"schema\":\"paperclip.prp.command.v1\",\"commandId\":\"command_prp_00000002\",\"controllerSeq\":2,\"type\":\"request.resolve\",\"issuedAt\":\"2026-10-29T20:32:36.067Z\",\"payload\":{\"requestId\":\"context-request\",\"turnId\":\"turn-test-1\",\"resolution\":{\"action\":\"accept\"}},\"status\":\"completed\",\"result\":{\"commandId\":\"command_prp_00000002\",\"controllerSeq\":2,\"commandType\":\"request.resolve\",\"status\":\"completed\",\"result\":{\"status\":\"delivered\",\"requestId\":\"context-request\"}}}" + ], + [ + "command:3", + "{\"schema\":\"paperclip.prp.command.v1\",\"commandId\":\"command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3\",\"controllerSeq\":3,\"type\":\"semantic_tool.result\",\"issuedAt\":\"2026-10-29T20:32:36.177Z\",\"payload\":{\"callId\":\"native-mcp-context-1\",\"operationId\":\"get_task_context\",\"sourceEventId\":\"semantic-event-4\",\"sourceEventType\":\"semantic_tool.input\",\"correlation\":{\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\"},\"inputDigest\":\"44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"result\":{\"company\":{\"id\":\"synthetic-company\"},\"actor\":{\"id\":\"synthetic-agent\"},\"activeTask\":{\"id\":\"synthetic-task\",\"description\":\"synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation synthetic operator continuation \"}},\"isError\":false},\"status\":\"completed\",\"result\":{\"commandId\":\"command_tool_a840ed4f1dc2c14279f748c5dd1ea8b3\",\"controllerSeq\":3,\"commandType\":\"semantic_tool.result\",\"status\":\"completed\",\"result\":{\"status\":\"delivered\",\"callId\":\"native-mcp-context-1\"}}}" + ], + [ + "event:1", + "{\"sourceSeq\":1,\"sourceEventId\":\"semantic-event-1\",\"eventType\":\"turn.started\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":1,\"sourceEventId\":\"semantic-event-1\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"turn.started\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"provider\":\"acpx\",\"providerTurnId\":\"native-turn-test\",\"status\":\"running\",\"turn\":{\"id\":\"native-turn-test\"}}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:2", + "{\"sourceSeq\":2,\"sourceEventId\":\"semantic-event-2\",\"eventType\":\"runtime_request.created\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":2,\"sourceEventId\":\"semantic-event-2\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"runtime_request.created\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"request\":{\"schema\":\"paperclip.runtime_request.v2\",\"requestKind\":\"permission_approval\",\"type\":\"permission\",\"status\":\"pending\",\"requestId\":\"context-request\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"prompt\":\"mcp__paperclip__get_task_context\",\"origin\":{\"adapter\":\"acpx-runtime-sidecar\",\"provider\":\"kimi\",\"method\":\"session/request_permission\"},\"choices\":[{\"key\":\"accept\",\"label\":\"Allow once\"},{\"key\":\"decline\",\"label\":\"Reject\"},{\"key\":\"cancel\",\"label\":\"Cancel\"}],\"details\":{\"toolCallId\":\"0:synthetic-context-call\"}}}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:3", + "{\"sourceSeq\":3,\"sourceEventId\":\"semantic-event-3\",\"eventType\":\"runtime_request.resolved\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":3,\"sourceEventId\":\"semantic-event-3\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"runtime_request.resolved\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"requestId\":\"context-request\",\"requestKind\":\"permission_approval\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"action\":\"accept\",\"status\":\"resolved\",\"provider\":\"acpx\"}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ], + [ + "event:4", + "{\"sourceSeq\":4,\"sourceEventId\":\"semantic-event-4\",\"eventType\":\"semantic_tool.input\",\"priority\":0,\"envelope\":{\"protocol\":\"paperclip.runner\",\"version\":1,\"kind\":\"event\",\"runnerInstanceId\":\"runner-test-1\",\"environmentLeaseId\":\"environment-test-1\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"payload\":{\"sourceSeq\":4,\"sourceEventId\":\"semantic-event-4\",\"sourceInstanceId\":\"runner-test-1\",\"sourceKind\":\"runner\",\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\",\"eventType\":\"semantic_tool.input\",\"schema\":\"paperclip.prp.event.v1\",\"schemaVersion\":1,\"priority\":0,\"emittedAt\":\"2026-08-25T18:00:00.000Z\",\"payload\":{\"semantic_tool\":{\"schema\":\"paperclip.prp.semantic_tool.v1\",\"schemaVersion\":1,\"phase\":\"input\",\"callId\":\"native-mcp-context-1\",\"operationId\":\"get_task_context\",\"correlation\":{\"runId\":\"00000000-0000-4000-8000-000000000001\",\"normalizedSessionId\":\"session-test-1\",\"turnId\":\"turn-test-1\",\"itemId\":\"item-test-1\"},\"idempotencyKey\":null,\"content\":{\"digest\":\"sha256:44136fa355b3678a1146ad16f7e8649e94fb4fc21fe77e8310c060f61caaff8a\",\"redactionDisposition\":\"digest_only\",\"references\":[]},\"input\":{}}}}},\"deliveryCount\":1,\"logicalEffectCount\":1}" + ] + ] + } +} diff --git a/tests/runner-e2e/harness-env-admission.test.ts b/tests/runner-e2e/harness-env-admission.test.ts index 29623e28df9..01b2cd5f70d 100644 --- a/tests/runner-e2e/harness-env-admission.test.ts +++ b/tests/runner-e2e/harness-env-admission.test.ts @@ -25,3 +25,13 @@ it("retains exact, explicit admission when a future profile is pending", async ( expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...copilot, suite: { ...copilot.suite, id: "unrelated-suite" } }])).toThrow("explicit"); expect(() => buildRunnerE2EProcessEnvironment({}, [copilot, { ...copilot, profile: { ...copilot.profile, model: "conflicting-model" } }])).toThrow("Conflicting"); }); + +it("strips ambient candidate startup records and never authorizes pending Kimi through the old model allowlist", async () => { + const { buildRunnerE2EProcessEnvironment, buildPaperclipServerEnvironment } = await import("./harness-env.js"); + const kimi = runnerMatrix.find(row => row.suite.id === "kimi-qualification")!; + const ambient = { PAPERCLIP_ACPX_CANDIDATE_ADMISSION: "forged", PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "forged" }; + const env = buildRunnerE2EProcessEnvironment(ambient, [kimi]); + expect(env.PAPERCLIP_ACPX_CANDIDATE_ADMISSION).toBeUndefined(); + expect(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined(); + expect(buildPaperclipServerEnvironment(ambient).PAPERCLIP_ACPX_CANDIDATE_ADMISSION).toBeUndefined(); +}); diff --git a/tests/runner-e2e/harness-env.ts b/tests/runner-e2e/harness-env.ts index 853786df670..b33e3f2125b 100644 --- a/tests/runner-e2e/harness-env.ts +++ b/tests/runner-e2e/harness-env.ts @@ -10,6 +10,7 @@ import type { MatrixExecution } from "./types.js"; const DATABASE_KEYS = ["DATABASE_URL", "DATABASE_MIGRATION_URL"] as const; const AMBIENT_PAPERCLIP_CREDENTIAL_KEYS = [ "PAPERCLIP_API_KEY", + "PAPERCLIP_ACPX_CANDIDATE_ADMISSION", "PAPERCLIP_AGENT_API_KEY", "PAPERCLIP_TASK_BRIDGE_TOKEN", "PAPERCLIP_E2E_GATEWAY_TOKEN", @@ -33,7 +34,7 @@ const AMBIENT_EXTERNAL_STATE_KEYS = [ "PAPERCLIP_STORAGE_S3_PREFIX", "PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE", ] as const; -const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|BRAINTRUST|DAYTONA|XAI|GROK|CURSOR|COPILOT|GITHUB|GH)(?:_|$)/; +const PROVIDER_SECRET_KEY = /^(?:OPENAI|ANTHROPIC|OPENROUTER|BRAINTRUST|DAYTONA|XAI|GROK|KIMI|CURSOR|COPILOT|GITHUB|GH)(?:_|$)/; export function applyEvalGatewayEnvironment(env: NodeJS.ProcessEnv, credential: string, gateway: { token: string; baseUrl: string; phasePath: string; observability?: { token: string; baseUrl: string }; @@ -149,11 +150,14 @@ export function buildRunnerE2EProcessEnvironment( // Discard ambient admission. Only explicit candidate cells authorize the // exact model in their isolated server; credentials still use company secrets. delete result.PAPERCLIP_RUNNER_ACPX_QUALIFICATION; + delete result.PAPERCLIP_ACPX_CANDIDATE_ADMISSION; const candidates = new Map(); for (const execution of executions) { const agent = execution.profile.qualificationCandidate; if (!agent) continue; - const admittedSuite = execution.suite.id === "extended-harnesses" + const admittedSuite = agent === "kimi" + ? execution.suite.id === "kimi-qualification" && execution.profile.id === "runner-acpx-kimi" + : execution.suite.id === "extended-harnesses" || execution.suite.id === "rich-acp-warm-continuity" || (execution.suite.id === "pi-native" && agent === "pi") || (execution.suite.id === "pi-controls" && agent === "pi") @@ -166,6 +170,7 @@ export function buildRunnerE2EProcessEnvironment( if (QUALIFIED_ACPX_PROFILES[agent].qualificationStatus !== "pending") { continue; } + if (agent === "kimi") continue; const prior = candidates.get(agent); if (prior !== undefined && prior !== execution.profile.model) throw new Error("Conflicting candidate models"); candidates.set(agent, execution.profile.model); diff --git a/tests/runner-e2e/journey-features.ts b/tests/runner-e2e/journey-features.ts index b0259b504d3..dc64adb43d7 100644 --- a/tests/runner-e2e/journey-features.ts +++ b/tests/runner-e2e/journey-features.ts @@ -27,7 +27,7 @@ associate(["antigravity-agent-journey"], { "saved-subscription-two-turn": ["agents/hire", "agents/instructions", "runs-adapters/validation", "runs-adapters/session", "tasks/create", "tasks/completion", "documents-artifacts/attachments", "documents-artifacts/work-products"], }); -associate(["core-compatibility", "openrouter-model-breadth", "extended-harnesses", "grok-qualification", "grok-subscription-qualification"], { +associate(["core-compatibility", "openrouter-model-breadth", "extended-harnesses", "grok-qualification", "grok-subscription-qualification", "kimi-qualification"], { "message-marker": ["runs-adapters/runs"], "hello-complete": ["runs-adapters/runs"], "plan-revise-accept": ["questions-and-approvals/plan-review"], @@ -35,7 +35,7 @@ associate(["core-compatibility", "openrouter-model-breadth", "extended-harnesses "ask-question": ["questions-and-approvals/questions"], "question-resume-complete": ["questions-and-approvals/questions", "questions-and-approvals/continuation"], }); -associate(["local-session-integrity", "extended-harnesses", "grok-qualification", "grok-subscription-qualification"], { +associate(["local-session-integrity", "extended-harnesses", "grok-qualification", "grok-subscription-qualification", "kimi-qualification"], { "structured-question-resume": ["questions-and-approvals/continuation", "runs-adapters/session"], "structured-question-restart-resume": ["questions-and-approvals/continuation", "runs-adapters/session"], }); @@ -64,7 +64,7 @@ associate(["confirmation-replies"], { "confirmation-ambiguous": ["questions-and-approvals/confirmations", "questions-and-approvals/audience"], "unanswered-question-return": ["questions-and-approvals/dismiss-and-reopen", "questions-and-approvals/continuation"], }); -associate(["agent-chat", "agent-chat-hardening", "grok-qualification", "grok-subscription-qualification", "stock-harness"], { +associate(["agent-chat", "agent-chat-hardening", "grok-qualification", "grok-subscription-qualification", "kimi-qualification", "stock-harness"], { "create-backlog": ["tasks/create"], "reassign-task": ["tasks/properties"], "continuity-restart": ["agent-chat/history", "runs-adapters/session"], @@ -106,6 +106,11 @@ associate(["everyday-workflows", "native-connection-guidance", "grok-qualificati "create-skill-studio": ["skills/authoring"], }); associate(["hiring-templates"], { "hire-coder-template-reuse": ["agents/hire", "task-coordination/delegation"] }); +associate(["kimi-qualification"], { + "file-edit-validate": ["documents-artifacts/work-products"], + "native-write-permission-stop": ["steering/stop-and-pause", "questions-and-approvals/tool-review"], + "native-write-provider-loss": ["recovery/diagnosis", "questions-and-approvals/tool-review"], +}); associate(["task-workspaces"], { "task-directory-repository-resume": ["workspaces/binding", "workspaces/provisioning", "projects/repositories"], }); diff --git a/tests/runner-e2e/kimi-context-checkpoint.ts b/tests/runner-e2e/kimi-context-checkpoint.ts new file mode 100644 index 00000000000..54ce03e8ede --- /dev/null +++ b/tests/runner-e2e/kimi-context-checkpoint.ts @@ -0,0 +1,150 @@ +import { createHash } from "node:crypto"; +import { readFile, realpath } from "node:fs/promises"; +import path from "node:path"; +import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; +import { inspectDurablePrpCheckpointHeader, type DurablePrpCheckpoint } from "../../packages/paperclip-runner/src/control-plane/durable-prp-control-plane.js"; +import { parseNativeExecutionInput } from "../../packages/paperclip-runner/src/contracts/native-execution.js"; +import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; +import type { KimiContextPublication } from "./kimi-context-rendezvous.js"; + +type Row = Record; +const rec = (x: unknown): Row => x !== null && typeof x === "object" && !Array.isArray(x) ? x as Row : {}; +const bytes = (x: string) => createHash("sha256").update(x).digest("hex"); +const hash = (x: unknown) => `sha256:${bytes(canonicalJson(x))}`; +const demand = (x: unknown): void => { if (!x) throw new Error("Kimi context checkpoint: exact bound result delivery unavailable"); }; +export interface KimiContextAuthority { + schema: "paperclip.e2e.kimi-context-authority.v2"; + companyId: string; issueId: string; agentId: string; environmentId: string; runId: string; + sessionId: string; sourceInstanceId: string; turnId: string; + observerLeaseId: string; executionWorkspaceId: string; controllerTurnId: string; controllerItemId: string; requestId: string; toolCallId: string; + requestCreatedSourceSeq: number; requestResolvedSourceSeq: number; requestCreatedSha256: string; requestResolvedSha256: string; turnStartSha256: string; resolutionCommandSha256: string; + actionSha256: string; callId: string; inputSourceSeq: number; inputSourceEventId: string; + inputSha256: string; inputEventSha256: string; resultSha256: string; correlationSha256: string; + commandId: string; controllerSeq: number; commandSha256: string; deliverySha256: string; + checkpointIdentitySha256: string; authenticatedConnectionCount: number; highestRecordedSourceSeq: number; +} +export function projectKimiContextCheckpoint(checkpoint: DurablePrpCheckpoint, p: KimiContextPublication, executionWorkspaceId: string): KimiContextAuthority | undefined { + demand(typeof p.observerLeaseId === "string" && p.observerLeaseId.length > 0 && checkpoint.runId === p.runId && checkpoint.records.size <= 20_000); + const identity = inspectDurablePrpCheckpointHeader(checkpoint.records.get("header") ?? "null"), evidence = checkpoint.recoveryEvidence; + demand(identity && identity.runId === p.runId && identity.normalizedSessionId === p.sessionId && identity.runnerInstanceId === p.sourceInstanceId + && typeof executionWorkspaceId === "string" && executionWorkspaceId.length > 0 && identity.environmentLeaseId === executionWorkspaceId && evidence.runnerInstanceId === p.sourceInstanceId && evidence.normalizedSessionId === p.sessionId + && Number.isSafeInteger(evidence.authenticatedConnectionCount) && evidence.authenticatedConnectionCount > 0 + && Number.isSafeInteger(evidence.highestRecordedSourceSeq) && evidence.highestRecordedSourceSeq > 0); + const events: Row[] = [], commands: Row[] = [], requests: Row[] = [], turnStarts: Row[] = [], resolutions: Row[] = []; let total = 0; + for (const [key, value] of checkpoint.records) { + if (!/^(event|command):[0-9]+$/.test(key)) continue; + total += Buffer.byteLength(value); demand(total <= 32 * 1024 * 1024); + const row = rec(JSON.parse(value)); + if (key.startsWith("event:") && row.eventType === "semantic_tool.input") events.push(row); + if (key.startsWith("command:") && row.type === "semantic_tool.result") commands.push(row); + if (key.startsWith("event:") && ["runtime_request.created", "runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(row.eventType) + && (rec(rec(rec(row.envelope).payload).payload).requestId === p.requestId || rec(rec(rec(rec(row.envelope).payload).payload).request).requestId === p.requestId)) requests.push(row); + if (key.startsWith("command:") && row.type === "turn.start") turnStarts.push(row); + if (key.startsWith("command:") && row.type === "request.resolve" && rec(row.payload).requestId === p.requestId) resolutions.push(row); + } + const controllerEvent = (saved: Row) => { + const envelope = rec(saved.envelope), event = rec(envelope.payload); + demand(envelope.protocol === "paperclip.runner" && envelope.kind === "event" && envelope.runId === p.runId + && envelope.normalizedSessionId === p.sessionId && envelope.runnerInstanceId === p.sourceInstanceId && envelope.turnId === identity!.turnId + && envelope.itemId === identity!.itemId && envelope.environmentLeaseId === executionWorkspaceId + && event.schema === "paperclip.prp.event.v1" && event.schemaVersion === 1 && event.sourceKind === "runner" + && event.runId === p.runId && event.normalizedSessionId === p.sessionId && event.sourceInstanceId === p.sourceInstanceId + && event.turnId === identity!.turnId && event.itemId === identity!.itemId && saved.sourceSeq === event.sourceSeq + && saved.sourceEventId === event.sourceEventId && saved.eventType === event.eventType && saved.logicalEffectCount === 1 + && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0 && event.sourceSeq <= evidence.highestRecordedSourceSeq); + return event; + }; + const completed = (command: Row, status: string) => { + const result = rec(command.result); + demand(command.schema === "paperclip.prp.command.v1" && command.status === "completed" && Number.isSafeInteger(command.controllerSeq) && command.controllerSeq > 0 + && result.commandId === command.commandId && result.controllerSeq === command.controllerSeq && result.commandType === command.type + && result.status === "completed" && result.result?.status === status); + return rec(result.result); + }; + demand(turnStarts.length <= 1 && resolutions.length <= 1 && requests.length <= 2); + if (!turnStarts.length || requests.length < 2 || !resolutions.length) return undefined; + const turnStart = turnStarts[0]!, accepted = completed(turnStart, "accepted"); + demand(turnStart.payload?.turnId === p.turnId && accepted.providerTurnId === p.turnId); + const created = requests.find(x => x.eventType === "runtime_request.created"), resolved = requests.find(x => x.eventType === "runtime_request.resolved"); + demand(created && resolved); + const request = rec(rec(controllerEvent(created!).payload).request), resolution = rec(controllerEvent(resolved!).payload); + demand(request.schema === "paperclip.runtime_request.v2" && request.requestKind === "permission_approval" && request.type === "permission" && request.status === "pending" + && request.requestId === p.requestId && request.turnId === identity!.turnId && request.itemId === identity!.itemId + && request.prompt === "mcp__paperclip__get_task_context" && hasAcpxNativeOrigin(request.origin, "kimi", "session/request_permission") + && request.details?.toolCallId === p.toolCallId && resolution.requestId === p.requestId && resolution.requestKind === "permission_approval" + && resolution.action === "accept" && resolution.turnId === identity!.turnId && resolution.itemId === identity!.itemId && created!.sourceSeq < resolved!.sourceSeq); + const resolutionCommand = resolutions[0]!, resolvedDelivery = completed(resolutionCommand, "delivered"); + demand(resolutionCommand.payload?.turnId === identity!.turnId && resolutionCommand.payload?.resolution?.action === "accept" + && resolvedDelivery.requestId === p.requestId && turnStart.controllerSeq < resolutionCommand.controllerSeq); + demand(events.length <= 1 && commands.length <= 1); + if (!events.length) return undefined; + const saved = events[0]!, envelope = rec(saved.envelope), event = rec(envelope.payload), tool = rec(rec(event.payload).semantic_tool), correlation = rec(tool.correlation); + demand(envelope.protocol === "paperclip.runner" && envelope.kind === "event" && envelope.runId === p.runId + && envelope.normalizedSessionId === p.sessionId && envelope.runnerInstanceId === p.sourceInstanceId && envelope.turnId === identity!.turnId + && envelope.environmentLeaseId === identity!.environmentLeaseId && envelope.itemId === identity!.itemId + && event.schema === "paperclip.prp.event.v1" && event.schemaVersion === 1 && event.eventType === "semantic_tool.input" && event.sourceKind === "runner" + && event.runId === p.runId && event.normalizedSessionId === p.sessionId && event.sourceInstanceId === p.sourceInstanceId && event.turnId === identity!.turnId && event.itemId === identity!.itemId + && event.sourceSeq > resolved!.sourceSeq && saved.sourceSeq === event.sourceSeq && saved.sourceEventId === event.sourceEventId && saved.logicalEffectCount === 1 + && Number.isSafeInteger(event.sourceSeq) && event.sourceSeq > 0 && event.sourceSeq <= evidence.highestRecordedSourceSeq + && tool.schema === "paperclip.prp.semantic_tool.v1" && tool.schemaVersion === 1 && tool.phase === "input" && tool.operationId === "get_task_context" + && typeof tool.callId === "string" && tool.callId.length > 0 && tool.idempotencyKey === null && canonicalJson(tool.input) === "{}" && tool.content?.digest === hash({}) + && canonicalJson(correlation) === canonicalJson({ runId: p.runId, normalizedSessionId: p.sessionId, turnId: identity!.turnId, itemId: identity!.itemId })); + if (!commands.length) return undefined; + const command = commands[0]!, body = rec(command.payload), delivered = rec(command.result); + demand(command.schema === "paperclip.prp.command.v1" && command.commandId === `command_tool_${bytes(`${p.runId}\0${tool.callId}`).slice(0, 32)}` + && Number.isSafeInteger(command.controllerSeq) && command.controllerSeq > resolutionCommand.controllerSeq && body.callId === tool.callId && body.operationId === "get_task_context" + && body.sourceEventId === event.sourceEventId && body.sourceEventType === event.eventType && body.inputDigest === bytes("{}") + && (body.input === undefined || canonicalJson(body.input) === "{}") && canonicalJson(body.correlation) === canonicalJson(correlation) && body.isError === false); + if (command.status === "pending") return undefined; + demand(command.status === "completed" && delivered.commandId === command.commandId && delivered.controllerSeq === command.controllerSeq + && delivered.commandType === command.type && delivered.status === "completed" && delivered.result?.status === "delivered" && delivered.result?.callId === tool.callId); + const value = rec(body.result); + demand(value.company?.id === p.companyId && value.actor?.id === p.agentId && value.activeTask?.id === p.issueId + && typeof value.activeTask.description === "string" && `sha256:${bytes(value.activeTask.description)}` === p.actionSha256); + return { schema: "paperclip.e2e.kimi-context-authority.v2", companyId: p.companyId, issueId: p.issueId, agentId: p.agentId, environmentId: p.environmentId, runId: p.runId, + sessionId: p.sessionId, sourceInstanceId: p.sourceInstanceId, turnId: p.turnId, observerLeaseId: p.observerLeaseId, executionWorkspaceId, controllerTurnId: identity!.turnId, controllerItemId: identity!.itemId, requestId: p.requestId, toolCallId: p.toolCallId, + requestCreatedSourceSeq: created!.sourceSeq, requestResolvedSourceSeq: resolved!.sourceSeq, requestCreatedSha256: hash(created), requestResolvedSha256: hash(resolved), turnStartSha256: hash(turnStart), resolutionCommandSha256: hash(resolutionCommand), actionSha256: p.actionSha256, callId: tool.callId, + inputSourceSeq: event.sourceSeq, inputSourceEventId: event.sourceEventId, inputSha256: hash({}), inputEventSha256: hash(saved), resultSha256: hash(value), correlationSha256: hash(correlation), + commandId: command.commandId, controllerSeq: command.controllerSeq, commandSha256: hash(command), deliverySha256: hash(delivered), + checkpointIdentitySha256: hash(identity), authenticatedConnectionCount: evidence.authenticatedConnectionCount, highestRecordedSourceSeq: evidence.highestRecordedSourceSeq }; +} +export async function assertKimiContextObserverConfig(env: NodeJS.ProcessEnv) { + const temporary = env.PAPERCLIP_RUNNER_E2E_TEMP_ROOT, config = env.PAPERCLIP_CONFIG, home = env.PAPERCLIP_HOME, instance = env.PAPERCLIP_INSTANCE_ID; + demand(temporary && config && home && instance && path.isAbsolute(temporary) && path.isAbsolute(config) && path.isAbsolute(home) && /^[A-Za-z0-9_-]+$/.test(instance)); + const [root, actualConfig, actualHome] = await Promise.all([realpath(temporary!), realpath(config!), realpath(home!)]); + demand(actualHome === path.join(root, "paperclip-home") && actualConfig === path.join(actualHome, "instances", instance!, "config.json")); + const { assertEmbeddedDatabaseIsolation } = await import("./instance-isolation.js"); + await assertEmbeddedDatabaseIsolation(actualConfig, root); + return actualConfig; +} +export async function createKimiContextCheckpointObserver(p: KimiContextPublication) { + const configPath = await assertKimiContextObserverConfig(process.env); + const [{ createDb }, { paperclipConfigSchema }, { loadNativeRunnerCheckpoint }, { createRequire }] = await Promise.all([ + import("../../packages/db/src/index.js"), import("../../packages/shared/src/config-schema.js"), import("../../server/src/services/native-runtime/native-runner-checkpoint.js"), import("node:module"), + ]); + const { sql } = createRequire(new URL("../../packages/db/package.json", import.meta.url))("drizzle-orm"); + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(configPath, "utf8"))); + const url = config.database.mode === "postgres" ? config.database.connectionString : `postgres://paperclip:paperclip@127.0.0.1:${config.database.embeddedPostgresPort}/paperclip`; + demand(url); const db = createDb(url!, { maxConnections: 1, connectTimeoutSeconds: 5 }); let closed = false; + return { + read: async () => { + demand(!closed); + return db.transaction(async tx => { + await tx.execute(sql`set local statement_timeout = '5000ms'`); + const [owned] = await tx.execute(sql`select l.execution_workspace_id as "executionWorkspaceId", r.runner_profile_json as profile from heartbeat_runs r join issues i on i.id = r.native_issue_id and i.company_id = r.company_id + join environment_leases l on l.heartbeat_run_id = r.id and l.company_id = r.company_id and l.issue_id = i.id + join execution_workspaces w on w.id = l.execution_workspace_id and w.id = i.execution_workspace_id and w.company_id = r.company_id and w.source_issue_id = i.id and w.status = 'active' + where l.id = ${p.observerLeaseId} and l.environment_id = ${p.environmentId} and l.status = 'active' and r.id = ${p.runId} and r.company_id = ${p.companyId} and r.agent_id = ${p.agentId} and r.native_issue_id = ${p.issueId} + and r.native_session_id = ${p.sessionId} and r.runner_instance_id = ${p.sourceInstanceId} and r.runtime_mode = 'native' and r.status = 'running' + and i.execution_run_id = r.id and i.assignee_agent_id = r.agent_id and i.status = 'in_progress'`); + demand(owned); + const execution = parseNativeExecutionInput(rec(owned.profile).nativeExecutionInput), binding = execution.binding; + demand(binding.companyId === p.companyId && binding.issueId === p.issueId && binding.agentId === p.agentId && binding.runId === p.runId + && binding.executionWorkspaceId === owned.executionWorkspaceId && execution.session.normalizedSessionId === p.sessionId); + const checkpoint = await loadNativeRunnerCheckpoint(tx as any, { companyId: p.companyId, issueId: p.issueId, runId: p.runId }); + return checkpoint ? projectKimiContextCheckpoint(checkpoint, p, owned.executionWorkspaceId as string) : undefined; + }, { isolationLevel: "repeatable read", accessMode: "read only" }); + }, + close: async () => { if (closed) return; closed = true; await db.$client.end({ timeout: 5 }); }, + }; +} diff --git a/tests/runner-e2e/kimi-context-rendezvous.test.ts b/tests/runner-e2e/kimi-context-rendezvous.test.ts new file mode 100644 index 00000000000..068281cd11f --- /dev/null +++ b/tests/runner-e2e/kimi-context-rendezvous.test.ts @@ -0,0 +1,87 @@ +import { createHash } from "node:crypto"; +import { mkdtemp, mkdir, readFile, rm, writeFile } from "node:fs/promises"; +import { tmpdir } from "node:os"; +import path from "node:path"; +import { describe, expect, it, vi } from "vitest"; +import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; +import { assertKimiContextObserverConfig, projectKimiContextCheckpoint } from "./kimi-context-checkpoint.js"; +import { pendingKimiContextPermission, publishKimiContextContinuation, readKimiContextProof, withoutProvenKimiContext } from "./kimi-context-rendezvous.js"; +import { observeKimiLifecyclePending } from "./kimi-lifecycle-evidence.js"; +const hash = (v: unknown) => `sha256:${createHash("sha256").update(canonicalJson(v)).digest("hex")}`; +const bytesHash = (v: string) => `sha256:${createHash("sha256").update(v).digest("hex")}`; +type Row = Record; +const produced = JSON.parse(await readFile(new URL("./fixtures/kimi-context-command-v2.json", import.meta.url), "utf8")); +function fixture() { + const identity = produced.identity, scope = { companyId: "synthetic-company", agentId: "synthetic-agent", environmentId: "synthetic-env", issueId: "synthetic-task", runId: identity.runId }, action = produced.action; + const row = (seq: number, eventType: string, payload: Row): Row => ({ companyId: scope.companyId, runId: scope.runId, seq, eventType, protocolSchemaVersion: 1, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", sourceInstanceId: identity.runnerInstanceId, sourceSeq: seq, sourceEventId: `${identity.runnerInstanceId}:${scope.runId}:${seq}`, emittedAt: "2026-10-29T00:00:00Z", normalizedSessionId: identity.normalizedSessionId, itemId: identity.itemId, turnId: produced.nativeTurnId, runId: scope.runId, eventType, payload } } }); + const tool = { schema: "paperclip.tool.execution.v1", executionId: "0:synthetic-context-call", transport: "mcp", namespace: "paperclip", name: "get_task_context", operation: "unknown", status: "running", target: null, output: "", outputTruncated: false }; + const request = { schema: "paperclip.runtime_request.v2", requestKind: "permission_approval", type: "permission", status: "pending", requestId: "context-request", turnId: produced.nativeTurnId, itemId: identity.itemId, method: "session/request_permission", prompt: "mcp__paperclip__get_task_context", origin: { adapter: "acpx-runtime-sidecar", provider: "kimi", method: "session/request_permission" }, choices: [{ key: "accept" }, { key: "decline" }, { key: "cancel" }], details: { toolCallId: tool.executionId, turnId: produced.nativeTurnId, itemId: identity.itemId } }; + const rows = [row(1, "tool.execution.started", tool), row(2, "runtime_request.created", { request })], pending = pendingKimiContextPermission(rows, scope)!; + const publication: any = { ...pending, schema: "paperclip.e2e.kimi-context-publication.v1", observerLeaseId: "synthetic-lease", actionSha256: bytesHash(action), setupSha256: bytesHash(action), observerSha256: hash({ complete: true }) }; + const records = new Map(produced.checkpoint.records), header: Row = { schema: "paperclip.runner.controller-checkpoint.v2", runnerVersion: "0.3.0", runnerDigest: `sha256:${"a".repeat(64)}`, state: { schema: "paperclip.runner.durable.control-plane-state.v1", checkpointEventPolicy: "control-v1", identity: structuredClone(identity) } }; + records.set("header", JSON.stringify(header)); + const checkpoint: any = { ...structuredClone(produced.checkpoint), records }; + const inputKey = [...records].find(([k,v]) => k.startsWith("event:") && JSON.parse(v).eventType === "semantic_tool.input")![0], commandKey = [...records].find(([k,v]) => k.startsWith("command:") && JSON.parse(v).type === "semantic_tool.result")![0]; + function modify(key: string, f: (v: Row) => void) { const v=JSON.parse(records.get(key)!);f(v);records.set(key,JSON.stringify(v)); } + function deliver() { + const command = JSON.parse(records.get(commandKey)!); + const output = JSON.stringify(command.payload.result).slice(0,4096) + "…"; + rows.push(row(44, "runtime_request.resolved", { requestId: request.requestId, requestKind: "permission_approval", turnId: produced.nativeTurnId, action: "accept" }), + row(48, "tool.execution.completed", { ...tool, name: null, namespace: null, transport: "builtin", status: "completed", output, outputBytes: Buffer.byteLength(output), outputDigest: bytesHash(output) })); + } + function write() { rows.push(row(49,"tool.execution.started",{...tool,executionId:"0:native-write",transport:"builtin",namespace:null,name:"Write",operation:"edit"}),row(50,"runtime_request.created",{request:{...request,requestId:"write-request",prompt:"Change file: denied/target.txt",details:{toolCallId:"0:native-write"}}})); } + return { scope, identity, action, row, rows, tool, request, pending, publication, checkpoint, records, header, inputKey, commandKey, modify, deliver, write }; +} +it("requires the independently selected execution workspace authority",()=>{const f=fixture();expect(()=>projectKimiContextCheckpoint(f.checkpoint,f.publication,undefined as any)).toThrow();expect(()=>projectKimiContextCheckpoint(f.checkpoint,f.publication,"foreign-workspace")).toThrow();}); +const payload = (r: Row) => r.payload.prpEvent.payload; +describe("Kimi checkpoint-bound context rendezvous", () => { + it("uses the calibrated actual checkpoint command shape and keeps only the unanswered native Write", () => { + const f=fixture(),authority=projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)!;expect(authority).toEqual(produced.projected);expect(authority.turnId).not.toBe(authority.controllerTurnId);expect(authority.observerLeaseId).not.toBe(authority.executionWorkspaceId);f.deliver();expect(Buffer.byteLength(payload(f.rows[3]!).output)).toBeGreaterThan(4096);expect(()=>JSON.parse(payload(f.rows[3]!).output)).toThrow(); + const proof=readKimiContextProof(f.rows,f.publication,authority)!;expect(proof).toBeDefined();f.write();expect(withoutProvenKimiContext(f.rows,proof)).toEqual(f.rows.slice(4)); + const run={id:f.scope.runId,companyId:f.scope.companyId,nativeIssueId:f.scope.issueId,status:"running",runtimeMode:"native",nativeSessionId:f.identity.normalizedSessionId,runnerInstanceId:f.identity.runnerInstanceId,resultJson:{}}; + expect(observeKimiLifecyclePending({run,issue:{id:f.scope.issueId,companyId:f.scope.companyId,status:"in_progress"},events:f.rows,scope:{companyId:f.scope.companyId,issueId:f.scope.issueId,runId:f.scope.runId,target:"denied/target.txt",contextProof:proof}})).toMatchObject({requestId:"write-request"}); + }); + it.each(["company","issue","agent","run","session","runner","turn","marker","no-auth","input-digest","extra-arguments","call","source","correlation","result-hash","failed-command","duplicate-input","duplicate-command","logical-lease","controller-turn","controller-item","request-id","request-tool","request-kind","resolved-turn","resolved-action","resolution-ack","start-turn","start-ack","input-order"])("rejects %s before accepting scoped result delivery", kind => { + const f=fixture(); + if(["company","issue","agent"].includes(kind)) f.publication[`${kind}Id`]="foreign"; + if(kind==="run")f.checkpoint.runId="foreign";if(kind==="session")f.publication.sessionId="foreign";if(kind==="runner")f.publication.sourceInstanceId="foreign";if(kind==="turn")f.publication.turnId="foreign"; + if(kind==="marker")f.records.set("header",JSON.stringify({schema:"paperclip.runner.durable.control-plane-external.v1"}));if(kind==="no-auth")f.checkpoint.recoveryEvidence.authenticatedConnectionCount=0; + if(kind==="input-digest")f.modify(f.commandKey,c=>c.payload.inputDigest="0".repeat(64));if(kind==="extra-arguments")f.modify(f.inputKey,c=>{c.envelope.payload.payload.semantic_tool.input={path:"foreign"};c.envelope.payload.payload.semantic_tool.content.digest=hash({path:"foreign"});}); + if(kind==="call")f.modify(f.commandKey,c=>c.payload.callId="foreign");if(kind==="source")f.modify(f.commandKey,c=>c.payload.sourceEventId="foreign");if(kind==="correlation")f.modify(f.commandKey,c=>c.payload.correlation.turnId="foreign"); + if(kind==="result-hash")f.modify(f.commandKey,c=>c.payload.result.activeTask.description="foreign");if(kind==="failed-command")f.modify(f.commandKey,c=>c.status="failed"); + if(kind==="duplicate-input")f.records.set("event:999",f.records.get(f.inputKey)!);if(kind==="duplicate-command")f.records.set("command:999",f.records.get(f.commandKey)!); + const requestKey=[...f.records].find(([k,v])=>k.startsWith("event:")&&JSON.parse(v).eventType==="runtime_request.created")![0], resolvedKey=[...f.records].find(([k,v])=>k.startsWith("event:")&&JSON.parse(v).eventType==="runtime_request.resolved")![0], startKey=[...f.records].find(([k,v])=>k.startsWith("command:")&&JSON.parse(v).type==="turn.start")![0], resolutionKey=[...f.records].find(([k,v])=>k.startsWith("command:")&&JSON.parse(v).type==="request.resolve")![0]; + if(kind==="logical-lease")f.header.state.identity.environmentLeaseId="foreign"; + if(kind==="controller-turn")f.header.state.identity.turnId="foreign";if(kind==="controller-item")f.header.state.identity.itemId="foreign"; + if(["logical-lease","controller-turn","controller-item"].includes(kind))f.records.set("header",JSON.stringify(f.header)); + if(kind==="request-id")f.publication.requestId="foreign";if(kind==="request-tool")f.modify(requestKey,c=>c.envelope.payload.payload.request.details.toolCallId="foreign"); + if(kind==="request-kind")f.modify(requestKey,c=>c.envelope.payload.payload.request.requestKind="runtime");if(kind==="resolved-turn")f.modify(resolvedKey,c=>c.envelope.payload.payload.turnId="foreign");if(kind==="resolved-action")f.modify(resolvedKey,c=>c.envelope.payload.payload.action="decline"); + if(kind==="resolution-ack")f.modify(resolutionKey,c=>c.result.result.status="queued");if(kind==="start-turn")f.modify(startKey,c=>c.payload.turnId="foreign");if(kind==="start-ack")f.modify(startKey,c=>c.result.result.providerTurnId="foreign");if(kind==="input-order")f.modify(f.inputKey,c=>c.envelope.payload.sourceSeq=2); + if(kind==="request-id")expect(projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)).toBeUndefined();else expect(()=>projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)).toThrow(); + }); + it("waits for the matching completed command and correlated native completion",()=>{const f=fixture();f.modify(f.commandKey,c=>c.status="pending");expect(projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)).toBeUndefined();f.deliver();expect(readKimiContextProof(f.rows,f.publication)).toBeUndefined();}); + it.each(["queued-only","wrong-request-kind","declined","native-output","input-order","foreign-turn","foreign-item","foreign-native-request-turn"])("rejects %s as delivered native context",kind=>{ + const f=fixture(),a=projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)!;f.deliver();if(kind==="queued-only"){f.rows.pop();expect(readKimiContextProof(f.rows,f.publication,a)).toBeUndefined();return;} + if(kind==="wrong-request-kind")payload(f.rows[2]!).requestKind="runtime";if(kind==="declined")payload(f.rows[2]!).action="decline";if(kind==="native-output")payload(f.rows[3]!).output="{}";if(kind==="input-order")a.inputSourceSeq=a.requestResolvedSourceSeq;if(kind==="foreign-item")payload(f.rows[1]!).request.details.itemId="foreign";if(kind==="foreign-native-request-turn")payload(f.rows[1]!).request.details.turnId="foreign";if(kind==="foreign-turn")a.turnId="foreign"; + expect(()=>readKimiContextProof(f.rows,f.publication,a)).toThrow(); + }); + it.each(["Read","Write","mcp__other__get_task_context","mcp__paperclip__call_api"])("never approves unknown native or foreign operation %s",name=>{const f=fixture();f.request.prompt=name;expect(()=>pendingKimiContextPermission(f.rows,f.scope)).toThrow();}); + it("accepts an honestly bounded native display whose metadata describes its full preimage",()=>{const f=fixture(),a=projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)!;f.deliver();const full=JSON.stringify(JSON.parse(f.records.get(f.commandKey)!).payload.result),view=payload(f.rows[3]!);view.outputTruncated=true;view.outputBytes=Buffer.byteLength(full);view.outputDigest=bytesHash(full);expect(readKimiContextProof(f.rows,f.publication,a)).toBeDefined();}); + it("never strips an unknown native Read",()=>{const f=fixture(),a=projectKimiContextCheckpoint(f.checkpoint,f.publication,f.identity.environmentLeaseId)!;f.deliver();const proof=readKimiContextProof(f.rows,f.publication,a)!;const read=f.row(49,"tool.execution.started",{...f.tool,executionId:"unknown",transport:"builtin",namespace:null,name:"Read",operation:"read"});expect(withoutProvenKimiContext([...f.rows,read],proof)).toContainEqual(read);}); + it("rejects ambient observer configuration before opening a database",async()=>{await expect(assertKimiContextObserverConfig({})).rejects.toThrow();}); + it("rejects a foreign physical config path before opening a database",async()=>{const root=await mkdtemp(path.join(tmpdir(),"kimi-context-config-"));try{const home=path.join(root,"paperclip-home");await mkdir(home);const foreign=path.join(root,"foreign.json");await writeFile(foreign,"{}");await expect(assertKimiContextObserverConfig({PAPERCLIP_RUNNER_E2E_TEMP_ROOT:root,PAPERCLIP_HOME:home,PAPERCLIP_INSTANCE_ID:"synthetic",PAPERCLIP_CONFIG:foreign})).rejects.toThrow();}finally{await rm(root,{recursive:true,force:true});}}); +}); +function publicationFixture() { + const f=fixture(),actionFile=`.paperclip-eval-action-${"a".repeat(36)}.txt`; + const snapshot:Row={complete:true,binding:{leaseId:"synthetic-lease",companyId:f.scope.companyId,environmentId:f.scope.environmentId,runId:f.scope.runId},watcher:{complete:true,targetMutationCount:0,workspaceMutationCount:0},setup:{path:actionFile,published:true,sha256:bytesHash(f.action)}}; + const issue:Row={id:f.scope.issueId,companyId:f.scope.companyId,assigneeAgentId:f.scope.agentId,executionRunId:f.scope.runId,status:"in_progress"},run:Row={id:f.scope.runId,companyId:f.scope.companyId,agentId:f.scope.agentId,status:"running",runtimeMode:"native",nativeIssueId:f.scope.issueId,nativeSessionId:f.identity.normalizedSessionId,runnerInstanceId:f.identity.runnerInstanceId}; + const api={get:vi.fn(async(p:string)=>structuredClone(p.includes("heartbeat-runs")?run:issue)),patch:vi.fn(async(_p:string,body:unknown)=>{Object.assign(issue,body);return structuredClone(issue);})},remote={actionFile,snapshot:vi.fn(async()=>snapshot)}; + const client = { + get: async(p:string):Promise => await api.get(p) as T, + patch: async(p:string,body:unknown):Promise => await api.patch(p,body) as T, + }; + return {...f,snapshot,issue,run,api,remote,publish:()=>publishKimiContextContinuation({pending:f.pending,action:f.action,fixture:remote,api:client})}; +} +it("publishes once only after immutable observer arming and exact active native ownership",async()=>{const f=publicationFixture();await expect(f.publish()).resolves.toMatchObject({actionSha256:bytesHash(f.action)});expect(f.remote.snapshot).toHaveBeenCalledBefore(f.api.patch);expect(f.api.patch).toHaveBeenCalledExactlyOnceWith(`/api/issues/${f.scope.issueId}`,{description:f.action});}); +it.each(["company","environment","run","unarmed","mutated","wrong-path","wrong-hash","unpublished","foreign-task","foreign-agent","terminal-run","foreign-session"])("blocks %s before operator description publication",async kind=>{const f=publicationFixture();if(kind==="company")f.snapshot.binding.companyId="foreign";if(kind==="environment")f.snapshot.binding.environmentId="foreign";if(kind==="run")f.snapshot.binding.runId="foreign";if(kind==="unarmed")f.snapshot.complete=false;if(kind==="mutated")f.snapshot.watcher.targetMutationCount=1;if(kind==="wrong-path")f.snapshot.setup.path="foreign";if(kind==="wrong-hash")f.snapshot.setup.sha256=bytesHash("foreign");if(kind==="unpublished")f.snapshot.setup.published=false;if(kind==="foreign-task")f.issue.id="foreign";if(kind==="foreign-agent")f.issue.assigneeAgentId="foreign";if(kind==="terminal-run")f.run.status="failed";if(kind==="foreign-session")f.run.nativeSessionId="foreign";await expect(f.publish()).rejects.toThrow();expect(f.api.patch).not.toHaveBeenCalled();}); diff --git a/tests/runner-e2e/kimi-context-rendezvous.ts b/tests/runner-e2e/kimi-context-rendezvous.ts new file mode 100644 index 00000000000..b6195796f2e --- /dev/null +++ b/tests/runner-e2e/kimi-context-rendezvous.ts @@ -0,0 +1,159 @@ +import { expect, type Page } from "@playwright/test"; +import { createHash } from "node:crypto"; +import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; +import { getTaskContextAction } from "../../packages/paperclip-runner/src/protocol-actions/get-task-context.js"; +import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; +import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; +import { bootstrapReadExecutionId } from "./native-bootstrap-read-proof.js"; + +import type { KimiContextAuthority } from "./kimi-context-checkpoint.js"; + +type Row = Record; +const rec = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {}; +const bytesHash = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`; +const hash = (value: unknown) => bytesHash(canonicalJson(value)); +const requireProof = (ok: unknown): void => { if (!ok) throw new Error("Kimi context rendezvous: exact scoped read proof is unavailable"); }; +export interface KimiContextScope { companyId: string; agentId: string; environmentId: string; issueId: string; runId: string } +export interface KimiContextPending extends KimiContextScope { + requestId: string; toolCallId: string; executionId: string; turnId: string; sessionId: string; sourceInstanceId: string; + createdSeq: number; createdSha256: string; +} +export interface KimiContextPublication extends KimiContextPending { + schema: "paperclip.e2e.kimi-context-publication.v1"; observerLeaseId: string; actionSha256: string; setupSha256: string; observerSha256: string; +} +export interface KimiContextProof extends Omit { + schema: "paperclip.e2e.kimi-context-proof.v1"; + callId: string; inputSha256: string; resultSha256: string; completedSeq: number; authority: KimiContextAuthority; + rows: Array<{ seq: number; sha256: string }>; +} +function frames(events: readonly unknown[], scope: KimiContextScope) { + requireProof(events.length > 0 && events.length <= 20_000); + const rows = events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined).map(row => ({ row, event: rec(rec(row.payload).prpEvent) })).sort((a, b) => a.row.seq - b.row.seq); + const seqs = new Set(), sources = new Set(), cursor = new Map(); + for (const { row, event } of rows) { + requireProof(row.companyId === scope.companyId && row.runId === scope.runId && isValidNativePrpEnvelope(event, row.protocolSchemaVersion) + && event.runId === scope.runId && event.eventType === row.eventType && Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq) + && typeof event.sourceInstanceId === "string" && event.sourceInstanceId && Number.isSafeInteger(event.sourceSeq) + && event.sourceSeq > (cursor.get(event.sourceInstanceId) ?? 0) && event.sourceEventId === `${event.sourceInstanceId}:${scope.runId}:${event.sourceSeq}` + && !sources.has(event.sourceEventId)); + seqs.add(row.seq); sources.add(event.sourceEventId); cursor.set(event.sourceInstanceId, event.sourceSeq); + } + return rows; +} +function sameStream(event: Row, pending: KimiContextPending) { + return event.sourceKind === "runner" && event.turnId === pending.turnId && event.normalizedSessionId === pending.sessionId && event.sourceInstanceId === pending.sourceInstanceId; +} +function declaredReadOnlyContext() { + requireProof(getTaskContextAction.canonical.operationId === "get_task_context" && getTaskContextAction.canonical.sideEffectClass === "read" + && canonicalJson(getTaskContextAction.live.descriptor.inputSchema) === canonicalJson({ type: "object", properties: {}, required: [], additionalProperties: false })); +} +export function kimiContextRendezvousPrompt(): string { + return "Call the already configured Paperclip get_task_context tool exactly once with {}. Wait for the operator's one Allow once decision. Its returned activeTask.description will contain the complete task continuation. Perform no native read, shell, write, search, discovery, semantic mutation, or completion before receiving that context. After reading the returned description, perform only that supplied native action."; +} +export function pendingKimiContextPermission(events: readonly unknown[], scope: KimiContextScope): KimiContextPending | undefined { + declaredReadOnlyContext(); + const rows = frames(events, scope), created = rows.filter(x => x.event.eventType === "runtime_request.created"); + if (!created.length) return undefined; + requireProof(created.length === 1); + const { row, event } = created[0]!, request = rec(rec(event.payload).request); + requireProof(request.schema === "paperclip.runtime_request.v2" && request.requestKind === "permission_approval" && request.type === "permission" && request.status === "pending" + && request.prompt === "mcp__paperclip__get_task_context" && request.method === "session/request_permission" + && hasAcpxNativeOrigin(request.origin, "kimi", "session/request_permission") && request.turnId === event.turnId + && typeof request.requestId === "string" && typeof request.details?.toolCallId === "string" + && Array.isArray(request.choices) && request.choices.map((x: Row) => x.key).sort().join(",") === "accept,cancel,decline"); + const pending: KimiContextPending = { ...scope, requestId: request.requestId, toolCallId: request.details.toolCallId, executionId: bootstrapReadExecutionId(request.details.toolCallId), + turnId: event.turnId, sessionId: event.normalizedSessionId, sourceInstanceId: event.sourceInstanceId, createdSeq: row.seq, createdSha256: hash(row) }; + requireProof([pending.turnId, pending.sessionId, pending.sourceInstanceId].every(value => typeof value === "string" && value.length > 0)); + const tools = rows.filter(x => x.event.eventType.startsWith("tool.execution.")); + requireProof(tools.every(x => sameStream(x.event, pending) && x.event.payload?.executionId === pending.executionId && x.event.payload?.target === null)); + const named = tools.filter(x => x.event.payload?.transport === "mcp" && x.event.payload?.namespace === "paperclip" && x.event.payload?.name === "get_task_context"); + if (!named.length) return undefined; + requireProof(tools.every(x => x.event.payload?.status === "running" && x.event.eventType !== "tool.execution.completed") + && rows.every(x => !x.event.eventType.startsWith("semantic_tool.") && !["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(x.event.eventType))); + return pending; +} +export async function publishKimiContextContinuation(input: { + pending: KimiContextPending; action: string; + fixture: { actionFile: string; snapshot(label: string): Promise }; + api: { get(path: string): Promise; patch(path: string, body: unknown): Promise }; +}): Promise { + const p = input.pending; requireProof(input.action.trim().length > 0 && Buffer.byteLength(input.action) <= 16_384); + const snapshot = await input.fixture.snapshot("before-kimi-context-publication"), binding = rec(snapshot.binding), setup = rec(snapshot.setup), actionSha256 = bytesHash(input.action); + requireProof(snapshot.complete === true && snapshot.watcher?.complete === true && snapshot.watcher?.targetMutationCount === 0 && snapshot.watcher?.workspaceMutationCount === 0 + && typeof binding.leaseId === "string" && binding.leaseId.length > 0 && binding.companyId === p.companyId && binding.environmentId === p.environmentId && binding.runId === p.runId && setup.path === input.fixture.actionFile && setup.published === true && setup.sha256 === actionSha256); + const [issue, run] = await Promise.all([input.api.get(`/api/issues/${p.issueId}`), input.api.get(`/api/heartbeat-runs/${p.runId}`)]); + const owned = (task: Row) => task.id === p.issueId && task.companyId === p.companyId && task.assigneeAgentId === p.agentId && task.executionRunId === p.runId && task.status === "in_progress"; + requireProof(owned(issue) && run.id === p.runId && run.companyId === p.companyId && run.agentId === p.agentId && run.status === "running" && run.runtimeMode === "native" && run.nativeIssueId === p.issueId + && run.nativeSessionId === p.sessionId && run.runnerInstanceId === p.sourceInstanceId); + const updated = await input.api.patch(`/api/issues/${p.issueId}`, { description: input.action }); + const confirmed = await input.api.get(`/api/issues/${p.issueId}`); + requireProof(owned(updated) && owned(confirmed) && updated.description === input.action && confirmed.description === input.action); + return { ...p, schema: "paperclip.e2e.kimi-context-publication.v1", observerLeaseId: binding.leaseId, actionSha256, setupSha256: setup.sha256, observerSha256: hash(snapshot) }; +} +export function readKimiContextProof(events: readonly unknown[], publication: KimiContextPublication, authority?: KimiContextAuthority): KimiContextProof | undefined { + declaredReadOnlyContext(); + requireProof(publication.schema === "paperclip.e2e.kimi-context-publication.v1" && /^sha256:[a-f0-9]{64}$/.test(publication.actionSha256) + && publication.setupSha256 === publication.actionSha256 && /^sha256:[a-f0-9]{64}$/.test(publication.observerSha256)); + const rows = frames(events, publication), same = rows.filter(x => sameStream(x.event, publication)); + const created = same.filter(x => x.event.eventType === "runtime_request.created" && x.event.payload?.request?.requestId === publication.requestId); + requireProof(created.length === 1 && created[0]!.row.seq === publication.createdSeq && hash(created[0]!.row) === publication.createdSha256); + const closed = same.filter(x => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(x.event.eventType) && x.event.payload?.requestId === publication.requestId); + if (!closed.length) return undefined; + requireProof(closed.length === 1 && closed[0]!.event.eventType === "runtime_request.resolved" && closed[0]!.event.payload.action === "accept" && closed[0]!.event.payload.requestKind === "permission_approval" + && closed[0]!.event.payload.turnId === publication.turnId && closed[0]!.row.seq > publication.createdSeq); + const beforeDecision = events.filter(value => rec(value).seq < closed[0]!.row.seq); + const original = pendingKimiContextPermission(beforeDecision, publication); + requireProof(original && Object.entries(original).every(([key, value]) => publication[key as keyof KimiContextPublication] === value)); + if (!authority) return undefined; + requireProof(authority.schema === "paperclip.e2e.kimi-context-authority.v2" + && ["companyId", "issueId", "agentId", "environmentId", "runId", "sessionId", "sourceInstanceId", "turnId", "actionSha256", "observerLeaseId", "requestId", "toolCallId"].every(key => (authority as unknown as Row)[key] === (publication as unknown as Row)[key]) + && authority.inputSha256 === hash({}) && typeof authority.callId === "string" && authority.callId.length > 0 + && [authority.inputEventSha256, authority.resultSha256, authority.correlationSha256, authority.commandSha256, authority.deliverySha256, authority.checkpointIdentitySha256, authority.requestCreatedSha256, authority.requestResolvedSha256, authority.turnStartSha256, authority.resolutionCommandSha256].every(value => /^sha256:[a-f0-9]{64}$/.test(value)) + && typeof authority.executionWorkspaceId === "string" && authority.executionWorkspaceId.length > 0 + && created[0]!.event.itemId === authority.controllerItemId && created[0]!.event.payload.request.itemId === authority.controllerItemId + && created[0]!.event.payload.request.details?.itemId === authority.controllerItemId + && created[0]!.event.payload.request.details?.turnId === publication.turnId + && Number.isSafeInteger(authority.inputSourceSeq) && authority.requestCreatedSourceSeq < authority.requestResolvedSourceSeq && authority.requestResolvedSourceSeq < authority.inputSourceSeq + && authority.authenticatedConnectionCount > 0 && authority.highestRecordedSourceSeq >= authority.inputSourceSeq); + const native = same.filter(x => x.event.eventType.startsWith("tool.execution.") && x.event.payload?.executionId === publication.executionId); + const finished = native.filter(x => x.event.eventType === "tool.execution.completed"); + if (!finished.length) return undefined; + requireProof(finished.length === 1 && native[0]?.event.eventType === "tool.execution.started" && native.at(-1) === finished[0] + && native.every((x, index) => x.event.payload?.schema === "paperclip.tool.execution.v1" && x.event.payload?.target === null + && (x.event.payload.name === null || x.event.payload.name === "get_task_context") + && (x.event.payload.namespace === null || x.event.payload.namespace === "paperclip") + && x.event.payload.status === (index === native.length - 1 ? "completed" : "running"))); + const final = finished[0]!, output = final.event.payload; + requireProof(final.row.seq > closed[0]!.row.seq && typeof output.output === "string" && Buffer.byteLength(output.output) <= 64 * 1024 + && typeof output.outputTruncated === "boolean" && Number.isSafeInteger(output.outputBytes) && output.outputBytes >= Buffer.byteLength(output.output) + && typeof output.outputDigest === "string" && /^sha256:[a-f0-9]{64}$/.test(output.outputDigest) + && (output.outputTruncated || output.outputBytes === Buffer.byteLength(output.output) && output.outputDigest === bytesHash(output.output))); + const retained = [...created, ...closed, ...native]; + requireProof(new Set(retained.map(x => x.row.seq)).size === retained.length); + return { ...publication, schema: "paperclip.e2e.kimi-context-proof.v1", callId: authority.callId, inputSha256: authority.inputSha256, resultSha256: authority.resultSha256, authority, + completedSeq: final.row.seq, rows: retained.map(x => ({ seq: x.row.seq, sha256: hash(x.row) })) }; +} +export function withoutProvenKimiContext(events: readonly unknown[], proof?: KimiContextProof): unknown[] { + if (!proof) return [...events]; + const checked = readKimiContextProof(events, { ...proof, schema: "paperclip.e2e.kimi-context-publication.v1" }, proof.authority); + requireProof(checked && canonicalJson(checked) === canonicalJson(proof)); + const remove = new Map(proof.rows.map(row => [row.seq, row.sha256])); + return events.filter(value => { const row = rec(value), expected = remove.get(row.seq); if (!expected) return true; requireProof(hash(row) === expected); return false; }); +} + +export async function approveKimiContextThroughUi(input: { + page: Page; publication: KimiContextPublication; load(): Promise<{ events: readonly unknown[] }>; + evidence(name: string, value: unknown): Promise; +}) { + const before = pendingKimiContextPermission((await input.load()).events, input.publication); + requireProof(before && Object.entries(before).every(([key, value]) => input.publication[key as keyof KimiContextPublication] === value)); + const card = input.page.getByTestId("task-chat-runtime-request").filter({ visible: true }); + await expect(card).toHaveCount(1); + await expect(card.getByText("mcp__paperclip__get_task_context", { exact: true })).toBeVisible(); + const endpoint = `/api/heartbeat-runs/${input.publication.runId}/runtime-requests/${encodeURIComponent(input.publication.requestId)}/resolve`; + const sent = input.page.waitForRequest(request => new URL(request.url()).pathname === endpoint && request.method() === "POST"); + await card.getByRole("button", { name: "Allow once", exact: true }).click(); + const body = (await sent).postDataJSON(); + requireProof(body.turnId === input.publication.turnId && body.requestKind === "permission_approval" && body.resolution?.action === "accept"); + await input.evidence("kimi-context-fixture-human-decision.json", { ...input.publication, action: "accept", source: "fixture human; normal browser Allow once for server-owned empty-input read", submittedBodySha256: hash(body) }); +} diff --git a/tests/runner-e2e/kimi-lifecycle-evidence.ts b/tests/runner-e2e/kimi-lifecycle-evidence.ts new file mode 100644 index 00000000000..7b8f7f816e6 --- /dev/null +++ b/tests/runner-e2e/kimi-lifecycle-evidence.ts @@ -0,0 +1,142 @@ +import { withoutProvenKimiContext, type KimiContextProof } from "./kimi-context-rendezvous.js"; +import { createHash } from "node:crypto"; +import { canonicalJson } from "../../packages/shared/src/portability-hash.js"; +import { hasAcpxNativeOrigin } from "./acpx-native-origin.js"; +import { bootstrapReadExecutionId } from "./native-bootstrap-read-proof.js"; +import { isValidNativePrpEnvelope } from "./native-event-envelope.js"; +import type { ActiveStopCaller } from "./native-active-stop-evidence.js"; + +type Row = Record; +export interface KimiLifecycleScope { companyId: string; issueId: string; runId: string; target: string; actionFile?: string; contextProof?: KimiContextProof } +export interface KimiLifecycleState { run: Row; issue: Row; events: readonly unknown[] } +export interface KimiLifecyclePending { + schema: "paperclip.e2e.kimi-native-write-pending.v1"; scope: KimiLifecycleScope; + requestId: string; toolCallId: string; executionId: string; itemId: string; turnId: string; + normalizedSessionId: string; sourceInstanceId: string; requestSourceSeq: number; startedSourceSeq: number; + requestRowSha256: string; startedRowSha256: string; observedMonotonicNs: string; +} +const rec = (value: unknown): Row => value !== null && typeof value === "object" && !Array.isArray(value) ? value as Row : {}; +const id = (value: unknown): value is string => typeof value === "string" && value.length > 0 && value.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(value) && !value.includes("[REDACTED]"); +const hash = (value: unknown) => `sha256:${createHash("sha256").update(canonicalJson(value)).digest("hex")}`; +const terminalTypes = new Set(["turn.completed", "turn.failed", "turn.cancelled", "turn.interrupted"]); +const closureTypes = new Set(["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"]); +function demand(value: unknown, reason: string): asserts value { if (!value) throw new Error(`Kimi native lifecycle: ${reason}`); } + +function origin(events: readonly unknown[], scope: KimiLifecycleScope) { + events = withoutProvenKimiContext(events, scope.contextProof); + demand([scope.companyId, scope.issueId, scope.runId, scope.target].every(id) && events.length > 0 && events.length <= 20_000 + && !scope.target.startsWith("/") && !scope.target.split("/").some(part => part === ".." || part === "." || part === ""), "bounded exact scope required"); + const projected = events.map(rec).filter(row => rec(row.payload).prpEvent !== undefined) + .map(row => ({ row, event: rec(rec(row.payload).prpEvent) })).sort((a, b) => a.row.seq - b.row.seq); + const seqs = new Set(), sourceIds = new Set(), cursors = new Map(); + for (const { row, event: e } of projected) { + demand(row.companyId === scope.companyId && row.runId === scope.runId && isValidNativePrpEnvelope(e, row.protocolSchemaVersion) + && e.runId === scope.runId && e.eventType === row.eventType && Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq) + && id(e.sourceInstanceId) && Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (cursors.get(e.sourceInstanceId) ?? 0) + && e.sourceEventId === `${e.sourceInstanceId}:${e.runId}:${e.sourceSeq}` && !sourceIds.has(e.sourceEventId) + && (e.sourceKind === "runner" || e.sourceKind === "control_plane" && ["run.result.accepted", "run.terminal"].includes(e.eventType)), "foreign, duplicate or reordered canonical event"); + seqs.add(row.seq); sourceIds.add(e.sourceEventId); cursors.set(e.sourceInstanceId, e.sourceSeq); + } + const rows = projected.filter(x => x.event.sourceKind === "runner"); + const created = rows.filter(x => x.event.eventType === "runtime_request.created"); + demand(created.length === 1, "one native Write permission required"); + const card = created[0]!, request = rec(rec(card.event.payload).request); + if (scope.contextProof) demand(card.row.seq > scope.contextProof.completedSeq && card.event.turnId === scope.contextProof.turnId + && card.event.normalizedSessionId === scope.contextProof.sessionId && card.event.sourceInstanceId === scope.contextProof.sourceInstanceId, + "native Write must follow the exact completed scoped context read"); + demand(request.schema === "paperclip.runtime_request.v2" && request.type === "permission" && request.requestKind === "permission_approval" && request.status === "pending" + && id(request.requestId) && id(request.itemId) && id(request.details?.toolCallId) && request.turnId === card.event.turnId + && id(card.event.turnId) && id(card.event.normalizedSessionId) && hasAcpxNativeOrigin(request.origin, "kimi", "session/request_permission") + && request.prompt === `Change file: ${scope.target}` && (request.details.kind === undefined || request.details.kind === "edit") + && Array.isArray(request.choices) && request.choices.some((choice: Row) => choice.key === "decline") + && request.choices.some((choice: Row) => choice.key === "accept") && !request.choices.some((choice: Row) => choice.key === "accept_for_session"), "exact native Write permission identity missing"); + const stream = (event: Row) => event.turnId === card.event.turnId && event.normalizedSessionId === card.event.normalizedSessionId && event.sourceInstanceId === card.event.sourceInstanceId; + demand(rows.filter(x => x.event.turnId != null).every(x => stream(x.event)), "foreign native turn, session or producer"); + const controls = projected.filter(x => x.event.sourceKind === "control_plane"), terminals = rows.filter(x => terminalTypes.has(x.event.eventType)); + demand(controls.length <= 2 && new Set(controls.map(x => x.event.eventType)).size === controls.length + && controls.every(x => terminals.length === 1 && x.row.seq > terminals[0]!.row.seq + && x.event.sourceInstanceId === `${card.event.sourceInstanceId}:control` + && x.event.turnId === card.event.turnId && x.event.normalizedSessionId === card.event.normalizedSessionId), "foreign or premature controller settlement"); + const executionId = bootstrapReadExecutionId(request.details.toolCallId); + const executions = rows.filter(x => x.event.eventType.startsWith("tool.execution.")); + const write = executions.filter(x => x.event.payload.executionId === executionId); + const started = write.filter(x => x.event.eventType === "tool.execution.started"), metadataIndex = write.findIndex(x => x.event.payload.name === "Write" && x.event.payload.operation === "edit"); + demand(started.length === 1 && write[0] === started[0] && metadataIndex >= 0 && write.filter(x => x.event.eventType === "tool.execution.completed").length <= 1 + && write.every((x, index) => x.event.payload.schema === "paperclip.tool.execution.v1" && x.event.payload.transport === "builtin" + && ["tool.execution.started", "tool.execution.progressed", "tool.execution.completed"].includes(x.event.eventType) + && (x.event.payload.name === "Write" || index < metadataIndex && x.event.payload.name === null) + && (x.event.payload.operation === "edit" || index < metadataIndex && x.event.payload.operation === "unknown") + && (x.event.payload.target === scope.target || x.event.payload.target === null) + && x.event.payload.status === (x.event.eventType === "tool.execution.completed" ? "failed" : "running")) + && !write.some((x, index) => x.event.eventType === "tool.execution.completed" && index !== write.length - 1), "exact native Write lifecycle missing or completed"); + const extra = executions.filter(x => x.event.payload.executionId !== executionId), reads = new Map(); + for (const x of extra) { + const p = rec(x.event.payload); + demand(scope.actionFile && /^\.paperclip-eval-action-[a-f0-9]{36}\.txt$/u.test(scope.actionFile) + && p.schema === "paperclip.tool.execution.v1" && p.transport === "builtin" + && x.row.seq < Math.min(card.row.seq, started[0]!.row.seq) && id(p.executionId), "extra native or semantic operation"); + const group = reads.get(p.executionId) ?? []; group.push(x); reads.set(p.executionId, group); + } + for (const group of reads.values()) { + const located = group.findIndex(x => x.event.payload.target === scope.actionFile && x.event.payload.name === "Read" && x.event.payload.operation === "read"); + demand(group.length >= 2 && located >= 0 && group[0]!.event.eventType === "tool.execution.started" + && group.at(-1)!.event.eventType === "tool.execution.completed" && ["completed", "failed"].includes(group.at(-1)!.event.payload.status) + && group.every((x, index) => (x.event.payload.name === "Read" || index < located && x.event.payload.name === null) + && (x.event.payload.operation === "read" || index < located && x.event.payload.operation === "unknown") + && (x.event.payload.target === scope.actionFile || index < located && x.event.payload.target === null)) + && group.slice(0, -1).every((x, index) => x.event.eventType === (index === 0 ? "tool.execution.started" : "tool.execution.progressed") && x.event.payload.status === "running"), "bootstrap Read did not close before Write"); + } + return { rows, card, request, started: started[0]!, executionId, stream }; +} + +export function observeKimiLifecyclePending(input: KimiLifecycleState & { scope: KimiLifecycleScope }): KimiLifecyclePending { + const { scope, run, issue } = input, p = origin(input.events, scope); + demand(run.id === scope.runId && run.companyId === scope.companyId && run.nativeIssueId === scope.issueId && run.status === "running" && run.runtimeMode === "native" + && run.nativeSessionId === p.card.event.normalizedSessionId && run.runnerInstanceId === p.card.event.sourceInstanceId + && issue.id === scope.issueId && issue.companyId === scope.companyId && issue.status === "in_progress" + && run.resultJson?.startupCancellation == null && run.resultJson?.nativeCancellation == null, "run is not original active native work"); + demand(!p.rows.some(x => terminalTypes.has(x.event.eventType) || closureTypes.has(x.event.eventType) + || x.event.eventType === "tool.execution.completed" && x.event.payload.executionId === p.executionId), "permission or provider already settled"); + return { schema: "paperclip.e2e.kimi-native-write-pending.v1", scope, requestId: p.request.requestId, toolCallId: p.request.details.toolCallId, + executionId: p.executionId, itemId: p.request.itemId, turnId: p.card.event.turnId, normalizedSessionId: p.card.event.normalizedSessionId, + sourceInstanceId: p.card.event.sourceInstanceId, requestSourceSeq: p.card.event.sourceSeq, startedSourceSeq: p.started.event.sourceSeq, + requestRowSha256: hash(p.card.row), startedRowSha256: hash(p.started.row), observedMonotonicNs: process.hrtime.bigint().toString() }; +} +export function assertSameKimiPending(before: KimiLifecyclePending, after: KimiLifecyclePending) { + demand(canonicalJson({ ...before, observedMonotonicNs: null }) === canonicalJson({ ...after, observedMonotonicNs: null }), "pending callback changed before action"); +} +export function readKimiLifecycleSettlement(input: KimiLifecycleState & { pending: KimiLifecyclePending; branch: "stop" | "loss"; dispatchMonotonicNs: string; caller?: ActiveStopCaller; cancellationRequestId?: string }) { + const b = input.pending, { run, issue } = input, p = origin(input.events, b.scope); + demand(b.schema === "paperclip.e2e.kimi-native-write-pending.v1" && hash(p.card.row) === b.requestRowSha256 && hash(p.started.row) === b.startedRowSha256 + && p.request.requestId === b.requestId && p.request.details.toolCallId === b.toolCallId && p.executionId === b.executionId && p.request.itemId === b.itemId + && p.card.event.turnId === b.turnId && p.card.event.normalizedSessionId === b.normalizedSessionId && p.card.event.sourceInstanceId === b.sourceInstanceId + && p.card.event.sourceSeq === b.requestSourceSeq && p.started.event.sourceSeq === b.startedSourceSeq + && /^[1-9][0-9]{0,29}$/u.test(b.observedMonotonicNs) && /^[1-9][0-9]{0,29}$/u.test(input.dispatchMonotonicNs) + && BigInt(b.observedMonotonicNs) < BigInt(input.dispatchMonotonicNs), "original pre-action pending receipt changed or missing"); + const closed = p.rows.filter(x => closureTypes.has(x.event.eventType)), terminal = p.rows.filter(x => terminalTypes.has(x.event.eventType)); + demand(closed.length === 1 && terminal.length === 1, "one callback closure and one provider terminal required"); + const c = closed[0]!, t = terminal[0]!, cp = rec(c.event.payload), tp = rec(t.event.payload); + demand(p.stream(c.event) && p.stream(t.event) && cp.requestId === b.requestId && cp.turnId === b.turnId && cp.itemId === b.itemId + && cp.requestKind === "permission_approval" && cp.action === undefined && cp.response === undefined && cp.replayAllowed !== true + && c.event.sourceSeq > Math.max(b.requestSourceSeq, b.startedSourceSeq) && t.event.sourceSeq > c.event.sourceSeq + && p.rows.filter(x => x.event.eventType.startsWith("tool.execution.")).every(x => x.event.sourceSeq < t.event.sourceSeq) + && !p.rows.some(x => ["completion.accepted", "task.completed", "run.result.proposed"].includes(x.event.eventType)), "callback accepted, replayed, or outlived its provider"); + demand(run.id === b.scope.runId && run.companyId === b.scope.companyId && run.nativeIssueId === b.scope.issueId && run.runtimeMode === "native" + && run.nativeSessionId === b.normalizedSessionId && run.runnerInstanceId === b.sourceInstanceId && issue.id === b.scope.issueId && issue.companyId === b.scope.companyId, "run ownership or generation changed"); + if (input.branch === "stop") { + const caller = input.caller, stop = rec(run.resultJson?.nativeCancellation), startup = rec(run.resultJson?.startupCancellation); + demand(caller?.type === "board" && caller.userId === "local-board" && caller.source === "local_implicit" + && /^[a-f0-9]{8}(?:-[a-f0-9]{4}){3}-[a-f0-9]{12}$/u.test(input.cancellationRequestId ?? "") + && c.event.eventType === "runtime_request.cancelled" && cp.reason === "turn_terminal" + && t.event.eventType === "turn.cancelled" && tp.status === "cancelled" && tp.error === null && run.status === "cancelled" && issue.status === "in_progress" + && startup.requestedBy?.type === caller.type && startup.requestedBy?.userId === caller.userId && startup.cancellationRequestId === input.cancellationRequestId + && stop.schema === "paperclip.native-cancellation.v1" && stop.intentId === `native-cancellation:${input.cancellationRequestId}` && stop.companyId === b.scope.companyId && stop.runId === run.id && stop.issueId === issue.id + && stop.scope === "run" && stop.dispatched === true && stop.dispatchState === "acknowledged" && stop.reasonCode === "cancellation_run_only" + && Array.isArray(stop.effects) && stop.effects.length === 1 && stop.effects[0] === "release_run_resources" + && id(stop.intentAuditId) && id(stop.acknowledgementAuditId) && stop.intentAuditId !== stop.acknowledgementAuditId, "caller-owned Stop acknowledgement missing"); + } else demand(c.event.eventType === "runtime_request.expired" && cp.reason === "provider_process_lost" && cp.requestType === "permission" && cp.replayAllowed === false + && cp.request === undefined && t.event.eventType === "turn.failed" && run.status === "failed" && issue.status === "blocked" + && run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null, "spontaneous native-worker loss did not expire the original callback"); + return { schema: "paperclip.e2e.kimi-native-write-settlement.v1", branch: input.branch, pending: b, + requestClosure: c.event.eventType, closedRowSha256: hash(c.row), terminalRowSha256: hash(t.row), replayAllowed: false, completedMutation: false }; +} diff --git a/tests/runner-e2e/kimi-lifecycle-tasks.ts b/tests/runner-e2e/kimi-lifecycle-tasks.ts new file mode 100644 index 00000000000..958084af199 --- /dev/null +++ b/tests/runner-e2e/kimi-lifecycle-tasks.ts @@ -0,0 +1,14 @@ +import type { RunnerTaskFixture } from "./types.js"; + +export const kimiLifecycleTasks: readonly RunnerTaskFixture[] = (["stop", "loss"] as const).map(branch => ({ + id: branch === "stop" ? "native-write-permission-stop" : "native-write-provider-loss", + label: branch === "stop" ? "Stop unanswered native Kimi Write" : "Lose the native Kimi worker during unanswered Write", + groups: [], workMode: "standard", flow: "kimi_native_lifecycle", expectedRunCount: 1, + attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, turnTimeoutMs: 120_000, + expectedTerminalState: { issue: branch === "stop" ? "in_progress" : "blocked", run: branch === "stop" ? "cancelled" : "failed" }, + automaticRetryPolicy: "single_attempt", + buildTitle: nonce => `Kimi native Write ${branch} ${nonce}`, + buildVisibleMarker: nonce => `KIMI-NATIVE-WRITE-${branch.toUpperCase()}-${nonce}`, + buildPrompt: () => "Use the native Kimi Write tool exactly once for the supplied absolute path and content MUST_NOT_EXIST. Leave its permission request unanswered for the board operator. Do not use Bash, Edit, delegation, another path or any semantic tool. Do not reject, approve, retry, cancel yourself, call paperclip_finish, or claim completion. The operator has already supplied the exact task title. This unfinished task will end while your real Write permission is still pending.", + buildMatchers: () => [], +})); diff --git a/tests/runner-e2e/kimi-lifecycle.test.ts b/tests/runner-e2e/kimi-lifecycle.test.ts new file mode 100644 index 00000000000..e2f39e8cd38 --- /dev/null +++ b/tests/runner-e2e/kimi-lifecycle.test.ts @@ -0,0 +1,250 @@ +import { createHash } from "node:crypto"; +import { execFileSync, spawn } from "node:child_process"; +import { statSync, realpathSync } from "node:fs"; +import { describe, expect, it } from "vitest"; +import { assertSameKimiPending, observeKimiLifecyclePending, readKimiLifecycleSettlement } from "./kimi-lifecycle-evidence.js"; +import { dispatchKimiLifecycleAction } from "./kimi-lifecycle.js"; +import { KIMI_MAC_PROCESS_OBSERVER, readKimiLocalProcesses, selectKimiLocalTarget, type KimiLocalProcess } from "./kimi-local-provider-fault.js"; + +type Row = Record; +const caller = { type: "board", userId: "local-board", source: "local_implicit" } as const; +const cancellationRequestId = "11111111-2222-4333-8444-555555555555"; +const sha = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`; +function fixture(remote = false) { + const scope = { companyId: "company", issueId: "issue", runId: "run", target: "denied/target.txt", ...(remote ? { actionFile: `.paperclip-eval-action-${"a".repeat(36)}.txt` } : {}) }; + const run: Row = { id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", nativeSessionId: "session", runnerInstanceId: "runner", resultJson: {} }; + const issue: Row = { id: "issue", companyId: "company", status: "in_progress" }; + const row = (eventType: string, sourceSeq: number, payload: Row): Row => ({ companyId: "company", runId: "run", seq: sourceSeq, eventType, protocolSchemaVersion: 1, + payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", normalizedSessionId: "session", + sourceInstanceId: "runner", sourceSeq, sourceEventId: `runner:run:${sourceSeq}`, emittedAt: "2026-10-28T00:00:00Z", payload } } }); + const tool = (executionId: string, name: string, operation: string, target: string | null, status = "running") => ({ schema: "paperclip.tool.execution.v1", executionId, transport: "builtin", name, operation, target, status }); + const request: Row = { schema: "paperclip.runtime_request.v2", type: "permission", requestKind: "permission_approval", status: "pending", requestId: "request", turnId: "turn", itemId: "permission-item", + prompt: `Change file: ${scope.target}`, choices: [{ key: "accept", label: "Allow once" }, { key: "decline", label: "Deny" }, { key: "cancel", label: "Cancel" }], + details: { toolCallId: "0:native-write" }, origin: { adapter: "acpx-runtime-sidecar", provider: "kimi", method: "session/request_permission" } }; + const events: Row[] = remote ? [row("tool.execution.started", 1, tool("0:read", "Read", "read", scope.actionFile!)), row("tool.execution.completed", 2, tool("0:read", "Read", "read", scope.actionFile!, "completed"))] : []; + events.push(row("tool.execution.started", events.length + 1, tool("0:native-write", "Write", "edit", null)), + row("tool.execution.progressed", events.length + 2, tool("0:native-write", "Write", "edit", null)), + row("runtime_request.created", events.length + 3, { request })); + const state = () => ({ run, issue, events }); + const observe = () => observeKimiLifecyclePending({ ...state(), scope }); + const settle = (branch: "stop" | "loss", cancelId = cancellationRequestId) => { + const pending = observe(); + if (branch === "stop") { + run.status = "cancelled"; + run.resultJson = { startupCancellation: { cancellationRequestId: cancelId, requestedBy: { type: "board", userId: "local-board" } }, nativeCancellation: { + schema: "paperclip.native-cancellation.v1", intentId: `native-cancellation:${cancelId}`, companyId: "company", runId: "run", issueId: "issue", scope: "run", + dispatched: true, dispatchState: "acknowledged", reasonCode: "cancellation_run_only", effects: ["release_run_resources"], intentAuditId: "intent-audit", acknowledgementAuditId: "ack-audit", + } }; + events.push(row("runtime_request.cancelled", events.length + 1, { requestId: "request", requestKind: "permission_approval", turnId: "turn", itemId: "permission-item", reason: "turn_terminal" })); + events.push(row("turn.cancelled", events.length + 1, { status: "cancelled", error: null })); + } else { + run.status = "failed"; issue.status = "blocked"; + events.push(row("runtime_request.expired", events.length + 1, { requestId: "request", requestKind: "permission_approval", requestType: "permission", turnId: "turn", itemId: "permission-item", reason: "provider_process_lost", replayAllowed: false })); + events.push(row("turn.failed", events.length + 1, { error: { code: "provider_transport_lost" } })); + } + const dispatchMonotonicNs = process.hrtime.bigint().toString(); + return { pending, dispatchMonotonicNs, caller, cancellationRequestId: cancelId, branch }; + }; + return { scope, run, issue, events, request, row, tool, state, observe, settle }; +} +const payload = (row: Row) => row.payload.prpEvent.payload; + +function retainedLocalStopRows() { + const scope = { companyId: "26725944-fac6-4382-883c-a57d5448a3ba", issueId: "9334a1c6-3f67-4536-8570-e81db0b70498", + runId: "209bc4ba-4f54-4ff8-98b2-49879cfb52f8", target: "pc-denied-6WPcjH/kimi-stop-c5be66dab0a0-1.txt" }; + const session = "18298d7f-4d8f-4185-8ae7-660daa69ead7", source = "38cdabd8-af4c-4c76-9f20-9931e36475df", + turn = "turn_lab_8e3a1a4a520442ad80c36025a1353ba6", item = `item-${scope.runId}`, toolCallId = "0:tool_uKx4azxIHxpU6e5TZ2xLHbVj"; + const origin = { method: "session/request_permission", adapter: "acpx-runtime-sidecar", provider: "kimi" }; + const choices = [{ key: "accept", label: "Allow once" }, { key: "decline", label: "Deny" }, { key: "cancel", label: "Cancel" }]; + const request = { type: "permission", itemId: item, method: "session/request_permission", origin, + prompt: `Change file: ${scope.target}`, schema: "paperclip.runtime_request.v2", status: "pending", turnId: turn, choices, + details: { itemId: item, origin, reason: `Change file: ${scope.target}`, turnId: turn, choices, + threadId: "paperclip-9558eb1e0ea79ed9b728d62643b53ab02477d5a4711ef634d8cf0c5b5d8d0af9", toolCallId }, + requestId: "acpx-input-25f63168069f72f421df7bd9", requestKind: "permission_approval" }; + const row = (eventType: string, seq: number, sourceSeq: number, emittedAt: string, value: Row): Row => ({ + companyId: scope.companyId, runId: scope.runId, seq, eventType, protocolSchemaVersion: 1, + payload: { prpEvent: { runId: scope.runId, itemId: item, schema: "paperclip.prp.event.v1", turnId: turn, payload: value, + priority: 1, emittedAt, eventType, sourceSeq, sourceKind: "runner", schemaVersion: 1, + sourceEventId: `${source}:${scope.runId}:${sourceSeq}`, sourceInstanceId: source, normalizedSessionId: session } }, + }); + const events = [row("tool.execution.started", 59, 30, "2026-10-28T14:48:05.538Z", { + name: "Write", output: "null", schema: "paperclip.tool.execution.v1", status: "running", target: null, exitCode: null, + progress: "Write (pending)", readOnly: false, namespace: null, operation: "edit", transport: "builtin", durationMs: null, + executionId: toolCallId, outputBytes: 4, inputUpdated: false, + outputDigest: "sha256:74234e98afe7498fb5daf1f36ac2d78acc339464f950703b8c019892f982b90b", outputTruncated: false, + }), row("runtime_request.created", 141, 112, "2026-10-28T14:48:06.604Z", { request })]; + return { scope, events, + run: { id: scope.runId, companyId: scope.companyId, nativeIssueId: scope.issueId, status: "running", runtimeMode: "native", + nativeSessionId: session, runnerInstanceId: source, resultJson: null }, + issue: { id: scope.issueId, companyId: scope.companyId, status: "in_progress" } }; +} + +describe("Kimi native Write lifecycle evidence", () => { + it("accepts retained local Stop rows with a null execution target and no permission kind", () => { + const actual = retainedLocalStopRows(); + expect(payload(actual.events[0]!).target).toBeNull(); + expect(payload(actual.events[1]!).request.details).not.toHaveProperty("kind"); + expect(observeKimiLifecyclePending(actual)).toMatchObject({ + requestId: "acpx-input-25f63168069f72f421df7bd9", toolCallId: "0:tool_uKx4azxIHxpU6e5TZ2xLHbVj", + executionId: "0:tool_uKx4azxIHxpU6e5TZ2xLHbVj", startedSourceSeq: 30, requestSourceSeq: 112, + }); + }); + it.each(["target", "kind", "callback", "edit-operation", "tool-name", "prompt"])("rejects conflicting retained local Stop %s", conflict => { + const actual = retainedLocalStopRows(), tool = payload(actual.events[0]!), request = payload(actual.events[1]!).request; + if (conflict === "target") tool.target = "another.txt"; + if (conflict === "kind") request.details.kind = "execute"; + if (conflict === "callback") request.details.toolCallId = "0:other"; + if (conflict === "edit-operation") tool.operation = "execute"; + if (conflict === "tool-name") tool.name = "Edit"; + if (conflict === "prompt") request.prompt = "Change file: another.txt"; + expect(() => observeKimiLifecyclePending(actual)).toThrow(); + }); + it("accepts explicit matching edit metadata without requiring it on the native permission", () => { + const f = fixture(); + f.request.details.kind = "edit"; payload(f.events[1]!).target = f.scope.target; + expect(f.observe().executionId).toBe("0:native-write"); + }); + it.each([false, true])("binds actual Write approval to its native edit execution (remote=%s)", remote => { + const f = fixture(remote), before = f.observe(), later = f.observe(); + expect(before).toMatchObject({ toolCallId: "0:native-write", executionId: "0:native-write", normalizedSessionId: "session", sourceInstanceId: "runner" }); + expect(() => assertSameKimiPending(before, later)).not.toThrow(); + }); + it.each([ + ["another provider", (f: ReturnType) => { f.request.origin.provider = "copilot"; }], + ["semantic permission", f => { f.request.origin.method = "semantic/tool"; }], + ["missing safe card path", f => { f.request.prompt = "Write"; }], + ["conflicting permission kind", f => { f.request.details.kind = "other"; }], + ["session-wide authority", f => { f.request.choices.push({ key: "accept_for_session" }); }], + ["missing decline", f => { f.request.choices = f.request.choices.filter((x: Row) => x.key !== "decline"); }], + ["wrong native call", f => { f.request.details.toolCallId = "foreign"; }], + ["different file", f => { payload(f.events[1]!).target = "elsewhere.txt"; }], + ["native Edit substituted for Write", f => { payload(f.events[1]!).name = "Edit"; }], + ["completed file effect", f => { f.events.push(f.row("tool.execution.completed", 4, f.tool("0:native-write", "Write", "edit", f.scope.target, "completed"))); }], + ["extra shell", f => { f.events.unshift(f.row("tool.execution.started", 0, f.tool("shell", "Bash", "execute", null))); }], + ["replayed start", f => { f.events.push(f.row("tool.execution.started", 4, f.tool("0:native-write", "Write", "edit", f.scope.target))); }], + ["foreign company", f => { f.events[0]!.companyId = "foreign"; }], + ["foreign source", f => { f.events[0]!.payload.prpEvent.sourceInstanceId = "other"; }], + ["restored session", f => { f.run.nativeSessionId = "replacement"; }], + ["another provider generation", f => { f.run.runnerInstanceId = "replacement"; }], + ] satisfies Array<[string, (f: ReturnType) => void]>)("rejects %s", (_name, mutate) => { + const f = fixture(); mutate(f); expect(f.observe).toThrow(); + }); + it.each(["wrong-path", "unclosed", "extra-write", "post-write-read"])("rejects ambiguous bootstrap %s", kind => { + const f = fixture(true); + if (kind === "wrong-path") payload(f.events[0]!).target = "another.txt"; + if (kind === "unclosed") f.events.splice(1, 1); + if (kind === "extra-write") payload(f.events[0]!).operation = "edit"; + if (kind === "post-write-read") { f.events[0]!.seq = 99; } + expect(f.observe).toThrow(); + }); + it.each(["stop", "loss"] as const)("accepts the original %s callback closure", branch => { + const f = fixture(), binding = f.settle(branch); + expect(readKimiLifecycleSettlement({ ...f.state(), ...binding })).toMatchObject({ branch, replayAllowed: false, completedMutation: false }); + payload(f.events.at(-2)!).requestId = "other"; + expect(() => readKimiLifecycleSettlement({ ...f.state(), ...binding })).toThrow(); + }); + it("keeps the observed permission row bound when native expiry omits its full request", () => { + const f = fixture(), binding = f.settle("loss"); + expect(payload(f.events.at(-2)!)).not.toHaveProperty("request"); + expect(readKimiLifecycleSettlement({ ...f.state(), ...binding })).toMatchObject({ branch: "loss", replayAllowed: false }); + f.request.choices[0].label = "Changed after pending observation"; + expect(() => readKimiLifecycleSettlement({ ...f.state(), ...binding })).toThrow("original pre-action pending receipt changed or missing"); + }); + it.each(["replay", "cancelled", "wrong-reason", "changed-request", "injected-original-request", "new-turn", "new-operation", "success", "Stop"])("rejects false native loss %s", kind => { + const f = fixture(), binding = f.settle("loss"), expiry = f.events.at(-2)!; + if (kind === "replay") payload(expiry).replayAllowed = true; + if (kind === "cancelled") expiry.eventType = expiry.payload.prpEvent.eventType = "runtime_request.cancelled"; + if (kind === "wrong-reason") payload(expiry).reason = "durable_handoff"; + if (kind === "changed-request") payload(expiry).request = { ...f.request, prompt: "replacement" }; + if (kind === "injected-original-request") payload(expiry).request = f.request; + if (kind === "new-turn") { f.events.at(-1)!.payload.prpEvent.turnId = "replacement"; } + if (kind === "new-operation") f.events.push(f.row("tool.execution.started", 6, f.tool("late", "Write", "edit", f.scope.target))); + if (kind === "success") f.run.status = "succeeded"; + if (kind === "Stop") f.run.resultJson.nativeCancellation = {}; + expect(() => readKimiLifecycleSettlement({ ...f.state(), ...binding })).toThrow(); + }); + it("refuses a changed pending callback before dispatching a Stop or fault", async () => { + const f = fixture(); let reads = 0, actions = 0; + await expect(dispatchKimiLifecycleAction({ scope: f.scope, caller, branch: "stop", deadlineAt: Date.now() + 1000, + load: async () => { if (++reads === 2) f.request.requestId = "replacement"; return f.state(); }, retain: async () => {}, + act: async () => { actions++; } })).rejects.toThrow("pending callback changed"); + expect(actions).toBe(0); + }); + it("retains and refreshes the pending observation before its only mutating action", async () => { + const f = fixture(), order: string[] = []; + const result = await dispatchKimiLifecycleAction({ scope: f.scope, caller, branch: "stop", deadlineAt: Date.now() + 1000, + load: async () => { order.push("load"); return f.state(); }, retain: async () => { order.push("retain"); }, + act: async (_pending, requestId) => { order.push("action"); f.settle("stop", requestId); return { source: "public-Stop-test-double" }; } }); + expect(order.slice(0, 4)).toEqual(["load", "retain", "load", "action"]); + expect(order.filter(x => x === "action")).toHaveLength(1); + expect(result.settlement.branch).toBe("stop"); + }); +}); + +function localFixture() { + const timestamp = Date.parse("2026-10-28T00:00:00Z"), distribution = "/private/tmp/paperclip-acpx-native-fixture/distribution"; + const bootId = "12345678-1234-1234-1234-123456789abc", loader = "exact compiled guarded loader"; + const proc = (pid: number, ppid: number, executable: string, command: string): KimiLocalProcess => ({ pid, ppid, pgid: pid === 21 ? 21 : 22, + startUs: String(timestamp * 1000 + pid), executable, executableDev: "1", executableIno: String(pid), command, argv: [executable], cwd: "/workspace", bootId }); + const processes = [proc(21, 1, "/runner", "/runner --run-id run --lifecycle-mode per_turn"), + proc(22, 21, "/host-node", `/host-node --eval ${loader} ${distribution} .paperclip-native-entry.cjs 0 0 commonjs [] 0`), + proc(23, 22, `${distribution}/node/bin/node`, "kimi-code")]; + processes[1]!.argv = ["/host-node", "--eval", loader, distribution, ".paperclip-native-entry.cjs", "0", "0", "commonjs", "[]", "0"]; + const input = { authority: { pid: 21, groupId: 21, startedAt: new Date(timestamp).toISOString(), runId: "run" }, + journal: processes.map(row => ({ pid: row.pid, parent: row.ppid, start: new Date(timestamp).toISOString() })), processes, workspacePath: "/workspace", guardLoaderSha256: sha(loader) }; + return { input, select: () => selectKimiLocalTarget(input) }; +} +describe("Kimi local native child selector", () => { + it("selects the exact pinned-path native descendant independently of its title", () => { + const f = localFixture(); expect(f.select().target.pid).toBe(23); + f.input.processes[2]!.command = "anything"; expect(f.select().target.pid).toBe(23); + }); + it("uses byte-exact loader argv even when ps renders a different command", () => { + const f = localFixture(); f.input.processes[1]!.command = "a rendered command that cannot preserve loader bytes"; + expect(f.select().target.pid).toBe(23); + }); + it("decodes exactly argc arguments and excludes the trailing environment", () => { + const source = KIMI_MAC_PROCESS_OBSERVER.slice(KIMI_MAC_PROCESS_OBSERVER.indexOf("def decode_arguments(data):"), KIMI_MAC_PROCESS_OBSERVER.indexOf("def arguments(pid):")); + const run = (bytes: Buffer) => execFileSync("/usr/bin/python3", ["-I", "-B", "-c", `import base64,json,struct,sys\n${source}\nprint(json.dumps(decode_arguments(base64.b64decode(sys.argv[1]))))`, bytes.toString("base64")], { encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }); + const args = ["/node", "--eval", "line-one\nline-two", ""]; + const header = Buffer.alloc(4); header.writeInt32LE(args.length); + const body = Buffer.from(`/node\0\0${args.join("\0")}\0CANARY=must-never-be-returned\0`); + const decoded = run(Buffer.concat([header, body])); expect(JSON.parse(decoded)).toEqual(args); expect(decoded).not.toContain("CANARY"); + for (const count of [0, 129]) { const wrong = Buffer.from(header); wrong.writeInt32LE(count); expect(() => run(Buffer.concat([wrong, body]))).toThrow(); } + const changed = Buffer.from(header); changed.writeInt32LE(5); + expect(() => run(Buffer.concat([changed, Buffer.from(`/node\0\0${args.join("\0")}\0`) ]))).toThrow(); + expect(() => run(Buffer.concat([header, Buffer.from("/node\0\0/node\0--eval\0truncated")]))).toThrow(); + expect(() => run(Buffer.alloc(2))).toThrow(); + }); + it.skipIf(process.platform !== "darwin" || process.arch !== "arm64")("reads literal multiline argv without retaining a real child's environment", async () => { + const program = 'process.stdout.write("ready");setInterval(()=>{},1000)'; + const literal = "line-one\nline-two"; + const child = spawn(process.execPath, ["--eval", program, literal], { env: { PATH: "/usr/bin:/bin", CANARY: "must-never-be-returned" }, stdio: ["ignore", "pipe", "ignore"] }); + const closed = new Promise(resolve => child.once("close", resolve)); + try { + await new Promise(resolve => child.stdout!.once("data", resolve)); + const rows = readKimiLocalProcesses([child.pid!]); + expect(rows[0]!.argv).toEqual([process.execPath, "--eval", program, literal]); + expect(JSON.stringify(rows)).not.toContain("must-never-be-returned"); + } finally { child.kill("SIGTERM"); await closed; } + }); + it.each(["root", "foreign-loader", "wrong-entry", "unowned", "wrong-cwd", "boot", "cycle", "ambiguous"])("fails closed for %s", kind => { + const f = localFixture(); + if (kind === "root") f.input.authority.runId = "other"; + if (kind === "foreign-loader") f.input.processes[1]!.argv[2] = "wrong loader"; + if (kind === "wrong-entry") f.input.processes[1]!.argv[4] = "foreign.cjs"; + if (kind === "unowned") f.input.journal.pop(); + if (kind === "wrong-cwd") f.input.processes[2]!.cwd = "/elsewhere"; + if (kind === "boot") f.input.processes[2]!.bootId = "foreign"; + if (kind === "cycle") f.input.processes[1]!.ppid = 23; + if (kind === "ambiguous") { f.input.processes.push({ ...f.input.processes[2]!, pid: 24 }); f.input.journal.push({ ...f.input.journal[2]!, pid: 24 }); } + expect(f.select).toThrow(); + }); + it.skipIf(process.platform !== "darwin" || process.arch !== "arm64")("reads real macOS kernel process birth and executable inode without environment inspection", () => { + const rows = readKimiLocalProcesses([process.pid]), current = rows.find(row => row.pid === process.pid); + expect(current?.executable).toBe(realpathSync(process.execPath)); + const stat = statSync(current!.executable); + expect(current?.executableDev).toBe(String(stat.dev)); expect(current?.executableIno).toBe(String(stat.ino)); + expect(Math.abs(Date.now() - Number(current!.startUs) / 1000 - process.uptime() * 1000)).toBeLessThan(5000); + }); +}); diff --git a/tests/runner-e2e/kimi-lifecycle.ts b/tests/runner-e2e/kimi-lifecycle.ts new file mode 100644 index 00000000000..6edba6e6ec0 --- /dev/null +++ b/tests/runner-e2e/kimi-lifecycle.ts @@ -0,0 +1,210 @@ +import { kimiContextRendezvousPrompt, pendingKimiContextPermission, publishKimiContextContinuation, approveKimiContextThroughUi, readKimiContextProof, type KimiContextProof } from "./kimi-context-rendezvous.js"; +import { kimiPendingWritePermissionCard } from "./kimi-permission-card.js"; +import { createKimiContextCheckpointObserver } from "./kimi-context-checkpoint.js"; +import { randomUUID } from "node:crypto"; +import { join } from "node:path"; +import { expect, type Page } from "@playwright/test"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { waitForEvalAgentReady } from "./budgeted-setup.js"; +import { collectRunEvents } from "./run-observations.js"; +import { createTaskThroughUi } from "./user-actions.js"; +import { createDeniedTargetFixture, exists, observeRunProcesses } from "./native-local-fixtures.js"; +import { assertNativeRemoteRetirement, nativeRemoteDeniedSample, prepareNativeRemoteAction, type NativeRemoteSnapshot } from "./native-remote-evidence.js"; +import type { RemoteNativeBootstrap } from "./remote-native-bootstrap.js"; +import type { RemoteNativeFixture } from "./remote-native-fixtures.js"; +import { readActiveStopCaller, type ActiveStopCaller } from "./native-active-stop-evidence.js"; +import { assertSameKimiPending, observeKimiLifecyclePending, readKimiLifecycleSettlement, type KimiLifecyclePending, type KimiLifecycleScope, type KimiLifecycleState } from "./kimi-lifecycle-evidence.js"; +import { terminateKimiLocalProvider } from "./kimi-local-provider-fault.js"; +import { kimiLifecycleTasks } from "./kimi-lifecycle-tasks.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import type { MatrixExecution } from "./types.js"; + +type Row = Record; +type Check = { id: string; passed: boolean; detail: string }; +export async function dispatchKimiLifecycleAction(input: { + scope: KimiLifecycleScope; branch: "stop" | "loss"; caller: ActiveStopCaller; deadlineAt: number; + load(): Promise; retain(value: KimiLifecyclePending): Promise; + act(pending: KimiLifecyclePending, cancellationRequestId: string): Promise; +}) { + const pending = observeKimiLifecyclePending({ ...await input.load(), scope: input.scope }); + await input.retain(pending); + assertSameKimiPending(pending, observeKimiLifecyclePending({ ...await input.load(), scope: input.scope })); + if (Date.now() >= input.deadlineAt) throw new Error("Kimi native lifecycle: pending action deadline expired"); + const cancellationRequestId = randomUUID(), dispatchMonotonicNs = process.hrtime.bigint().toString(); + const actionReceipt = await input.act(pending, cancellationRequestId); + const binding = { pending, branch: input.branch, caller: input.caller, cancellationRequestId, dispatchMonotonicNs }; + const state = await pollUntil({ label: `Kimi pending native Write ${input.branch} settlement`, deadlineAt: input.deadlineAt, intervalMs: 200, + load: input.load, reject: value => ["succeeded", "timed_out", input.branch === "stop" ? "failed" : "cancelled"].includes(value.run.status) ? "Unexpected native lifecycle terminal" : undefined, + accept: value => { readKimiLifecycleSettlement({ ...value, ...binding }); return true; } }); + return { ...binding, actionReceipt, settlement: readKimiLifecycleSettlement({ ...state, ...binding }) }; +} + +export async function runKimiLifecycleFlow(input: { + page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number; + remoteBootstrap?: RemoteNativeBootstrap; + registerCleanupAssertion(callback: () => Promise): void; + registerBeforeEnvironmentTeardownAssertion(callback: () => Promise): void; + observe(issue: Row, runs: Row[]): void; + capture(id: string, label: string, file: string): Promise; + evidence(name: string, value: unknown): Promise; +}) { + const { api, page, execution, fixtures, nonce } = input; + if (execution.suite.id !== "kimi-qualification" || execution.profile.qualificationCandidate !== "kimi" + || !kimiLifecycleTasks.some(task => task.id === execution.task.id)) throw new Error("Unknown Kimi native lifecycle task"); + const branch = execution.task.id === "native-write-permission-stop" ? "stop" : "loss", remote = execution.environment.id === "daytona"; + if ((!remote && execution.environment.id !== "local") || remote && !input.remoteBootstrap) throw new Error("Kimi lifecycle requires its admitted isolated environment"); + const checks: Check[] = [], samples: Array<{ phase: string; absent: boolean }> = []; + const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); }; + const name = `kimi-${branch}-${nonce}.txt`, local = remote ? undefined : await createDeniedTargetFixture(input.workspacePath, name), target = local?.targetRelativePath ?? name; + const observer = remote ? undefined : observeRunProcesses(); + let issue: Row = {}, runs: Row[] = [], events: Row[] = [], processIdentity: string | undefined, processError = false; + let processes: ReturnType["sample"]> = { captured: false, journal: [], live: [] }; + let fixture: RemoteNativeFixture | undefined, baseline: NativeRemoteSnapshot | undefined, pendingSnapshot: NativeRemoteSnapshot | undefined, sealed: NativeRemoteSnapshot | undefined; + let completed: Awaited> | undefined; + let contextProof: KimiContextProof | undefined, publishedAction: string | undefined; + const authority = () => { const run = runs[0]!; return { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id }; }; + const observeProcesses = () => { + if (!observer || !runs[0]?.processPid) return; + const value = authority(), identity = JSON.stringify(value); + if (processIdentity && processIdentity !== identity) processError = true; + processIdentity ??= identity; processes = observer.sample(value); + }; + const load = async (): Promise => { + if (issue.id) issue = await api.get(`/api/issues/${issue.id}`); + const list = await api.get(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`); + runs = await Promise.all(list.map(run => api.get(`/api/heartbeat-runs/${run.id}`))); + if (runs.length > 1) throw new Error("Kimi native lifecycle dispatched an extra run"); + events = runs[0] ? await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : []; + observeProcesses(); input.observe(issue, runs); return { issue, run: runs[0] ?? {}, events }; + }; + const scope = (): KimiLifecycleScope => ({ companyId: fixtures.company.id, issueId: issue.id, runId: runs[0]!.id, target, ...(fixture ? { actionFile: fixture.actionFile } : {}), ...(contextProof ? { contextProof } : {}) }); + const sample = async (phase: string) => { + const absent = !await exists(local!.targetPath); samples.push({ phase, absent }); check(`no-effect-${phase}`, absent, "Exact unapproved Write target remains absent"); + }; + const timer = observer ? setInterval(() => { try { observeProcesses(); } catch { processError = true; } }, 100) : undefined; + (remote ? input.registerBeforeEnvironmentTeardownAssertion : input.registerCleanupAssertion)(async () => { + const receipts: Check[] = []; + try { + await load(); + if (!completed) throw new Error("Kimi native lifecycle settlement was not established"); + if (remote) { + if (!fixture || !baseline || !pendingSnapshot) throw new Error("Kimi remote watcher was not armed before the native action"); + sealed ??= await fixture.finish(); assertNativeRemoteRetirement(sealed, baseline); assertNativeRemoteRetirement(sealed, pendingSnapshot); + for (const snapshot of [baseline, pendingSnapshot, sealed]) if (nativeRemoteDeniedSample(snapshot, baseline, target, "terminal").exists) throw new Error("Kimi native Write had a remote file effect"); + if (!(BigInt(baseline.observedMonotonicNs) < BigInt(pendingSnapshot.observedMonotonicNs) && BigInt(pendingSnapshot.observedMonotonicNs) < BigInt(sealed.observedMonotonicNs))) throw new Error("Kimi remote causal samples were reused or reordered"); + for (const snapshot of [baseline, pendingSnapshot]) if (!snapshot.processes.journal.every(p => sealed!.processes.journal.some(q => p.pid === q.pid && p.startTicks === q.startTicks && p.bootId === q.bootId))) throw new Error("Kimi remote process journal lost a native descendant"); + } else { + await pollUntil({ label: "Kimi native owned process retirement", deadlineAt: Date.now() + 5000, intervalMs: 100, + load: async () => { observeProcesses(); return processes; }, accept: value => value.captured && value.live.length === 0 }); + await sample("after-cleanup"); + const watcher = local!.watcher.finish(); + if (processError || !watcher.complete || watcher.targetMutationCount !== 0 || samples.map(row => row.phase).join(",") !== "before-request,pending,terminal,after-cleanup" || !samples.every(row => row.absent)) throw new Error("Kimi local physical retirement or continuous no-effects proof is incomplete"); + } + readKimiLifecycleSettlement({ ...await load(), ...completed }); + receipts.push({ id: "native-worker-retired-without-effects", passed: true, detail: remote ? "Exact owned run tree retired with continuous remote no-effects coverage through its retirement seal" : "Exact owned run tree retired and watched target remained absent through cleanup" }); + return receipts; + } finally { + if (timer) clearInterval(timer); + try { await input.evidence("kimi-native-lifecycle-cleanup.json", { processes, processError, samples, baseline, pendingSnapshot, sealed, watcher: local?.watcher.finish(), receipts }); } + finally { await fixture?.close(); } + } + }); + const prompt = () => `${execution.task.buildPrompt(nonce)}\nThe exact absolute native Write path is ${remote ? join(fixture!.remoteCwd, target) : local!.targetPath}.`; + try { + const caller = readActiveStopCaller(await api.get("/api/health"), await api.get("/api/auth/get-session")); + await waitForEvalAgentReady({ environmentId: execution.environment.id, agentId: fixtures.agent.id, companyId: fixtures.company.id, + deadlineAt: input.deadlineAt, api, observe: observations => input.evidence("kimi-lifecycle-agent-ready.json", { observations }) }); + const agent = await api.get(`/api/agents/${fixtures.agent.id}`); + const configured = await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: "approve-reads", lifecycleMode: "per_turn", timeoutSec: 120 } }); + check("per-turn-native-policy", configured.adapterConfig?.acpxPermissionMode === "approve-reads" && configured.adapterConfig?.lifecycleMode === "per_turn", "Restrictive native permission policy and per-turn ownership selected before dispatch"); + const project = await api.post(`/api/companies/${fixtures.company.id}/projects`, { name: `Kimi native lifecycle ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, workspace: { name: "Primary", sourceType: "local_path", cwd: input.workspacePath, isPrimary: true } }); + if (!remote) await sample("before-request"); + if (remote) input.remoteBootstrap!.prompt(nonce); + const created = await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), + prompt: remote ? kimiContextRendezvousPrompt() : prompt(), workMode: "standard", projectName: project.name, requireExplicitTitle: true }); + issue = (await pollUntil({ label: "browser-created Kimi lifecycle task", deadlineAt: input.deadlineAt, load: async () => (await api.get(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(row => row.id === created.issueId), accept: Boolean }))!; + if (remote) { + await pollUntil({ label: "Kimi native remote bootstrap", deadlineAt: input.deadlineAt, load, accept: value => value.run.status === "running" }); + const bound = await input.remoteBootstrap!.bindAndRelease({ issueId: issue.id, runId: runs[0]!.id, targets: [target], actionPrompt: async value => { + fixture = value; + const prepared = await prepareNativeRemoteAction({ fixture: value, companyId: fixtures.company.id, environmentId: fixtures.environment.id, runId: runs[0]!.id, target, prompt: prompt() }); + baseline = prepared.baseline; publishedAction = prepared.prompt; await input.evidence("kimi-native-before-action-remote.json", baseline); return prepared.prompt; + } }); + if (bound !== fixture) throw new Error("Kimi native action publication changed its fixture identity"); + const contextScope = { companyId: fixtures.company.id, agentId: fixtures.agent.id, environmentId: fixtures.environment.id, issueId: issue.id, runId: runs[0]!.id }; + const pending = await pollUntil({ label: "exact scoped Kimi context permission", deadlineAt: input.deadlineAt, intervalMs: 200, + load: async () => pendingKimiContextPermission((await load()).events, contextScope), accept: Boolean }); + if (!pending || publishedAction === undefined) throw new Error("Kimi context rendezvous lacks its exact action publication"); + const publication = await publishKimiContextContinuation({ pending, action: publishedAction, fixture: fixture!, api }); + await input.evidence("kimi-context-publication.json", publication); + await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`); + const contextObserver = await createKimiContextCheckpointObserver(publication); + try { + await approveKimiContextThroughUi({ page, publication, load, evidence: input.evidence }); + contextProof = await pollUntil({ label: "delivered scoped Kimi context result", deadlineAt: input.deadlineAt, intervalMs: 200, + load: async () => { + const state = await load(), authority = await contextObserver.read(); + return readKimiContextProof(state.events, publication, authority); + }, accept: Boolean }); + } finally { await contextObserver.close(); } + if (!contextProof) throw new Error("Kimi context rendezvous result was not delivered"); + await input.evidence("kimi-context-delivered-proof.json", contextProof); + } + const pendingState = await pollUntil({ label: "unanswered actual Kimi Write permission", deadlineAt: input.deadlineAt, intervalMs: 200, load, + reject: value => ["succeeded", "failed", "cancelled", "timed_out"].includes(value.run.status) ? "Provider ended before actual native Write approval" : undefined, + accept: value => { observeKimiLifecyclePending({ ...value, scope: scope() }); return true; } }); + await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`); + const card = await kimiPendingWritePermissionCard({ page, + pending: observeKimiLifecyclePending({ ...pendingState, scope: scope() }), + readPending: async () => observeKimiLifecyclePending({ ...await load(), scope: scope() }), + }); + await input.capture("kimi-native-write-pending", "Actual native Kimi Write permission remains unanswered", "tool-review-pending.png"); + if (remote) { + pendingSnapshot = await fixture!.snapshot("pending-native-write"); + check("remote-pending-no-effect", !nativeRemoteDeniedSample(pendingSnapshot, baseline!, target, "pending").exists, "Remote native Write is pending without an effect"); + await input.evidence("kimi-native-pending-remote.json", pendingSnapshot); + } else await sample("pending"); + completed = await dispatchKimiLifecycleAction({ scope: scope(), branch, caller, deadlineAt: input.deadlineAt, load, + retain: value => input.evidence("kimi-native-write-pending.json", value), act: async (_pending, cancellationRequestId) => { + if (branch === "stop") { + const response = await api.post(`/api/heartbeat-runs/${runs[0]!.id}/cancel`, { cancellationRequestId }); + if (response.id !== runs[0]!.id || response.resultJson?.nativeCancellation?.intentId !== `native-cancellation:${cancellationRequestId}`) throw new Error("Kimi Stop response names a foreign intent"); + return { intentId: response.resultJson.nativeCancellation.intentId }; + } + if (remote) { + const run = runs[0]!, workspace = run.contextSnapshot?.executionWorkspaceId; + if (typeof workspace !== "string" || run.runnerProfileJson?.nativeExecutionInput?.binding?.executionWorkspaceId !== workspace + || run.contextSnapshot?.paperclipEnvironment?.leaseId !== fixture!.binding.leaseId || fixture!.baseline.scope.observedPrpEnvironmentLeaseId !== workspace + || !fixture!.terminateKimiProvider) throw new Error("Kimi fault lacks exact public run/workspace/lease authority"); + return fixture!.terminateKimiProvider(workspace); + } + if (!processes.captured) throw new Error("Kimi local fault lacks an observed native tree"); + return terminateKimiLocalProvider({ authority: authority(), journal: processes.journal, workspacePath: input.workspacePath }); + } }); + check("original-pending-callback-settled", true, branch === "stop" ? "Caller-owned Stop cancelled the unanswered native callback" : "Exact spontaneous native worker loss expired the unanswered callback and failed the original run"); + await input.evidence("kimi-native-write-settlement.json", completed); + const stale = await api.request.post(`/api/heartbeat-runs/${runs[0]!.id}/runtime-requests/${encodeURIComponent(completed.pending.requestId)}/resolve`, { + data: { turnId: completed.pending.turnId, requestKind: "permission_approval", resolution: { action: "accept" } }, + }); + check("stale-allow-refused", stale.status() === 409, "A late Allow cannot revive the ended provider or replay its Write"); + if (remote) { sealed = await fixture!.finish(); assertNativeRemoteRetirement(sealed, baseline!); await input.evidence("kimi-native-retirement-remote.json", sealed); } + else await sample("terminal"); + await page.reload(); + await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${branch === "stop" ? "In Progress" : "Blocked"}`, exact: false })).toBeVisible(); + const thread = page.getByTestId("task-chat-thread"); + await expect(thread.getByTestId("task-chat-turn-summary").getByText(/^Stopped(?: for .+)?$/)).toBeVisible(); + await expect(card.getByText(`Change file: ${target}`, { exact: true })).toBeVisible(); + await expect(card.getByText(branch === "stop" ? "cancelled" : "expired", { exact: true })).toBeVisible(); + await expect(thread.getByTestId("task-chat-current-activity")).toHaveCount(0); + await expect(thread.getByTestId("task-chat-live-turn")).toHaveCount(0); + await expect(thread.getByTestId("task-chat-live-run-pill")).toHaveCount(0); + await expect(page.getByTestId("task-chat-history-loading")).toHaveCount(0); + await expect(card.getByRole("button", { name: "Allow once", exact: true })).toHaveCount(0); + await expect(card.getByRole("button", { name: "Deny", exact: true })).toHaveCount(0); + readKimiLifecycleSettlement({ ...await load(), ...completed }); + check("no-replacement-run-or-false-completion", runs.length === 1 && issue.status === (branch === "stop" ? "in_progress" : "blocked"), "One unfinished task retains the original terminal; no automatic follow-up or completed mutation"); + await input.capture("final-state", "Ended native Write is no longer answerable", "final-state.png"); + await input.evidence("api-state.json", { issue, run: runs[0], runs, runEvents: events, runEventsByRun: [{ runId: runs[0]!.id, events }], checks, completed }); + return { issue, runs, checks }; + } finally { await input.evidence("kimi-native-lifecycle-checks.json", { issue, runs, samples, checks, completed }); } +} diff --git a/tests/runner-e2e/kimi-local-provider-fault.ts b/tests/runner-e2e/kimi-local-provider-fault.ts new file mode 100644 index 00000000000..28bc5095f3b --- /dev/null +++ b/tests/runner-e2e/kimi-local-provider-fault.ts @@ -0,0 +1,184 @@ +import { execFileSync } from "node:child_process"; +import { createHash } from "node:crypto"; +import { constants, closeSync, fstatSync, lstatSync, openSync, readFileSync, realpathSync } from "node:fs"; +import { dirname, join } from "node:path"; +import { parseNativeAcpxDistributionEntries } from "../../packages/paperclip-runner/src/drivers/acpx/native-distribution-integrity.js"; +import { isPerTurnRunProcess, type RunProcessAuthority } from "./native-local-fixtures.js"; +import { expectedNativeBootstrap, expectedNativeModuleGuard, KIMI_FAULT_ENTRY, kimiFaultRuntimeContract } from "./kimi-provider-fault-contract.js"; + +export interface KimiLocalProcess { pid: number; ppid: number; pgid: number; startUs: string; executable: string; executableDev: string; executableIno: string; command: string; argv: string[]; cwd: string; bootId: string } +interface JournalIdentity { pid: number; parent: number; start: string } +const sha = (value: string | Buffer) => `sha256:${createHash("sha256").update(value).digest("hex")}`; +function demand(value: unknown, code: string): asserts value { if (!value) throw new Error(`kimi_local_provider_fault:${code}`); } +export const KIMI_MAC_PROCESS_OBSERVER = String.raw`import ctypes,json,os,struct,subprocess,sys +class Bsd(ctypes.Structure): + _fields_=[(n,ctypes.c_uint32) for n in ['flags','status','xstatus','pid','ppid','uid','gid','ruid','rgid','svuid','svgid','rfu']]+[('comm',ctypes.c_char*16),('name',ctypes.c_char*32)]+[(n,ctypes.c_uint32) for n in ['nfiles','pgid','pjobc','e_tdev','e_tpgid']]+[('nice',ctypes.c_int32),('start_sec',ctypes.c_uint64),('start_usec',ctypes.c_uint64)] +lib=ctypes.CDLL('/usr/lib/libproc.dylib',use_errno=True) +lib.proc_pidinfo.argtypes=[ctypes.c_int,ctypes.c_int,ctypes.c_uint64,ctypes.c_void_p,ctypes.c_int] +lib.proc_pidpath.argtypes=[ctypes.c_int,ctypes.c_void_p,ctypes.c_uint32] +kernel=ctypes.CDLL('/usr/lib/libSystem.B.dylib',use_errno=True) +kernel.sysctl.argtypes=[ctypes.POINTER(ctypes.c_int),ctypes.c_uint,ctypes.c_void_p,ctypes.POINTER(ctypes.c_size_t),ctypes.c_void_p,ctypes.c_size_t] +def decode_arguments(data): + if not 8<=len(data)<=2097152:raise RuntimeError('argv_size') + count=struct.unpack_from('i',data)[0] + if not 1<=count<=128:raise RuntimeError('argv_count') + at=data.find(b'\0',4) + if at<5:raise RuntimeError('argv_executable') + at+=1 + while at131072:raise RuntimeError('argv_truncated') + total+=end-at + if total>262144:raise RuntimeError('argv_bound') + result.append(data[at:end].decode('utf8'));at=end+1 + return result +def arguments(pid): + mib=(ctypes.c_int*3)(1,49,pid);size=ctypes.c_size_t() + if kernel.sysctl(mib,3,None,ctypes.byref(size),None,0)!=0 or not 8<=size.value<=2097152:raise RuntimeError('argv_size') + buf=ctypes.create_string_buffer(size.value) + if kernel.sysctl(mib,3,buf,ctypes.byref(size),None,0)!=0:raise RuntimeError('argv_read') + return decode_arguments(buf.raw[:size.value]) +def command(args,maximum=131072): + r=subprocess.run(args,env={'PATH':'/usr/bin:/bin','LC_ALL':'C'},stdin=subprocess.DEVNULL,stdout=subprocess.PIPE,stderr=subprocess.DEVNULL,timeout=2,check=True) + if len(r.stdout)>maximum:raise RuntimeError('bound') + return r.stdout.decode('utf8').strip() +try: + pids=json.loads(sys.argv[1]) + if not isinstance(pids,list) or not 0 = process): boolean { + return host.platform === "darwin" && host.arch === "arm64"; +} + +export function readKimiLocalProcesses(pids: readonly number[]): KimiLocalProcess[] { + demand(supportsKimiLocalProviderFault(), "supported_platform"); + let output: string; + try { + output = execFileSync("/usr/bin/python3", ["-I", "-B", "-c", KIMI_MAC_PROCESS_OBSERVER, JSON.stringify(pids)], { + env: { PATH: "/usr/bin:/bin", LC_ALL: "C" }, timeout: 15_000, maxBuffer: 2 * 1024 * 1024, stdio: ["ignore", "pipe", "ignore"], encoding: "utf8", + }); + } catch { throw new Error("kimi_local_provider_fault:process_inspection_failed"); } + const rows: unknown = JSON.parse(output); + demand(Array.isArray(rows) && rows.length <= 128 && rows.every(row => row && Number.isSafeInteger(row.pid) && row.pid > 1 + && Number.isSafeInteger(row.ppid) && row.ppid > 0 && Number.isSafeInteger(row.pgid) && row.pgid > 1 + && /^[1-9][0-9]{0,19}$/u.test(row.startUs) && /^[a-f0-9-]{36}$/u.test(row.bootId) + && /^\d+$/u.test(row.executableDev) && /^\d+$/u.test(row.executableIno) + && [row.executable, row.command, row.cwd].every(value => typeof value === "string" && value.length > 0 && value.length <= 131072) + && Array.isArray(row.argv) && row.argv.length > 0 && row.argv.length <= 128 && row.argv.every((value: unknown) => typeof value === "string" && !value.includes("\0") && Buffer.byteLength(value) <= 131072)), "process_shape"); + return rows; +} + +export function selectKimiLocalTarget(input: { + authority: RunProcessAuthority; journal: readonly JournalIdentity[]; processes: readonly KimiLocalProcess[]; + workspacePath: string; guardLoaderSha256: string; +}) { + const byPid = new Map(input.processes.map(row => [row.pid, row])), root = byPid.get(input.authority.pid); + const retainedRoot = input.journal.find(row => row.pid === input.authority.pid); + demand(root && retainedRoot && input.processes.length <= 128 && new Set(input.processes.map(row => row.pid)).size === input.processes.length + && isPerTurnRunProcess(input.authority, retainedRoot, root.command) && root.pgid === input.authority.groupId + && Math.floor(Number(root.startUs) / 1_000_000) === Math.floor(Date.parse(input.authority.startedAt) / 1000), "original_run_root"); + const matches = []; + for (const candidate of input.processes) { + if (candidate.pid === root.pid || candidate.cwd !== input.workspacePath || candidate.bootId !== root.bootId) continue; + const match = /^(\/[^\u0000-\u0020]*\/paperclip-acpx-native-[A-Za-z0-9_-]+\/distribution)\/node\/bin\/node$/u.exec(candidate.executable); + if (!match) continue; + const distribution = match[1]!, parent = byPid.get(candidate.ppid); + if (!parent || parent.cwd !== input.workspacePath || parent.bootId !== root.bootId) continue; + if (!Array.isArray(parent.argv) || parent.argv.length !== 10 || parent.argv[1] !== "--eval" || parent.argv[3] !== distribution + || JSON.stringify(parent.argv.slice(4)) !== JSON.stringify([".paperclip-native-entry.cjs", "0", "0", "commonjs", "[]", "0"]) + || sha(parent.argv[2]!) !== input.guardLoaderSha256) continue; + const ancestry = [candidate]; let cursor = candidate; + while (cursor.pid !== root.pid) { + const next = byPid.get(cursor.ppid); + demand(next && ancestry.length < 64 && !ancestry.some(row => row.pid === next.pid) + && next.bootId === root.bootId && input.journal.some(row => row.pid === next.pid && row.parent === next.ppid + && Math.floor(Date.parse(row.start) / 1000) === Math.floor(Number(next.startUs) / 1_000_000)), "owned_ancestry"); + ancestry.push(next); cursor = next; + } + demand(input.journal.some(row => row.pid === candidate.pid && row.parent === candidate.ppid + && Math.floor(Date.parse(row.start) / 1000) === Math.floor(Number(candidate.startUs) / 1_000_000)), "unobserved_native_child"); + matches.push({ target: candidate, ancestry, distribution }); + } + demand(matches.length === 1, "unique_native_child"); + return matches[0]!; +} + +function sealedFile(path: string, maximum: number) { + demand(realpathSync(path) === path, "sealed_realpath"); + const fd = openSync(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + try { + const before = fstatSync(fd, { bigint: true }); + demand(before.isFile() && before.nlink === 1n && before.size > 0n && before.size <= BigInt(maximum) && (before.mode & 0o222n) === 0n, "sealed_file"); + const bytes = readFileSync(fd), after = fstatSync(fd, { bigint: true }), named = lstatSync(path, { bigint: true }); + demand(BigInt(bytes.length) === before.size && [after, named].every(value => value.dev === before.dev && value.ino === before.ino + && value.size === before.size && value.mtimeNs === before.mtimeNs && value.ctimeNs === before.ctimeNs && value.mode === before.mode && value.nlink === before.nlink), "sealed_file_changed"); + return { bytes, digest: sha(bytes), dev: String(before.dev), ino: String(before.ino) }; + } finally { closeSync(fd); } +} + +export function terminateKimiLocalProvider(input: { authority: RunProcessAuthority; journal: readonly JournalIdentity[]; workspacePath: string }) { + demand(supportsKimiLocalProviderFault(), "supported_platform"); + const contract = kimiFaultRuntimeContract("darwin-arm64"); + const pids = input.journal.map(row => row.pid); + const before = selectKimiLocalTarget({ ...input, processes: readKimiLocalProcesses(pids), guardLoaderSha256: contract.guardLoaderSha256 }); + demand(before.target.pid !== process.pid && before.target.pid !== input.authority.pid, "not_runner"); + for (const path of [before.distribution, dirname(before.distribution)]) { + const stat = lstatSync(path); + demand(stat.isDirectory() && !stat.isSymbolicLink() && (stat.mode & 0o222) === 0 && realpathSync(path) === path, "snapshot_seal"); + } + const metadata = JSON.parse(readFileSync(new URL("../../packages/paperclip-runner/provider-assets/kimi/darwin-arm64/native-closure.json", import.meta.url), "utf8")); + const entries = parseNativeAcpxDistributionEntries(metadata, contract.closureSha256); + const node = sealedFile(join(before.distribution, "node/bin/node"), 256 * 1024 * 1024); + const entry = sealedFile(join(before.distribution, KIMI_FAULT_ENTRY), 256 * 1024 * 1024); + const guard = sealedFile(join(before.distribution, ".paperclip-native-module-guard.cjs"), 4 * 1024 * 1024); + const bootstrap = sealedFile(join(before.distribution, ".paperclip-native-entry.cjs"), 65536); + demand(node.dev === before.target.executableDev && node.ino === before.target.executableIno + && node.digest === contract.nodeSha256 && node.digest === `sha256:${entries.find(row => row.path === "node/bin/node")?.sha256}` + && entry.digest === `sha256:${entries.find(row => row.path === KIMI_FAULT_ENTRY)?.sha256}` + && guard.bytes.toString("utf8") === expectedNativeModuleGuard(before.distribution, entries) + && bootstrap.bytes.toString("utf8") === expectedNativeBootstrap(before.distribution, contract.nodeSha256), "pinned_native_bytes"); + const fresh = selectKimiLocalTarget({ ...input, processes: readKimiLocalProcesses(pids), guardLoaderSha256: contract.guardLoaderSha256 }); + demand(JSON.stringify(fresh) === JSON.stringify(before), "native_identity_changed"); + process.kill(fresh.target.pid, "SIGKILL"); + const identity = ({ pid, ppid, pgid, startUs, bootId }: KimiLocalProcess) => ({ pid, ppid, pgid, startUs, bootId }); + return { schema: "paperclip.e2e.kimi-local-provider-death.v1", runId: input.authority.runId, target: identity(before.target), ancestry: before.ancestry.map(identity), + nodeSha256: node.digest, entrypointSha256: entry.digest, closureSha256: contract.closureSha256, + guardLoaderSha256: contract.guardLoaderSha256, guardSha256: guard.digest, bootstrapSha256: bootstrap.digest, + entrypointAttribution: "pinned_native_bootstrap_parent", signalled: true, signal: "SIGKILL", targetKind: "kimi_native_child", workerSignalled: false }; +} diff --git a/tests/runner-e2e/kimi-permission-card.spec.ts b/tests/runner-e2e/kimi-permission-card.spec.ts new file mode 100644 index 00000000000..326d302f1da --- /dev/null +++ b/tests/runner-e2e/kimi-permission-card.spec.ts @@ -0,0 +1,64 @@ +import { expect, test } from "@playwright/test"; +import { kimiPendingWritePermissionCard } from "./kimi-permission-card.js"; +import type { KimiLifecyclePending } from "./kimi-lifecycle-evidence.js"; + +const pending: KimiLifecyclePending = { + schema: "paperclip.e2e.kimi-native-write-pending.v1", + scope: { companyId: "synthetic-company", issueId: "synthetic-issue", runId: "synthetic-run", target: "denied/expected.txt" }, + requestId: "native-write-request", toolCallId: "0:native-write", executionId: "0:native-write", itemId: "native-item", turnId: "native-turn", + normalizedSessionId: "native-session", sourceInstanceId: "native-runner", requestSourceSeq: 9, startedSourceSeq: 7, + requestRowSha256: `sha256:${"a".repeat(64)}`, startedRowSha256: `sha256:${"b".repeat(64)}`, observedMonotonicNs: "1", +}; +const card = (prompt: string, status: string, controls = "") => `
Runtime permission${status}

${prompt}

${controls}
`; +const decisions = ''; +const history = card("mcp__paperclip__get_task_context", "resolved"); +const write = () => card(`Change file: ${pending.scope.target}`, "pending", decisions); + +test("selects the exact pending Write while resolved context remains visible, then follows the same terminal card", async ({ page }) => { + await page.setContent(history + write()); + await expect(page.getByTestId("task-chat-runtime-request")).toHaveCount(2); + const selected = await kimiPendingWritePermissionCard({ page, pending, readPending: async () => structuredClone(pending) }); + await expect(selected).toHaveCount(1); + await expect(page.getByText("mcp__paperclip__get_task_context", { exact: true })).toBeVisible(); + for (const status of ["cancelled", "expired"]) { + await page.setContent(history + card(`Change file: ${pending.scope.target}`, status)); + await expect(selected).toHaveCount(1); + await expect(selected.getByText(status, { exact: true })).toBeVisible(); + await expect(selected.getByRole("button")).toHaveCount(0); + await expect(page.getByText("mcp__paperclip__get_task_context", { exact: true })).toBeVisible(); + } +}); + +test("rejects a foreign target and never borrows its decisions for a closed matching card", async ({ page }) => { + await page.setContent(history + card(`Change file: ${pending.scope.target}`, "resolved") + card("Change file: foreign.txt", "pending", decisions)); + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => structuredClone(pending) })).rejects.toThrow(); +}); + +test("rejects a target that only contains the expected filename", async ({ page }) => { + await page.setContent(history + card(`Change file: ${pending.scope.target}.foreign`, "pending", decisions)); + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => structuredClone(pending) })).rejects.toThrow(); +}); + +test("rejects duplicate matching pending cards", async ({ page }) => { + await page.setContent(history + write() + write()); + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => structuredClone(pending) })).rejects.toThrow(); +}); + +for (const key of ["requestId", "toolCallId", "turnId", "sourceInstanceId"] as const) { + test(`rejects a changed native ${key} before locating the card`, async ({ page }) => { + await page.setContent(history + write()); + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => ({ ...pending, [key]: "foreign" }) })).rejects.toThrow("pending callback changed"); + }); +} + +test("rechecks the native request after the UI lookup", async ({ page }) => { + await page.setContent(history + write()); + let reads = 0; + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => ++reads === 1 ? structuredClone(pending) : { ...pending, requestId: "replacement" } })).rejects.toThrow("pending callback changed"); + expect(reads).toBe(2); +}); + +test("rejects disabled pending decisions", async ({ page }) => { + await page.setContent(history + card(`Change file: ${pending.scope.target}`, "pending", '')); + await expect(kimiPendingWritePermissionCard({ page, pending, readPending: async () => structuredClone(pending) })).rejects.toThrow(); +}); diff --git a/tests/runner-e2e/kimi-permission-card.ts b/tests/runner-e2e/kimi-permission-card.ts new file mode 100644 index 00000000000..f6b66dfdc64 --- /dev/null +++ b/tests/runner-e2e/kimi-permission-card.ts @@ -0,0 +1,21 @@ +import { expect, type Page } from "@playwright/test"; +import { assertSameKimiPending, type KimiLifecyclePending } from "./kimi-lifecycle-evidence.js"; + +export async function kimiPendingWritePermissionCard(input: { + page: Page; + pending: KimiLifecyclePending; + readPending(): Promise; +}) { + const { page, pending, readPending } = input; + assertSameKimiPending(pending, await readPending()); + const prompt = `Change file: ${pending.scope.target}`; + const card = page.getByTestId("task-chat-runtime-request") + .filter({ visible: true }) + .filter({ has: page.getByText(prompt, { exact: true }) }); + await expect(card).toHaveCount(1); + await expect(card.getByText(prompt, { exact: true })).toBeVisible(); + await expect(card.getByRole("button", { name: "Deny", exact: true })).toBeEnabled(); + await expect(card.getByRole("button", { name: "Allow once", exact: true })).toBeEnabled(); + assertSameKimiPending(pending, await readPending()); + return card; +} diff --git a/tests/runner-e2e/kimi-provider-fault-contract.ts b/tests/runner-e2e/kimi-provider-fault-contract.ts new file mode 100644 index 00000000000..76d9855c50a --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault-contract.ts @@ -0,0 +1,88 @@ +import { createHash } from "node:crypto"; +import { constants, closeSync, fstatSync, lstatSync, openSync, readFileSync } from "node:fs"; +import { runInNewContext } from "node:vm"; +import { KIMI_DISTRIBUTION_CLOSURE_SHA256, KIMI_NODE_DISTRIBUTIONS } from "../../packages/paperclip-runner/src/drivers/acpx/kimi-closure-pins.js"; +import type { RemoteNativeBinding, RemoteProcessIdentity } from "./remote-native-fixtures.js"; + +export const KIMI_FAULT_ENTRY = "node_modules/@moonshot-ai/kimi-code/dist/main.mjs"; +export const KIMI_FAULT_SNAPSHOT = "/__paperclip_kimi_fault_snapshot__/distribution"; +const sha = (bytes: string | Buffer) => `sha256:${createHash("sha256").update(bytes).digest("hex")}`; +const requireValue = (value: unknown, code: string): void => { if (!value) throw new Error(`kimi_provider_fault:${code}`); }; + +function compiledBuilders() { + const path = new URL("../../packages/paperclip-runner/dist/cli/acpx-runtime-sidecar.cjs", import.meta.url); + const fd = openSync(path, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK); + let source: string; + try { + const before = fstatSync(fd, { bigint: true }); + requireValue(before.isFile() && before.nlink === 1n && before.size > 0n && before.size <= 16n * 1024n * 1024n, "compiled_sidecar_shape"); + const bytes = readFileSync(fd), after = fstatSync(fd, { bigint: true }), named = lstatSync(path, { bigint: true }); + requireValue(BigInt(bytes.length) === before.size && before.size === after.size && before.mtimeNs === after.mtimeNs + && before.ctimeNs === after.ctimeNs && before.dev === named.dev && before.ino === named.ino && !named.isSymbolicLink(), "compiled_sidecar_changed"); + source = bytes.toString("utf8"); + } finally { closeSync(fd); } + const declarations: string[] = []; + for (const name of ["COMMAND_SOURCE_FD", "COMMAND_DIRECTORY_FD", "DEPENDENCY_ANCESTOR_FD_START", "MAX_DEPENDENCY_ANCESTORS", "MAX_AGENT_COMMAND_BYTES", "VERIFIED_PROVIDER_RUNTIME_TARGET_ENV", "ACPX_PRIVATE_SNAPSHOT_ENV"]) { + const found = [...source.matchAll(new RegExp(`^var ${name} = ([^\\n]+);$`, "gm"))]; + requireValue(found.length === 1 && /^(?:[0-9 *+()]+|"[A-Z0-9_]+")$/u.test(found[0]![1]!), "compiled_constant"); + declarations.push(found[0]![0]); + } + for (const name of ["snapshotBootstrap", "reapCurrentProviderProcessGroup", "guardSnapshotModuleLookup", "guardSnapshotModuleResolution", "snapshotDescriptorAncestorIndex", "snapshotDescriptorResolution", "nativeBootstrap", "nativeModuleGuard"]) { + const found = [...source.matchAll(new RegExp(`^function ${name}\\([^]*?^}`, "gm"))]; + requireValue(found.length === 1, "compiled_function"); + declarations.push(found[0]![0]); + } + return runInNewContext(`${declarations.join("\n")}\n({loader:snapshotBootstrap("commonjs",true),nativeBootstrap,nativeModuleGuard})`, Object.create(null), { timeout: 1000 }) as { + loader: string; nativeBootstrap(executable: string, digest: string, args: string[]): string; + nativeModuleGuard(digests: Record): string; + }; +} + +export function kimiFaultRuntimeContract(target: keyof typeof KIMI_DISTRIBUTION_CLOSURE_SHA256) { + const loader = compiledBuilders().loader; + requireValue(typeof loader === "string" && Buffer.byteLength(loader) > 0 && Buffer.byteLength(loader) <= 65536, "compiled_loader"); + return { guardLoaderSha256: sha(loader), closureSha256: KIMI_DISTRIBUTION_CLOSURE_SHA256[target], + nodeSha256: `sha256:${KIMI_NODE_DISTRIBUTIONS[target].executableSha256}` }; +} + +export function expectedNativeBootstrap(distribution: string, nodeSha256: string): string { + const digest = nodeSha256.replace(/^sha256:/u, ""); + requireValue(/^[a-f0-9]{64}$/u.test(digest), "node_digest"); + return compiledBuilders().nativeBootstrap(`${distribution}/node/bin/node`, digest, + ["--require", `${distribution}/.paperclip-native-module-guard.cjs`, `${distribution}/${KIMI_FAULT_ENTRY}`, "acp"]); +} + +export function expectedNativeModuleGuard(distribution: string, entries: readonly { path: string; sha256: string }[]): string { + return compiledBuilders().nativeModuleGuard(Object.fromEntries(entries.map(entry => [`${distribution}/${entry.path}`, entry.sha256]))); +} + +export interface KimiProviderDeathReceipt { + schema: "paperclip.e2e.kimi-provider-death.v1"; binding: RemoteNativeBinding; runtimeEnvironmentLeaseId: string; + target: RemoteProcessIdentity; ancestry: RemoteProcessIdentity[]; nodeSha256: string; entrypointSha256: string; closureSha256: string; + guardLoaderSha256: string; guardSha256: string; bootstrapSha256: string; + entrypointAttribution: "pinned_native_bootstrap_parent"; originalChildArgvAvailable: false; observedChildTitle: "kimi-code"; + signalled: true; signal: "SIGKILL"; targetKind: "kimi_native_child"; workerSignalled: false; +} + +export function validateKimiProviderDeathReceipt(value: unknown, binding: RemoteNativeBinding, root: RemoteProcessIdentity, lease: string): KimiProviderDeathReceipt { + requireValue(value && typeof value === "object" && !Array.isArray(value), "receipt_shape"); + const row = value as Record, target = row.target as RemoteProcessIdentity; + const contract = kimiFaultRuntimeContract("linux-x64"); + const keys = ["schema", "binding", "runtimeEnvironmentLeaseId", "target", "ancestry", "nodeSha256", "entrypointSha256", "closureSha256", "guardLoaderSha256", "guardSha256", "bootstrapSha256", "entrypointAttribution", "originalChildArgvAvailable", "observedChildTitle", "signalled", "signal", "targetKind", "workerSignalled"]; + requireValue(Object.keys(row).length === keys.length && Object.keys(row).every(key => keys.includes(key)) + && row.schema === "paperclip.e2e.kimi-provider-death.v1" && JSON.stringify(row.binding) === JSON.stringify(binding) + && row.runtimeEnvironmentLeaseId === lease && row.nodeSha256 === contract.nodeSha256 && row.closureSha256 === contract.closureSha256 + && row.guardLoaderSha256 === contract.guardLoaderSha256 && [row.entrypointSha256, row.guardSha256, row.bootstrapSha256].every(value => typeof value === "string" && /^sha256:[a-f0-9]{64}$/u.test(value)) + && row.entrypointAttribution === "pinned_native_bootstrap_parent" && row.originalChildArgvAvailable === false && row.observedChildTitle === "kimi-code" + && row.signalled === true && row.signal === "SIGKILL" && row.targetKind === "kimi_native_child" && row.workerSignalled === false, "receipt_identity"); + const ancestry = row.ancestry as RemoteProcessIdentity[]; + requireValue(target && Array.isArray(ancestry) && ancestry.length >= 3 && ancestry.length <= 64 + && new Set(ancestry.map(process => process?.pid)).size === ancestry.length + && ancestry.every(process => process && Object.keys(process).sort().join(",") === "bootId,pid,ppid,startTicks" + && Number.isSafeInteger(process.pid) && process.pid > 1 && Number.isSafeInteger(process.ppid) && process.ppid > 0 + && typeof process.startTicks === "string" && /^[0-9]+$/u.test(process.startTicks) && process.bootId === root.bootId) + && JSON.stringify(ancestry[0]) === JSON.stringify(target) + && ["pid", "ppid", "startTicks", "bootId"].every(key => (ancestry.at(-1) as unknown as Record)?.[key] === (root as unknown as Record)[key]) + && ancestry.slice(0, -1).every((process, index) => process.ppid === ancestry[index + 1]!.pid) && target.pid !== root.pid, "receipt_ancestry"); + return value as KimiProviderDeathReceipt; +} diff --git a/tests/runner-e2e/kimi-provider-fault-diagnostic.py b/tests/runner-e2e/kimi-provider-fault-diagnostic.py new file mode 100644 index 00000000000..48825651c5a --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault-diagnostic.py @@ -0,0 +1,53 @@ +import base64,hashlib,importlib.util,json,os,re,sys +from pathlib import Path +sys.dont_write_bytecode=True +EXPECTED_SOURCE_SHA256='585494ff2f10125c98c4102df5b6b6dba643de6c92266186711695b1e993efe6' + +def run(fault,config,allow_signal): + state={'schema':'paperclip.kimi11.remote-fault-diagnostic.v1','qualification':False,'stage':'inspect','status':'FAIL','failingGuard':None,'errno':None,'signalAttempted':False,'signalDispatched':False,'receipt':None,'externalModelCalls':0} + original_signal=getattr(fault.signal,'pidfd_send_signal',None);original_inspect=fault.inspect;original_open=getattr(fault.os,'pidfd_open',None);inspections=0 + def inspect(config): + nonlocal inspections + inspections+=1;state['stage']='inspect_before' if inspections==1 else 'inspect_held' + result=original_inspect(config) + if inspections>1:state['stage']='pidfd_identity' + return result + def pidfd_open(pid): + state['stage']='pidfd_open' + return original_open(pid) + def signal(fd,sig,info=None,flags=0): + state['stage']='signal';state['signalAttempted']=True;state['signalDispatched']=None + result=original_signal(fd,sig,info,flags);state['signalDispatched']=True;state['stage']='receipt';return result + fault.inspect=inspect + if original_signal is not None:fault.signal.pidfd_send_signal=signal + if original_open is not None:fault.os.pidfd_open=pidfd_open + try: + if allow_signal: + state['stage']='terminate';state['receipt']=fault.terminate(config) + else:state['receipt']=fault.inspect(config) + state['status']='PASS';state['stage']='complete' + except OSError as e:state['errno']=e.errno;state['failingGuard']=None + except RuntimeError as e: + value=e.args[0] if len(e.args)==1 and isinstance(e.args[0],str) else '' + code=value.removeprefix('kimi_provider_fault:') if value.startswith('kimi_provider_fault:') else '' + state['failingGuard']=code if code in ALLOWED_GUARDS else 'unknown_guard' + except Exception:pass + finally: + fault.inspect=original_inspect + if original_signal is not None:fault.signal.pidfd_send_signal=original_signal + if original_open is not None:fault.os.pidfd_open=original_open + return state + +ALLOWED_GUARDS=frozenset(['ancestry', 'ancestry_birth', 'ancestry_changed', 'argv_bound', 'boot_changed', 'bootstrap_template', 'closure_bound', 'closure_inventory', 'closure_metadata', 'closure_path', 'closure_pin', 'closure_shape', 'config_keys', 'cwd', 'dead_or_invalid_process', 'digest_shape', 'executable', 'file_bound', 'file_changed', 'file_realpath', 'file_shape', 'file_writable', 'guard_template', 'identity_changed_after_pidfd', 'input_bound', 'native_bootstrap_bytes', 'native_executable', 'native_guard_bytes', 'native_title', 'not_worker', 'parent_bootstrap', 'parent_descriptor_changed', 'parent_descriptor_shape', 'parent_directory_descriptor', 'parent_memfd_required', 'parent_memfd_seals', 'parent_node', 'parent_node_descriptor', 'parent_node_path', 'parent_source_pipe', 'pidfd_open', 'pidfd_required', 'pidfd_retired', 'pidfd_send_signal', 'process_bound', 'required_entries', 'root_binding', 'root_changed', 'root_flag', 'root_session', 'runner_changed', 'runner_executable', 'runner_shape', 'snapshot_digest', 'snapshot_path', 'snapshot_seal', 'stat_pid', 'unique_native_child']) + +if __name__=='__main__': + from types import ModuleType + result={'schema':'paperclip.kimi11.remote-fault-diagnostic.v1','qualification':False,'stage':'input','status':'FAIL','failingGuard':None,'errno':None,'signalAttempted':False,'signalDispatched':False,'receipt':None,'externalModelCalls':0} + try: + if len(sys.argv)!=3 or len(sys.argv[1])>131072 or len(sys.argv[2])>131072:raise ValueError() + source=base64.b64decode(sys.argv[1],validate=True) + if hashlib.sha256(source).hexdigest()!=EXPECTED_SOURCE_SHA256:raise ValueError() + fault=ModuleType('kimi_fault_diagnostic_original');exec(compile(source,'','exec'),fault.__dict__) + config=json.loads(base64.b64decode(sys.argv[2],validate=True));result=run(fault,config,True) + except Exception:pass + print(json.dumps(result,separators=(',',':')));sys.exit(0 if result['status']=='PASS' else 1) diff --git a/tests/runner-e2e/kimi-provider-fault-diagnostic.test.py b/tests/runner-e2e/kimi-provider-fault-diagnostic.test.py new file mode 100644 index 00000000000..164bd5ae4fb --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault-diagnostic.test.py @@ -0,0 +1,123 @@ +import importlib.util +import base64 +import json +from pathlib import Path +import signal +import subprocess +from types import SimpleNamespace +import sys +import unittest +from unittest.mock import Mock, patch + +def load(name, file): + spec = importlib.util.spec_from_file_location(name, Path(__file__).with_name(file)) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + +fault = load('original_fault', 'kimi-provider-fault.py') +diagnostic = load('diagnostic_fault', 'kimi-provider-fault-diagnostic.py') + +class DiagnosticTests(unittest.TestCase): + def test_exact_selector_source_hash_admits_only_the_frozen_source(self): + worker = Path(__file__).with_name('kimi-provider-fault-diagnostic.py') + source = Path(__file__).with_name('kimi-provider-fault.py').read_bytes() + config = base64.b64encode(b'{}').decode() + response = subprocess.run([sys.executable, '-I', str(worker), base64.b64encode(source).decode(), config], capture_output=True, text=True, timeout=5) + self.assertEqual(response.returncode, 1) + result = json.loads(response.stdout) + self.assertNotEqual(result['stage'], 'input') + self.assertIn(result['failingGuard'], ['pidfd_required', 'config_keys']) + self.assertIs(result['signalDispatched'], False) + rejected = subprocess.run([sys.executable, '-I', str(worker), base64.b64encode(source + b'\n').decode(), config], capture_output=True, text=True, timeout=5) + self.assertEqual(rejected.returncode, 1) + refused = json.loads(rejected.stdout) + self.assertEqual(refused['stage'], 'input') + self.assertIs(refused['signalDispatched'], False) + + def test_missing_signal_capability_is_preserved_and_fails_original_guard(self): + missing = SimpleNamespace(SIGKILL=signal.SIGKILL) + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'signal', missing), patch.object(fault, 'os', SimpleNamespace(pidfd_open=Mock())): + result = diagnostic.run(fault, {}, True) + self.assertFalse(hasattr(missing, 'pidfd_send_signal')) + self.assertEqual(result['failingGuard'], 'pidfd_required') + self.assertEqual(result['stage'], 'terminate') + self.assertIs(result['signalDispatched'], False) + + def test_malformed_worker_input_has_one_closed_full_schema(self): + worker = Path(__file__).with_name('kimi-provider-fault-diagnostic.py') + response = subprocess.run([sys.executable, '-I', str(worker), 'private_invalid_source', 'private_invalid_config'], capture_output=True, text=True, timeout=5) + self.assertEqual(response.returncode, 1) + result = json.loads(response.stdout) + self.assertEqual(set(result), {'schema', 'qualification', 'stage', 'status', 'failingGuard', 'errno', 'signalAttempted', 'signalDispatched', 'receipt', 'externalModelCalls'}) + self.assertEqual(result['stage'], 'input') + self.assertIs(result['signalDispatched'], False) + self.assertNotIn('private', response.stdout) + + def test_original_require_remains_untouched_and_terminal_guard_is_closed(self): + original = fault.require + with patch.object(sys, 'platform', 'linux'), patch.object(fault.os, 'pidfd_open', create=True) as opened, patch.object(signal, 'pidfd_send_signal', create=True) as sent: + result = diagnostic.run(fault, {}, True) + self.assertEqual(result['failingGuard'], 'config_keys') + self.assertEqual(result['stage'], 'inspect_before') + self.assertIs(result['signalDispatched'], False) + self.assertIs(fault.require, original) + opened.assert_not_called() + sent.assert_not_called() + + def test_actual_original_terminate_keeps_two_inspections_and_one_signal(self): + before = {'target': {'pid': 23}} + original = fault.require + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', Mock(return_value=before)) as inspected, patch.object(fault.os, 'pidfd_open', Mock(return_value=42), create=True) as opened, patch.object(signal, 'pidfd_send_signal', Mock(), create=True) as sent, patch.object(fault.os, 'close') as closed, patch.object(Path, 'read_text', return_value='Pid:\t23\n'): + result = diagnostic.run(fault, {'root': {'pid': 21}, 'binding': {}, 'runtimeEnvironmentLeaseId': 'lease'}, True) + self.assertEqual(result['status'], 'PASS') + self.assertIs(result['signalDispatched'], True) + self.assertEqual(inspected.call_count, 2) + opened.assert_called_once_with(23) + sent.assert_called_once_with(42, signal.SIGKILL, None, 0) + closed.assert_called_once_with(42) + self.assertIs(fault.require, original) + + def test_failed_signal_call_remains_unknown_and_restores_wrappers(self): + before = {'target': {'pid': 23}} + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', Mock(return_value=before)) as inspected, patch.object(fault.os, 'pidfd_open', Mock(return_value=42), create=True) as opened, patch.object(signal, 'pidfd_send_signal', Mock(side_effect=OSError(38, 'private kernel details')), create=True) as sent, patch.object(fault.os, 'close'), patch.object(Path, 'read_text', return_value='Pid:\t23\n'): + result = diagnostic.run(fault, {'root': {'pid': 21}, 'binding': {}, 'runtimeEnvironmentLeaseId': 'lease'}, True) + self.assertIs(fault.inspect, inspected) + self.assertIs(fault.os.pidfd_open, opened) + self.assertIs(fault.signal.pidfd_send_signal, sent) + self.assertEqual(result['stage'], 'signal') + self.assertIsNone(result['signalDispatched']) + self.assertNotIn('private', json.dumps(result)) + + def test_caught_unrelated_ancestry_rejection_does_not_become_a_failure(self): + def inspect(config): + try: + fault.require(False, 'ancestry') + except RuntimeError: + pass + return {'target': {'pid': 23}} + with patch.object(fault, 'inspect', inspect), patch.object(signal, 'pidfd_send_signal', Mock(), create=True) as sent: + result = diagnostic.run(fault, {}, False) + self.assertEqual(result['status'], 'PASS') + self.assertIsNone(result['failingGuard']) + self.assertIs(result['signalDispatched'], False) + sent.assert_not_called() + + def test_caught_guard_cannot_mask_a_later_os_error(self): + def inspect(config): + try: + fault.require(False, 'ancestry') + except RuntimeError: + pass + raise OSError(13, 'private path') + with patch.object(fault, 'inspect', inspect), patch.object(signal, 'pidfd_send_signal', Mock(), create=True) as sent: + result = diagnostic.run(fault, {}, False) + self.assertEqual(result['status'], 'FAIL') + self.assertIsNone(result['failingGuard']) + self.assertEqual(result['stage'], 'inspect_before') + self.assertIs(result['signalDispatched'], False) + self.assertNotIn('private', json.dumps(result)) + sent.assert_not_called() + +if __name__ == '__main__': + unittest.main() diff --git a/tests/runner-e2e/kimi-provider-fault.py b/tests/runner-e2e/kimi-provider-fault.py new file mode 100644 index 00000000000..90a813cbc6a --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault.py @@ -0,0 +1,270 @@ +import base64 +import hashlib +import fcntl +import json +import os +from pathlib import Path +import re +import signal +import stat +import sys + +PACK = '/opt/paperclip-runner/provider-pack' +ENTRY = 'node_modules/@moonshot-ai/kimi-code/dist/main.mjs' +NODE = 'node/bin/node' +GUARD = '.paperclip-native-module-guard.cjs' +BOOTSTRAP = '.paperclip-native-entry.cjs' +PLACEHOLDER = '/__paperclip_kimi_fault_snapshot__/distribution' + + +def require(value, code): + if not value: + raise RuntimeError('kimi_provider_fault:' + code) + + +def digest(data): + return hashlib.sha256(data).hexdigest() + + +def proc(pid, boot): + raw = Path(f'/proc/{pid}/stat').read_text() + require(raw.startswith(f'{pid} ('), 'stat_pid') + fields = raw.rsplit(') ', 1)[1].split() + require(len(fields) >= 20 and fields[0] != 'Z' and fields[19].isdigit(), 'dead_or_invalid_process') + return {'pid': pid, 'ppid': int(fields[1]), 'startTicks': fields[19], 'bootId': boot} + + +def argv(pid): + raw = Path(f'/proc/{pid}/cmdline').read_bytes() + require(0 < len(raw) <= 65536, 'argv_bound') + return raw.decode('utf8').rstrip('\0').split('\0') + + +def flag(args, name): + require(args.count(name) == 1 and args.index(name) + 1 < len(args), 'root_flag') + return args[args.index(name) + 1] + + +def checked_file(path, maximum, sealed=False): + require(os.path.realpath(path) == path, 'file_realpath') + fd = os.open(path, os.O_RDONLY | os.O_NOFOLLOW | os.O_NONBLOCK) + try: + before = os.fstat(fd) + require(stat.S_ISREG(before.st_mode) and before.st_nlink == 1 and 0 < before.st_size <= maximum, 'file_shape') + require(not sealed or before.st_mode & 0o222 == 0, 'file_writable') + chunks, size = [], 0 + while True: + chunk = os.read(fd, 1024 * 1024) + if not chunk: + break + chunks.append(chunk) + size += len(chunk) + require(size <= maximum, 'file_bound') + after, named = os.fstat(fd), os.lstat(path) + identity = lambda value: (value.st_dev, value.st_ino, value.st_size, value.st_mtime_ns, value.st_ctime_ns, value.st_mode, value.st_nlink) + require(size == before.st_size and identity(before) == identity(after) == identity(named), 'file_changed') + return b''.join(chunks), (before.st_dev, before.st_ino) + finally: + os.close(fd) + + +def checked_parent_descriptor(pid): + path = f'/proc/{pid}/fd/10' + executable = f'/proc/{pid}/exe' + fd = os.open(path, os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC) + identity = lambda value: (value.st_dev, value.st_ino, value.st_size, value.st_mtime_ns, value.st_ctime_ns, value.st_mode, value.st_nlink) + try: + before = os.fstat(fd) + require(stat.S_ISREG(before.st_mode) and before.st_nlink in (0, 1) and 0 < before.st_size <= 256 * 1024 * 1024, 'parent_descriptor_shape') + require(identity(before) == identity(os.stat(path)) == identity(os.stat(executable)), 'parent_node_descriptor') + seals = None + if before.st_nlink == 0: + require(all(hasattr(fcntl, name) for name in ('F_GET_SEALS', 'F_SEAL_WRITE', 'F_SEAL_GROW', 'F_SEAL_SHRINK', 'F_SEAL_SEAL')), 'parent_memfd_required') + require(os.readlink(f'/proc/self/fd/{fd}') == '/memfd:paperclip-verified-launch (deleted)' and before.st_mode & 0o222 == 0, 'parent_memfd_required') + required = fcntl.F_SEAL_WRITE | fcntl.F_SEAL_GROW | fcntl.F_SEAL_SHRINK | fcntl.F_SEAL_SEAL + seals = fcntl.fcntl(fd, fcntl.F_GET_SEALS) + require(type(seals) is int and seals & required == required, 'parent_memfd_seals') + chunks, size = [], 0 + while True: + chunk = os.read(fd, 1024 * 1024) + if not chunk: + break + chunks.append(chunk) + size += len(chunk) + require(size <= 256 * 1024 * 1024, 'parent_descriptor_shape') + require(size == before.st_size and identity(before) == identity(os.fstat(fd)) == identity(os.stat(path)) == identity(os.stat(executable)), 'parent_descriptor_changed') + if seals is not None: + require(fcntl.fcntl(fd, fcntl.F_GET_SEALS) == seals, 'parent_memfd_seals') + return b''.join(chunks), (before.st_dev, before.st_ino) + finally: + os.close(fd) + + +def checked_runner(path): + before = os.lstat(path) + require(stat.S_ISREG(before.st_mode) or stat.S_ISLNK(before.st_mode), 'runner_shape') + resolved = os.path.realpath(path) + content, inode = checked_file(resolved, 256 * 1024 * 1024) + after = os.lstat(path) + identity = lambda value: (value.st_dev, value.st_ino, value.st_mode, value.st_mtime_ns, value.st_ctime_ns) + require(identity(before) == identity(after) and os.path.realpath(path) == resolved, 'runner_changed') + return content, inode + + +def parse_closure(content, expected): + manifest = json.loads(content) + entries = manifest.get('entries') if isinstance(manifest, dict) else None + require(isinstance(entries, list) and 0 < len(entries) <= 30000, 'closure_inventory') + normalized, previous, total = [], '', 0 + for entry in entries: + require(isinstance(entry, dict) and set(entry) == {'path', 'sha256', 'size', 'executable'}, 'closure_shape') + path = entry['path'] + require(isinstance(path, str) and 0 < len(path) <= 4096 and not os.path.isabs(path) + and not re.search(r'[\x00-\x1f\x7f\\]', path) and path > previous + and all(part not in ('', '.', '..') for part in path.split('/')) + and path != 'manifest.json' and not path.startswith('.paperclip-'), 'closure_path') + require(isinstance(entry['sha256'], str) and re.fullmatch(r'[a-f0-9]{64}', entry['sha256']) + and type(entry['size']) is int and 0 <= entry['size'] <= 384 * 1024 * 1024 + and type(entry['executable']) is bool, 'closure_metadata') + total += entry['size'] + require(total <= 1024 * 1024 * 1024, 'closure_bound') + previous = path + normalized.append({key: entry[key] for key in ('path', 'sha256', 'size', 'executable')}) + require(re.fullmatch(r'[a-f0-9]{64}', expected) and digest(json.dumps(normalized, separators=(',', ':'), ensure_ascii=False).encode()) == expected, 'closure_pin') + return normalized + + +def expected_sources(config, distribution, entries): + bootstrap, guard = config['bootstrapTemplate'], config['guardTemplate'] + require(isinstance(bootstrap, str) and 0 < len(bootstrap) <= 32768 and PLACEHOLDER in bootstrap, 'bootstrap_template') + require(isinstance(guard, str) and 0 < len(guard) <= 32768 and guard.count('const digests={};') == 1, 'guard_template') + digests = {distribution + '/' + entry['path']: entry['sha256'] for entry in entries} + return bootstrap.replace(PLACEHOLDER, distribution).encode(), guard.replace('const digests={};', 'const digests=' + json.dumps(digests, separators=(',', ':'), ensure_ascii=False) + ';').encode() + + +def inspect(config): + require(set(config) == {'root', 'binding', 'runtimeEnvironmentLeaseId', 'runnerdSha256', 'closureSha256', 'nodeSha256', 'guardLoaderSha256', 'bootstrapTemplate', 'guardTemplate'}, 'config_keys') + require(all(isinstance(config[key], str) and re.fullmatch(r'sha256:[a-f0-9]{64}', config[key]) for key in ('runnerdSha256', 'nodeSha256', 'guardLoaderSha256')), 'digest_shape') + binding, expected = config['binding'], config['root'] + boot = Path('/proc/sys/kernel/random/boot_id').read_text().strip() + require(re.fullmatch(r'[a-f0-9-]{36}', boot) and boot == expected['bootId'], 'boot_changed') + root = proc(expected['pid'], boot) + require(root == expected and os.getpgid(root['pid']) == root['pid'], 'root_changed') + runtime = binding['remoteCwd'] + '/.paperclip-runtime/paperclip-runner' + runner = runtime + '/bin/paperclip-runnerd' + root_args = argv(root['pid']) + require(root_args[0] == runner and flag(root_args, '--run-id') == binding['runId'] + and flag(root_args, '--environment-lease-id') == config['runtimeEnvironmentLeaseId'] + and flag(root_args, '--lifecycle-mode') == 'per_turn', 'root_binding') + require(re.fullmatch(re.escape(runtime) + r'/sessions/[a-f0-9]{64}/runner', flag(root_args, '--state-dir')), 'root_session') + runner_bytes, runner_inode = checked_runner(runner) + root_exe = os.stat(f'/proc/{root["pid"]}/exe') + require((root_exe.st_dev, root_exe.st_ino) == runner_inode and 'sha256:' + digest(runner_bytes) == config['runnerdSha256'], 'runner_executable') + metadata, _ = checked_file(PACK + '/provider-assets/kimi/linux-x64/native-closure.json', 4 * 1024 * 1024) + ordered_entries = parse_closure(metadata, config['closureSha256']) + entries = {entry['path']: entry for entry in ordered_entries} + require(NODE in entries and ENTRY in entries and entries[NODE]['executable'] and entries[NODE]['size'] > 0 + and entries[ENTRY]['size'] > 0 and 'sha256:' + entries[NODE]['sha256'] == config['nodeSha256'], 'required_entries') + pids = [int(pid) for pid in os.listdir('/proc') if pid.isdigit() and int(pid) > 1] + require(len(pids) <= 4096, 'process_bound') + table = {} + for pid in pids: + try: + table[pid] = proc(pid, boot) + except (FileNotFoundError, ProcessLookupError): + continue + except RuntimeError as error: + if str(error).endswith(':dead_or_invalid_process'): + continue + raise + + def ancestry(pid): + chain = [] + while pid != root['pid']: + require(pid in table and len(chain) < 64 and pid not in [item['pid'] for item in chain], 'ancestry') + chain.append(table[pid]) + pid = table[pid]['ppid'] + result = chain + [root] + require(all(int(child['startTicks']) >= int(parent['startTicks']) for child, parent in zip(result, result[1:])), 'ancestry_birth') + return result + + matches = [] + for pid in table: + try: + chain = ancestry(pid) + except RuntimeError: + continue + if len(chain) < 3: + continue + parent = table[pid]['ppid'] + try: + args = argv(parent) + except (FileNotFoundError, ProcessLookupError): + continue + if len(args) != 10 or args[1] != '--eval' or args[4] != BOOTSTRAP: + continue + distribution = args[3] + require(re.fullmatch(r'/tmp/paperclip-acpx-native-[A-Za-z0-9_-]+/distribution', distribution), 'snapshot_path') + require(args[4:] == [BOOTSTRAP, '0', '0', 'commonjs', '[]', '0'] and 'sha256:' + digest(args[2].encode()) == config['guardLoaderSha256'], 'parent_bootstrap') + require(argv(pid) == ['kimi-code'], 'native_title') + for directory in (distribution, os.path.dirname(distribution)): + status = os.lstat(directory) + require(stat.S_ISDIR(status.st_mode) and not status.st_mode & 0o222 and os.path.realpath(directory) == directory, 'snapshot_seal') + require(os.readlink(f'/proc/{pid}/cwd') == binding['remoteCwd'] and os.readlink(f'/proc/{parent}/cwd') == binding['remoteCwd'], 'cwd') + identities = {} + for name in (NODE, ENTRY): + content, inode = checked_file(distribution + '/' + name, 256 * 1024 * 1024, True) + require(len(content) == entries[name]['size'] and digest(content) == entries[name]['sha256'], 'snapshot_digest') + identities[name] = inode + bootstrap, guard = expected_sources(config, distribution, ordered_entries) + require(checked_file(distribution + '/' + BOOTSTRAP, 32768, True)[0] == bootstrap, 'native_bootstrap_bytes') + require(checked_file(distribution + '/' + GUARD, 4 * 1024 * 1024, True)[0] == guard, 'native_guard_bytes') + native_exe, held_node = os.stat(f'/proc/{pid}/exe'), os.stat(f'/proc/{pid}/fd/7') + require((native_exe.st_dev, native_exe.st_ino) == identities[NODE] == (held_node.st_dev, held_node.st_ino), 'native_executable') + if args[0] == '/proc/self/fd/10': + parent_bytes, parent_inode = checked_parent_descriptor(parent) + else: + parent_exe_path = os.path.realpath(f'/proc/{parent}/exe') + parent_bytes, parent_inode = checked_file(parent_exe_path, 256 * 1024 * 1024) + require('sha256:' + digest(parent_bytes) == config['nodeSha256'], 'parent_node') + if args[0] != '/proc/self/fd/10': + require(args[0] == PACK + '/node_modules/node/bin/node' and checked_file(args[0], 256 * 1024 * 1024)[1] == parent_inode, 'parent_node_path') + directory_fd = os.stat(f'/proc/{parent}/fd/4') + directory = os.lstat(distribution) + require(stat.S_ISDIR(directory_fd.st_mode) and (directory_fd.st_dev, directory_fd.st_ino) == (directory.st_dev, directory.st_ino), 'parent_directory_descriptor') + source_mode = os.stat(f'/proc/{parent}/fd/3').st_mode + require(stat.S_ISFIFO(source_mode) or stat.S_ISSOCK(source_mode), 'parent_source_pipe') + for identity in chain: + require(proc(identity['pid'], boot) == identity, 'ancestry_changed') + matches.append({'target': table[pid], 'ancestry': chain, 'nodeSha256': config['nodeSha256'], + 'entrypointSha256': 'sha256:' + entries[ENTRY]['sha256'], 'closureSha256': config['closureSha256'], + 'guardLoaderSha256': config['guardLoaderSha256'], 'guardSha256': 'sha256:' + digest(guard), 'bootstrapSha256': 'sha256:' + digest(bootstrap), + 'entrypointAttribution': 'pinned_native_bootstrap_parent', 'originalChildArgvAvailable': False, 'observedChildTitle': 'kimi-code'}) + require(len(matches) == 1, 'unique_native_child') + return matches[0] + + +def terminate(config): + require(sys.platform == 'linux' and hasattr(os, 'pidfd_open') and hasattr(signal, 'pidfd_send_signal'), 'pidfd_required') + before = inspect(config) + pid = before['target']['pid'] + require(pid not in (os.getpid(), config['root']['pid']), 'not_worker') + fd = os.pidfd_open(pid) + try: + require(inspect(config) == before, 'identity_changed_after_pidfd') + fields = dict(line.split(':', 1) for line in Path(f'/proc/self/fdinfo/{fd}').read_text().splitlines() if ':' in line) + require(int(fields.get('Pid', '-1').strip()) == pid, 'pidfd_retired') + signal.pidfd_send_signal(fd, signal.SIGKILL, None, 0) + return {'schema': 'paperclip.e2e.kimi-provider-death.v1', 'binding': config['binding'], 'runtimeEnvironmentLeaseId': config['runtimeEnvironmentLeaseId'], + **before, 'signalled': True, 'signal': 'SIGKILL', 'targetKind': 'kimi_native_child', 'workerSignalled': False} + finally: + os.close(fd) + + +if __name__ == '__main__': + try: + require(len(sys.argv) == 2 and len(sys.argv[1]) <= 131072, 'input_bound') + print(json.dumps(terminate(json.loads(base64.b64decode(sys.argv[1], validate=True))), separators=(',', ':'))) + except Exception: + print(json.dumps({'ok': False, 'code': 'kimi_provider_identity_or_signal_failed'})) + sys.exit(1) diff --git a/tests/runner-e2e/kimi-provider-fault.test.py b/tests/runner-e2e/kimi-provider-fault.test.py new file mode 100644 index 00000000000..a2c0ec31508 --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault.test.py @@ -0,0 +1,412 @@ +import importlib.util +from contextlib import contextmanager +import json +import os +from pathlib import Path +import select +import shutil +import signal +import stat +import subprocess +import sys +import tempfile +import time +from types import SimpleNamespace +import unittest +from unittest.mock import patch, Mock, call + +spec = importlib.util.spec_from_file_location('kimi_fault', Path(__file__).with_name('kimi-provider-fault.py')) +fault = importlib.util.module_from_spec(spec) +spec.loader.exec_module(fault) +BOOT = '12345678-1234-1234-1234-123456789abc' +ROOT = {'pid': 21, 'ppid': 1, 'startTicks': '100', 'bootId': BOOT} +PARENT = {'pid': 22, 'ppid': 21, 'startTicks': '101', 'bootId': BOOT} +TARGET = {'pid': 23, 'ppid': 22, 'startTicks': '102', 'bootId': BOOT} +DIST = '/tmp/paperclip-acpx-native-fixture/distribution' +RUNTIME = '/workspace/.paperclip-runtime/paperclip-runner' +HOST_NODE = fault.PACK + '/node_modules/node/bin/node' +LOADER = 'admitted guarded loader' +CANCELLED = False + + +def check_cancelled(): + if CANCELLED: + raise KeyboardInterrupt('calibration cancelled') + + +class IdentityTests(unittest.TestCase): + def setUp(self): + self.entries = [{'path': name, 'sha256': fault.digest(name.encode()), 'size': len(name), 'executable': name == fault.NODE} for name in sorted([fault.NODE, fault.ENTRY])] + self.config = {'root': ROOT, 'binding': {'remoteCwd': '/workspace', 'runId': 'run'}, 'runtimeEnvironmentLeaseId': 'workspace-id', + 'runnerdSha256': 'sha256:' + fault.digest(b'runner'), 'nodeSha256': 'sha256:' + fault.digest(fault.NODE.encode()), + 'guardLoaderSha256': 'sha256:' + fault.digest(LOADER.encode()), 'closureSha256': fault.digest(json.dumps(self.entries, separators=(',', ':')).encode()), + 'bootstrapTemplate': 'bootstrap ' + fault.PLACEHOLDER, 'guardTemplate': 'const digests={};'} + bootstrap, guard = fault.expected_sources(self.config, DIST, self.entries) + self.files = {DIST + '/' + fault.NODE: (fault.NODE.encode(), (1, 100)), DIST + '/' + fault.ENTRY: (fault.ENTRY.encode(), (1, 101)), + DIST + '/' + fault.BOOTSTRAP: (bootstrap, (1, 102)), DIST + '/' + fault.GUARD: (guard, (1, 103)), + RUNTIME + '/bin/paperclip-runnerd': (b'runner', (1, 90)), HOST_NODE: (fault.NODE.encode(), (1, 110)), + fault.PACK + '/provider-assets/kimi/linux-x64/native-closure.json': (json.dumps({'entries': self.entries}, indent=2).encode(), (1, 80))} + self.table = {21: ROOT, 22: PARENT, 23: TARGET} + self.args = {21: [RUNTIME + '/bin/paperclip-runnerd', '--run-id', 'run', '--environment-lease-id', 'workspace-id', '--lifecycle-mode', 'per_turn', '--state-dir', RUNTIME + '/sessions/' + 'a' * 64 + '/runner'], + 22: [HOST_NODE, '--eval', LOADER, DIST, fault.BOOTSTRAP, '0', '0', 'commonjs', '[]', '0'], 23: ['kimi-code']} + self.stats = {'/proc/21/exe': (90, stat.S_IFREG), '/proc/23/exe': (100, stat.S_IFREG), '/proc/23/fd/7': (100, stat.S_IFREG), + '/proc/24/exe': (100, stat.S_IFREG), '/proc/24/fd/7': (100, stat.S_IFREG), + '/proc/22/exe': (110, stat.S_IFREG), '/proc/22/fd/10': (110, stat.S_IFREG), '/proc/22/fd/4': (2, stat.S_IFDIR), '/proc/22/fd/3': (3, stat.S_IFIFO)} + def checked_fixture_file(name, *args): + if name.startswith('/memfd:'): + raise FileNotFoundError(2, 'deleted executable fixture') + return self.files[name] + patches = [patch.object(fault, 'proc', side_effect=lambda pid, boot: self.table[pid]), patch.object(fault, 'argv', side_effect=lambda pid: self.args[pid]), + patch.object(fault, 'checked_runner', side_effect=lambda name: self.files[name]), patch.object(fault, 'checked_file', side_effect=checked_fixture_file), + patch.object(Path, 'read_text', return_value=BOOT), patch.object(os, 'listdir', side_effect=lambda name: [str(pid) for pid in self.table]), + patch.object(os, 'getpgid', return_value=21), patch.object(os, 'lstat', return_value=SimpleNamespace(st_mode=stat.S_IFDIR | 0o500, st_dev=1, st_ino=2)), + patch.object(os.path, 'realpath', side_effect=lambda name: HOST_NODE if name == '/proc/22/exe' else name), + patch.object(os, 'readlink', return_value='/workspace'), patch.object(os, 'stat', side_effect=lambda name: SimpleNamespace(st_dev=1, st_ino=self.stats[name][0], st_mode=self.stats[name][1]))] + for mocked in patches: + mocked.start() + self.addCleanup(mocked.stop) + + @contextmanager + def parent_descriptor(self, anonymous=False, content=None): + content = fault.NODE.encode() if content is None else content + def metadata(inode=110): + return SimpleNamespace(st_mode=stat.S_IFREG | 0o500, st_dev=1, st_ino=inode, st_size=len(content), st_mtime_ns=3, st_ctime_ns=4, st_nlink=0 if anonymous else 1) + original_stat = os.stat + def proc_stat(path): + if path in ('/proc/22/exe', '/proc/22/fd/10'): + return metadata(self.stats[path][0]) + return original_stat(path) + kernel = SimpleNamespace(F_GET_SEALS=1034, F_SEAL_SEAL=1, F_SEAL_SHRINK=2, F_SEAL_GROW=4, F_SEAL_WRITE=8, fcntl=lambda fd, op: 15) + with patch.object(os, 'open', return_value=42), patch.object(os, 'close'), patch.object(os, 'fstat', side_effect=lambda fd: metadata()), patch.object(os, 'read', side_effect=[content, b'']), patch.object(os, 'stat', side_effect=proc_stat), patch.object(os, 'readlink', side_effect=lambda path: '/memfd:paperclip-verified-launch (deleted)' if path == '/proc/self/fd/42' else '/workspace'), patch.object(fault, 'fcntl', kernel, create=True): + yield + + def test_memfd_parent_uses_the_live_descriptor_without_reopening_deleted_alias(self): + self.args[22][0] = '/proc/self/fd/10' + with patch.object(os.path, 'realpath', side_effect=lambda name: '/memfd:paperclip-verified-launch (deleted)' if name == '/proc/22/exe' else name), self.parent_descriptor(True): + receipt = fault.inspect(self.config) + self.assertEqual(receipt['target'], TARGET) + self.assertFalse(any('/memfd:' in str(call) for call in fault.checked_file.call_args_list)) + + def test_matching_memfd_inode_and_seals_do_not_replace_the_full_node_digest(self): + self.args[22][0] = '/proc/self/fd/10' + with self.parent_descriptor(True, b'x' * len(fault.NODE.encode())), self.assertRaisesRegex(RuntimeError, 'parent_node'): + fault.inspect(self.config) + + def test_exact_pinned_parent_attests_title_overwritten_child(self): + receipt = fault.inspect(self.config) + self.assertEqual(receipt['target'], TARGET) + self.assertEqual(receipt['ancestry'], [TARGET, PARENT, ROOT]) + self.assertEqual(receipt['entrypointAttribution'], 'pinned_native_bootstrap_parent') + self.assertFalse(receipt['originalChildArgvAvailable']) + + def test_held_host_node_descriptor_requires_the_same_inode(self): + self.args[22][0] = '/proc/self/fd/10' + with self.parent_descriptor(): + self.assertEqual(fault.inspect(self.config)['target'], TARGET) + self.stats['/proc/22/fd/10'] = (999, stat.S_IFREG) + with self.parent_descriptor(), self.assertRaisesRegex(RuntimeError, 'parent_node_descriptor'): + fault.inspect(self.config) + + def test_title_alone_and_root_loss_never_select(self): + self.args[22][4] = 'foreign.cjs' + with self.assertRaisesRegex(RuntimeError, 'unique_native_child'): + fault.inspect(self.config) + self.args[22][4] = fault.BOOTSTRAP + self.args[23] = ['node'] + with self.assertRaisesRegex(RuntimeError, 'native_title'): + fault.inspect(self.config) + + def test_wrong_run_lease_birth_boot_and_root_executable_fail(self): + for field, value in [('runtimeEnvironmentLeaseId', 'foreign'), ('runnerdSha256', 'sha256:' + '0' * 64), ('nodeSha256', 'sha256:' + '0' * 64), ('closureSha256', '0' * 64), ('guardLoaderSha256', 'sha256:' + '0' * 64)]: + with self.subTest(field=field), self.assertRaises(RuntimeError): + fault.inspect({**self.config, field: value}) + for changed in [{'startTicks': '999'}, {'bootId': 'foreign'}, {'ppid': 99}]: + with self.subTest(changed=changed), self.assertRaises(RuntimeError): + fault.inspect({**self.config, 'root': {**ROOT, **changed}}) + self.args[21][2] = 'foreign' + with self.assertRaisesRegex(RuntimeError, 'root_binding'): + fault.inspect(self.config) + + def test_native_main_node_guard_bootstrap_and_parent_code_are_exact(self): + for name in [fault.NODE, fault.ENTRY, fault.GUARD, fault.BOOTSTRAP]: + original = self.files[DIST + '/' + name] + self.files[DIST + '/' + name] = (b'changed', original[1]) + with self.subTest(name=name), self.assertRaises(RuntimeError): + fault.inspect(self.config) + self.files[DIST + '/' + name] = original + for index, value in [(2, 'foreign loader'), (5, '1'), (8, '["module"]'), (0, '/foreign/node')]: + original = self.args[22][index] + self.args[22][index] = value + with self.subTest(index=index), self.assertRaises(RuntimeError): + fault.inspect(self.config) + self.args[22][index] = original + + def test_live_executable_and_retained_descriptors_must_match(self): + for name in ['/proc/23/exe', '/proc/23/fd/7', '/proc/22/fd/4']: + original = self.stats[name] + self.stats[name] = (999, original[1]) + with self.subTest(name=name), self.assertRaises(RuntimeError): + fault.inspect(self.config) + self.stats[name] = original + self.stats['/proc/22/fd/3'] = (3, stat.S_IFREG) + with self.assertRaisesRegex(RuntimeError, 'parent_source_pipe'): + fault.inspect(self.config) + + def test_cwd_ancestry_birth_and_ambiguity_fail_closed(self): + with patch.object(os, 'readlink', return_value='/foreign'), self.assertRaisesRegex(RuntimeError, 'cwd'): + fault.inspect(self.config) + self.table[23] = {**TARGET, 'startTicks': '99'} + with self.assertRaisesRegex(RuntimeError, 'unique_native_child'): + fault.inspect(self.config) + self.table[23] = TARGET + self.table[24] = {**TARGET, 'pid': 24} + self.args[24] = ['kimi-code'] + with self.assertRaisesRegex(RuntimeError, 'unique_native_child'): + fault.inspect(self.config) + + def test_canonical_closure_metadata_rejects_replaced_unsorted_or_open_entries(self): + for changed in [list(reversed(self.entries)), [*self.entries, self.entries[-1]], [{**self.entries[0], 'path': '../escape'}, self.entries[1]], [{**self.entries[0], 'extra': True}, self.entries[1]]]: + with self.subTest(changed=changed), self.assertRaises(RuntimeError): + fault.parse_closure(json.dumps({'entries': changed}).encode(), self.config['closureSha256']) + + +class ParentDescriptorTests(unittest.TestCase): + def setUp(self): + self.content = b'pinned node bytes' + self.before = SimpleNamespace(st_mode=stat.S_IFREG | 0o500, st_dev=1, st_ino=110, st_size=len(self.content), st_mtime_ns=3, st_ctime_ns=4, st_nlink=0) + self.stats = {'/proc/22/fd/10': self.before, '/proc/22/exe': self.before} + self.opened = Mock(return_value=42) + self.closed = Mock() + self.fstat = Mock(side_effect=lambda fd: self.before) + self.read = Mock(side_effect=[self.content, b'']) + self.readlink = Mock(return_value='/memfd:paperclip-verified-launch (deleted)') + self.kernel = SimpleNamespace(F_GET_SEALS=1034, F_SEAL_SEAL=1, F_SEAL_SHRINK=2, F_SEAL_GROW=4, F_SEAL_WRITE=8, fcntl=Mock(return_value=15)) + patches = [patch.object(os, 'open', self.opened), patch.object(os, 'close', self.closed), patch.object(os, 'fstat', self.fstat), patch.object(os, 'read', self.read), patch.object(os, 'stat', side_effect=lambda path: self.stats[path]), patch.object(os, 'readlink', self.readlink), patch.object(fault, 'fcntl', self.kernel, create=True)] + for mocked in patches: + mocked.start() + self.addCleanup(mocked.stop) + + def test_sealed_memfd_requires_complete_bytes_and_same_kernel_executable(self): + self.assertEqual(fault.checked_parent_descriptor(22), (self.content, (1, 110))) + self.opened.assert_called_once_with('/proc/22/fd/10', os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC) + self.kernel.fcntl.assert_has_calls([call(42, 1034), call(42, 1034)]) + self.closed.assert_called_once_with(42) + + def test_linked_descriptor_keeps_regular_single_link_checks_without_memfd_claim(self): + self.before.st_nlink = 1 + self.assertEqual(fault.checked_parent_descriptor(22), (self.content, (1, 110))) + self.kernel.fcntl.assert_not_called() + self.readlink.assert_not_called() + + def test_every_missing_kernel_seal_fails_before_read(self): + for bit in (1, 2, 4, 8): + with self.subTest(bit=bit): + self.kernel.fcntl.return_value = 15 ^ bit + with self.assertRaisesRegex(RuntimeError, 'parent_memfd_seals'): + fault.checked_parent_descriptor(22) + self.read.assert_not_called() + self.assertEqual(self.closed.call_count, 4) + + def test_an_unlinked_regular_file_cannot_impersonate_the_authenticated_memfd(self): + self.readlink.return_value = '/tmp/node (deleted)' + with self.assertRaisesRegex(RuntimeError, 'parent_memfd_required'): + fault.checked_parent_descriptor(22) + self.read.assert_not_called() + self.kernel.fcntl.assert_not_called() + + def test_memfd_write_mode_and_missing_seal_api_fail_closed(self): + self.before.st_mode = stat.S_IFREG | 0o700 + with self.assertRaisesRegex(RuntimeError, 'parent_memfd_required'): + fault.checked_parent_descriptor(22) + self.before.st_mode = stat.S_IFREG | 0o500 + del self.kernel.F_GET_SEALS + with self.assertRaisesRegex(RuntimeError, 'parent_memfd_required'): + fault.checked_parent_descriptor(22) + self.read.assert_not_called() + + def test_different_held_descriptor_or_executable_never_attests(self): + for path in ('/proc/22/fd/10', '/proc/22/exe'): + with self.subTest(path=path): + self.stats[path] = SimpleNamespace(**{**vars(self.before), 'st_ino': 999}) + with self.assertRaisesRegex(RuntimeError, 'parent_node_descriptor'): + fault.checked_parent_descriptor(22) + self.stats[path] = self.before + self.read.assert_not_called() + + def test_descriptor_metadata_change_during_read_is_rejected(self): + changed = SimpleNamespace(**{**vars(self.before), 'st_ctime_ns': 99}) + self.fstat.side_effect = [self.before, changed] + with self.assertRaisesRegex(RuntimeError, 'parent_descriptor_changed'): + fault.checked_parent_descriptor(22) + self.closed.assert_called_once_with(42) + + def test_short_read_or_changed_seals_cannot_attest(self): + self.read.side_effect = [b'short', b''] + with self.assertRaisesRegex(RuntimeError, 'parent_descriptor_changed'): + fault.checked_parent_descriptor(22) + self.read.side_effect = [self.content, b''] + self.kernel.fcntl.side_effect = [15, 7] + with self.assertRaisesRegex(RuntimeError, 'parent_memfd_seals'): + fault.checked_parent_descriptor(22) + + def test_nonregular_oversized_or_multilink_handles_fail_before_read(self): + for field, value in [('st_mode', stat.S_IFIFO), ('st_nlink', 2), ('st_size', 256 * 1024 * 1024 + 1)]: + with self.subTest(field=field): + before = getattr(self.before, field) + setattr(self.before, field, value) + with self.assertRaisesRegex(RuntimeError, 'parent_descriptor_shape'): + fault.checked_parent_descriptor(22) + setattr(self.before, field, before) + self.read.assert_not_called() + + +class PidfdTests(unittest.TestCase): + def test_changed_retired_or_reused_pidfd_never_signals(self): + before = {'target': TARGET} + for second, fdinfo in [({'target': {**TARGET, 'startTicks': '999'}}, 'Pid:\t23\n'), (before, 'Pid:\t-1\n')]: + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', side_effect=[before, second]), patch.object(os, 'pidfd_open', return_value=42, create=True), patch.object(signal, 'pidfd_send_signal', create=True) as sent, patch.object(os, 'close') as closed, patch.object(Path, 'read_text', return_value=fdinfo): + with self.assertRaises(RuntimeError): + fault.terminate({'root': ROOT}) + sent.assert_not_called() + closed.assert_called_once_with(42) + + def test_only_held_native_child_pidfd_receives_sigkill(self): + before = {'target': TARGET} + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', return_value=before), patch.object(os, 'pidfd_open', return_value=42, create=True) as opened, patch.object(signal, 'pidfd_send_signal', create=True) as sent, patch.object(os, 'close') as closed, patch.object(Path, 'read_text', return_value='Pid:\t23\n'): + receipt = fault.terminate({'root': ROOT, 'binding': {}, 'runtimeEnvironmentLeaseId': 'lease'}) + opened.assert_called_once_with(23) + sent.assert_called_once_with(42, signal.SIGKILL, None, 0) + closed.assert_called_once_with(42) + self.assertFalse(receipt['workerSignalled']) + + def test_worker_or_current_process_never_opens_a_pidfd(self): + for pid in [ROOT['pid'], os.getpid()]: + with patch.object(sys, 'platform', 'linux'), patch.object(fault, 'inspect', return_value={'target': {**TARGET, 'pid': pid}}), patch.object(os, 'pidfd_open', create=True) as opened, patch.object(signal, 'pidfd_send_signal', create=True): + with self.assertRaisesRegex(RuntimeError, 'not_worker'): + fault.terminate({'root': ROOT}) + opened.assert_not_called() + + def test_sealed_files_reject_links_mutability_and_extra_hardlinks(self): + with tempfile.TemporaryDirectory() as temporary: + path = Path(temporary).resolve() / 'pinned' + path.write_bytes(b'pinned') + with self.assertRaisesRegex(RuntimeError, 'file_writable'): + fault.checked_file(str(path), 100, True) + path.chmod(0o400) + self.assertEqual(fault.checked_file(str(path), 100, True)[0], b'pinned') + link = path.with_name('link') + link.symlink_to(path) + with self.assertRaisesRegex(RuntimeError, 'file_realpath'): + fault.checked_file(str(link), 100, True) + os.link(path, path.with_name('hardlink')) + with self.assertRaisesRegex(RuntimeError, 'file_shape'): + fault.checked_file(str(path), 100, True) + + def test_preinstalled_runner_link_is_bound_to_its_resolved_inode(self): + with tempfile.TemporaryDirectory() as temporary: + binary = Path(temporary).resolve() / 'installed-runner' + binary.write_bytes(b'pinned runner') + link = binary.with_name('runtime-runner') + link.symlink_to(binary) + self.assertEqual(fault.checked_runner(str(link)), fault.checked_runner(str(binary))) + + @unittest.skipUnless(sys.platform == 'linux' and hasattr(os, 'pidfd_open') and hasattr(signal, 'pidfd_send_signal'), 'Native Linux pidfd calibration required') + def test_real_title_overwritten_native_child_uses_pidfd_and_leaves_worker_alive(self): + node = shutil.which('node') + self.assertIsNotNone(node) + base = Path(tempfile.mkdtemp(prefix='kimi-fault-calibration-', dir='/tmp')) + snapshot = Path(tempfile.mkdtemp(prefix='paperclip-acpx-native-', dir='/tmp')) + distribution = snapshot / 'distribution' + workspace = base / 'workspace' + runtime = workspace / '.paperclip-runtime/paperclip-runner' + pack = base / 'pack' + host_node = pack / 'node_modules/node/bin/node' + root_child = None + owned = [] + try: + for path in [distribution / fault.NODE, distribution / fault.ENTRY, runtime / 'bin/paperclip-runnerd', host_node]: + path.parent.mkdir(parents=True, exist_ok=True) + for path in [distribution / fault.NODE, runtime / 'bin/paperclip-runnerd', host_node]: + shutil.copyfile(node, path) + path.chmod(0o500) + ready = workspace / 'ready.json' + (distribution / fault.ENTRY).write_text('import fs from "node:fs";process.title="kimi-code";fs.writeFileSync(' + json.dumps(str(ready)) + ',JSON.stringify({pid:process.pid}));setInterval(()=>{},1000);') + entries = [{'path': name, 'sha256': fault.digest((distribution / name).read_bytes()), 'size': (distribution / name).stat().st_size, 'executable': name == fault.NODE} for name in sorted([fault.NODE, fault.ENTRY])] + closure = pack / 'provider-assets/kimi/linux-x64/native-closure.json' + closure.parent.mkdir(parents=True) + closure.write_text(json.dumps({'entries': entries})) + loader = 'const source=require("node:fs").readFileSync(3,"utf8");require("node:vm").runInThisContext(source);' + template = 'const fs=require("node:fs"),cp=require("node:child_process");const fd=fs.openSync(' + json.dumps(fault.PLACEHOLDER + '/' + fault.NODE) + ',"r");const child=cp.spawn("/proc/self/fd/7",["--require",' + json.dumps(fault.PLACEHOLDER + '/' + fault.GUARD) + ',' + json.dumps(fault.PLACEHOLDER + '/' + fault.ENTRY) + ',"acp"],{stdio:["ignore","ignore","ignore","ignore","ignore","ignore","ignore",fd]});fs.closeSync(fd);child.on("exit",()=>process.exit(0));process.on("SIGTERM",()=>child.kill("SIGTERM"));' + config = {'binding': {'remoteCwd': str(workspace), 'runId': 'fixture'}, 'runtimeEnvironmentLeaseId': 'workspace-id', 'runnerdSha256': 'sha256:' + fault.digest((runtime / 'bin/paperclip-runnerd').read_bytes()), + 'closureSha256': fault.digest(json.dumps(entries, separators=(',', ':'), ensure_ascii=False).encode()), 'nodeSha256': 'sha256:' + fault.digest((distribution / fault.NODE).read_bytes()), + 'guardLoaderSha256': 'sha256:' + fault.digest(loader.encode()), 'bootstrapTemplate': template, 'guardTemplate': 'const digests={};'} + bootstrap, guard = fault.expected_sources(config, str(distribution), entries) + (distribution / fault.BOOTSTRAP).write_bytes(bootstrap) + (distribution / fault.GUARD).write_bytes(guard) + script = base / 'root.cjs' + script.write_text('const fs=require("node:fs"),cp=require("node:child_process");const directory=fs.openSync(' + json.dumps(str(distribution)) + ',"r");const child=cp.spawn(' + json.dumps(str(host_node)) + ',["--eval",' + json.dumps(loader) + ',' + json.dumps(str(distribution)) + ',' + json.dumps(fault.BOOTSTRAP) + ',"0","0","commonjs","[]","0"],{detached:true,stdio:["ignore","ignore","ignore","pipe",directory]});fs.closeSync(directory);child.stdio[3].end(fs.readFileSync(' + json.dumps(str(distribution / fault.BOOTSTRAP)) + '));process.on("SIGTERM",()=>{if(child.exitCode!==null||child.signalCode!==null)process.exit(0);child.once("exit",()=>process.exit(0));child.kill("SIGTERM")});setInterval(()=>{},1000);') + for directory, _, files in os.walk(snapshot): + for name in files: + path = Path(directory) / name + path.chmod(0o500 if path == distribution / fault.NODE else 0o400) + Path(directory).chmod(0o500) + check_cancelled() + root_child = subprocess.Popen([str(runtime / 'bin/paperclip-runnerd'), str(script), '--run-id', 'fixture', '--environment-lease-id', 'workspace-id', '--lifecycle-mode', 'per_turn', '--state-dir', str(runtime / 'sessions' / ('a' * 64) / 'runner')], cwd=workspace, env={'PATH': '/usr/bin:/bin'}, start_new_session=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL) + owned.append(os.pidfd_open(root_child.pid)) + deadline = time.monotonic() + 5 + while not ready.exists() and time.monotonic() < deadline: + check_cancelled() + time.sleep(.02) + self.assertTrue(ready.exists()) + target = json.loads(ready.read_text())['pid'] + self.assertEqual(fault.argv(target), ['kimi-code']) + boot = Path('/proc/sys/kernel/random/boot_id').read_text().strip() + config['root'] = fault.proc(root_child.pid, boot) + with patch.object(fault, 'PACK', str(pack)): + before = fault.inspect(config) + for identity in before['ancestry'][:-1]: + fd = os.pidfd_open(identity['pid']) + owned.append(fd) + self.assertEqual(fault.proc(identity['pid'], boot), identity) + check_cancelled() + receipt = fault.terminate(config) + self.assertEqual(receipt['target']['pid'], target) + self.assertFalse(receipt['workerSignalled']) + self.assertEqual(select.select([owned[0]], [], [], 0)[0], []) + self.assertTrue(select.select([owned[1]], [], [], 3)[0]) + finally: + if root_child is not None: + if owned: + try: + signal.pidfd_send_signal(owned[0], signal.SIGTERM, None, 0) + except ProcessLookupError: + pass + elif root_child.poll() is None: + root_child.terminate() + try: + root_child.wait(timeout=3) + except subprocess.TimeoutExpired: + if owned: + signal.pidfd_send_signal(owned[0], signal.SIGKILL, None, 0) + root_child.wait(timeout=3) + for fd in reversed(owned[1:]): + if not select.select([fd], [], [], 0)[0]: + try: + signal.pidfd_send_signal(fd, signal.SIGKILL, None, 0) + except ProcessLookupError: + pass + for fd in owned: + os.close(fd) + for directory, _, _ in os.walk(snapshot): + os.chmod(directory, 0o700) + shutil.rmtree(snapshot) + shutil.rmtree(base) + + +if __name__ == '__main__': + def cancelled(_signum, _frame): + global CANCELLED + CANCELLED = True + signal.signal(signal.SIGTERM, cancelled) + signal.signal(signal.SIGINT, cancelled) + unittest.main() diff --git a/tests/runner-e2e/kimi-provider-fault.test.ts b/tests/runner-e2e/kimi-provider-fault.test.ts new file mode 100644 index 00000000000..65ec400c35b --- /dev/null +++ b/tests/runner-e2e/kimi-provider-fault.test.ts @@ -0,0 +1,70 @@ +import { spawnSync } from "node:child_process"; +import { dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import { describe, expect, it } from "vitest"; +import { expectedNativeBootstrap, expectedNativeModuleGuard, KIMI_FAULT_ENTRY, kimiFaultRuntimeContract, validateKimiProviderDeathReceipt } from "./kimi-provider-fault-contract.js"; + +const bootId = "12345678-1234-1234-1234-123456789abc"; +const root = { pid: 21, ppid: 1, startTicks: "100", bootId }; +const parent = { pid: 22, ppid: 21, startTicks: "101", bootId }; +const target = { pid: 23, ppid: 22, startTicks: "102", bootId }; +const binding = { companyId: "company", environmentId: "environment", runId: "run", leaseId: "lease", sandboxId: "sandbox", image: `registry/image@sha256:${"a".repeat(64)}`, remoteCwd: "/workspace" }; +const hash = `sha256:${"a".repeat(64)}`; +function receipt() { + const contract = kimiFaultRuntimeContract("linux-x64"); + return { schema: "paperclip.e2e.kimi-provider-death.v1", binding, runtimeEnvironmentLeaseId: "workspace-id", target, ancestry: [target, parent, root], + ...contract, entrypointSha256: hash, guardSha256: hash, bootstrapSha256: hash, + entrypointAttribution: "pinned_native_bootstrap_parent", originalChildArgvAvailable: false, observedChildTitle: "kimi-code", + signalled: true, signal: "SIGKILL", targetKind: "kimi_native_child", workerSignalled: false }; +} + +describe("Kimi native child fault contract", () => { + it("uses compiled production loader, module guard and native bootstrap bytes", () => { + const contract = kimiFaultRuntimeContract("linux-x64"), distribution = "/tmp/paperclip-acpx-native-test/distribution"; + expect(contract.guardLoaderSha256).toMatch(/^sha256:[a-f0-9]{64}$/u); + expect(contract.closureSha256).toMatch(/^[a-f0-9]{64}$/u); + const bootstrap = expectedNativeBootstrap(distribution, contract.nodeSha256); + expect(bootstrap).toContain(`const executable="${distribution}/node/bin/node"`); + expect(bootstrap).toContain(`"${distribution}/${KIMI_FAULT_ENTRY}","acp"`); + expect(bootstrap).toContain('const childExecutable=process.platform==="linux"?"/proc/self/fd/"+stdio.length:executable'); + expect(expectedNativeModuleGuard(distribution, [{ path: KIMI_FAULT_ENTRY, sha256: "a".repeat(64) }])) + .toContain(`const digests={"${distribution}/${KIMI_FAULT_ENTRY}":"${"a".repeat(64)}"};`); + expect(() => expectedNativeBootstrap(distribution, "invalid")).toThrow("node_digest"); + }); + + it("accepts the exact child and rejects misleading or incomplete receipts", () => { + const valid = receipt(); + expect(validateKimiProviderDeathReceipt(valid, binding, root, "workspace-id")).toEqual(valid); + for (const alteration of [{ workerSignalled: true }, { targetKind: "runnerd" }, { target: root }, { signalled: false }, + { entrypointAttribution: "process_title" }, { originalChildArgvAvailable: true }, { observedChildTitle: "node" }, + { nodeSha256: hash }, { closureSha256: "a".repeat(64) }, { guardLoaderSha256: hash }, { bootstrapSha256: "invalid" }, + { runtimeEnvironmentLeaseId: "foreign" }, { binding: { ...binding, runId: "foreign" } }, { arbitrary: true }, + { ancestry: [target, root] }, { ancestry: [target, { ...parent, startTicks: "" }, root] }, + { ancestry: [target, { ...parent, ppid: 999 }, root] }, { ancestry: [target, parent, { ...root, startTicks: "999" }] }, + ]) expect(() => validateKimiProviderDeathReceipt({ ...valid, ...alteration }, binding, root, "workspace-id")).toThrow(); + }); + + it("accepts real observer root diagnostics without accepting changed identity or open receipt rows", () => { + const valid = receipt(), observedRoot = { ...root, group: root.pid, state: "S" }; + expect(validateKimiProviderDeathReceipt(valid, binding, observedRoot, "workspace-id")).toEqual(valid); + for (const changed of [{ pid: 999 }, { ppid: 999 }, { startTicks: "999" }, { bootId: "ffffffff-ffff-ffff-ffff-ffffffffffff" }]) { + expect(() => validateKimiProviderDeathReceipt(valid, binding, { ...observedRoot, ...changed }, "workspace-id")).toThrow("receipt_ancestry"); + } + expect(() => validateKimiProviderDeathReceipt({ ...valid, ancestry: [target, parent, observedRoot] }, binding, observedRoot, "workspace-id")) + .toThrow("receipt_ancestry"); + }); + + it("calibrates identity negatives and native Linux pidfd without credentials or network", () => { + const result = spawnSync("/usr/bin/python3", ["-B", fileURLToPath(new URL("./kimi-provider-fault.test.py", import.meta.url))], { + env: { PATH: `${dirname(process.execPath)}:/usr/bin:/bin`, PYTHONDONTWRITEBYTECODE: "1" }, + cwd: dirname(fileURLToPath(import.meta.url)), timeout: 30_000, killSignal: "SIGTERM", maxBuffer: 64 * 1024, + encoding: "utf8", stdio: ["ignore", "pipe", "pipe"], + }); + expect(result.error).toBeUndefined(); + expect(result.signal).toBeNull(); + expect(result.status, result.stderr).toBe(0); + expect(result.stderr).toContain("Ran 25 tests"); + if (process.platform === "linux") expect(result.stderr).not.toContain("skipped"); + else expect(result.stderr).toContain("skipped=1"); + }, 35_000); +}); diff --git a/tests/runner-e2e/kimi-qualification.test.ts b/tests/runner-e2e/kimi-qualification.test.ts new file mode 100644 index 00000000000..8c28f996578 --- /dev/null +++ b/tests/runner-e2e/kimi-qualification.test.ts @@ -0,0 +1,94 @@ +import { describe, expect, it } from "vitest"; +import { runnerMatrix, runnerSuites } from "./catalog.js"; +import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; +import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js"; +import { assertRunnerE2EPrerequisites } from "./prerequisites.js"; + +const suiteId = "kimi-qualification"; +const cells = runnerMatrix.filter(cell => cell.suite.id === suiteId); + +describe("Kimi Code explicit Product qualification", () => { + it("keeps twenty separately selected user journeys without expanding other harness suites", () => { + expect(cells).toHaveLength(20); + expect(new Set(cells.map(cell => cell.environment.id))).toEqual(new Set(["local", "daytona"])); + expect(new Set(cells.map(cell => cell.task.id))).toEqual(new Set([ + "message-marker", "plan-revise-accept", "ask-question", "structured-question-resume", + "structured-question-restart-resume", "file-edit-validate", "stop-new-resume", "continuity-restart", + "native-write-permission-stop", "native-write-provider-loss", + ])); + expect(selectRunnerExecutions(parseRunnerSelectors(["--suite", suiteId]))).toHaveLength(20); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(cell => cell.suite.id === suiteId)).toBe(false); + expect(() => selectRunnerExecutions(parseRunnerSelectors(["--profile", "runner-acpx-kimi"]))).toThrow("zero executions"); + expect(runnerMatrix.filter(cell => cell.suite.id === "extended-harnesses")).toHaveLength(30); + expect(runnerSuites.find(suite => suite.id === suiteId)?.definitionMetadata).toMatchObject({ + qualification: "qualified", scheduling: "explicit-only", modelDiscovery: "exact-model-authenticated-observed", + nativeQuestions: "not-claimed", nativePlans: "not-claimed", accounting: "usage-and-cost-unavailable", + }); + }); + + it.each([ + { platform: "linux" as const, arch: "x64" as const }, + { platform: "darwin" as const, arch: "x64" as const }, + { platform: "win32" as const, arch: "x64" as const }, + ])("rejects the local provider-loss cell before admission on $platform-$arch", host => { + const cell = cells.find(cell => cell.environment.id === "local" && cell.task.id === "native-write-provider-loss")!; + expect(() => assertRunnerE2EPrerequisites([cell], host)).toThrow(/Local native Kimi provider-loss.*No provider credentials were loaded or sent/); + }); + + it("keeps the twenty logical cells and qualified Mac-local/Linux-Daytona paths available", () => { + for (const cell of cells) { + expect(() => assertRunnerE2EPrerequisites([cell], { platform: "darwin", arch: "arm64" })).not.toThrow(); + if (cell.environment.id === "daytona" || cell.task.id !== "native-write-provider-loss") { + expect(() => assertRunnerE2EPrerequisites([cell], { platform: "linux", arch: "x64" })).not.toThrow(); + } + } + expect(cells).toHaveLength(20); + }); + + it("binds only the selected encrypted Code credential and marks each attempt single-shot", () => { + const ref = { type: "secret_ref" as const, secretId: "11111111-1111-4111-8111-111111111111", version: "latest" as const }; + for (const cell of cells) { + expect(cell.suite.manualOnly).toBe(true); + expect(cell.profile).toMatchObject({ qualificationCandidate: "kimi", model: "kimi-for-coding", + modelQualification: { source: "qualified_runner_profile" } }); + expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); + expect(cell.requiredCredentials).toEqual(cell.environment.id === "local" ? ["KIMI_API_KEY"] : ["KIMI_API_KEY", "DAYTONA_API_KEY"]); + expect(cell.profile.buildAgent({ executionId: cell.id, environmentId: "env", environmentFixtureId: cell.environment.id, + workspacePath: "/workspace", secretRefs: { KIMI_API_KEY: ref } })).toMatchObject({ + adapterType: "paperclip_runner", adapterConfig: { provider: "acpx", acpxAgent: "kimi", model: "kimi-for-coding", + lifecycleMode: "per_turn", timeoutSec: 120, env: { KIMI_API_KEY: ref } }, + }); + } + }); + + it("requires its own manual suite without inheriting candidate authority", () => { + const cell = cells[0]!; + const ambient = { PAPERCLIP_RUNNER_ACPX_QUALIFICATION: "ambient", PAPERCLIP_ACPX_CANDIDATE_ADMISSION: "ambient" }; + const env = buildRunnerE2EProcessEnvironment(ambient, [cell]); + expect(env.PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined(); + expect(env.PAPERCLIP_ACPX_CANDIDATE_ADMISSION).toBeUndefined(); + for (const suite of [{ ...cell.suite, manualOnly: false }, { ...cell.suite, id: "extended-harnesses" }, { ...cell.suite, id: "unrelated-manual-suite" }]) { + expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, suite }])).toThrow("explicit provider qualification suite"); + } + expect(() => buildRunnerE2EProcessEnvironment({}, [{ ...cell, profile: { ...cell.profile, id: "unrelated-profile" } }])).toThrow("explicit provider qualification suite"); + const empty = buildRunnerE2EProcessEnvironment(ambient, []); + expect(empty.PAPERCLIP_RUNNER_ACPX_QUALIFICATION).toBeUndefined(); + expect(empty.PAPERCLIP_ACPX_CANDIDATE_ADMISSION).toBeUndefined(); + }); + + it("removes ambient Kimi secrets and login-home aliases before starting the product server", () => { + const source = { KIMI_API_KEY: "selected", KIMI_MODEL_API_KEY: "ambient-key", KIMI_CODE_HOME: "/ambient-login", + KIMI_CODE_BASE_URL: "https://unselected.example", KIMI_MODEL_NAME: "unselected", KIMI_WEB_SEARCH_API_KEY: "other-secret" }; + const server = buildPaperclipServerEnvironment(source); + for (const name of Object.keys(source)) expect(server[name]).toBeUndefined(); + }); + + it("requires independent workspace and downloaded file bytes in the delivery cell", () => { + for (const cell of cells.filter(cell => cell.task.id === "file-edit-validate")) { + expect(cell.task.buildMatchers("nonce", cell)).toEqual(expect.arrayContaining([ + { kind: "file_exact", path: "extended-nonce.txt", expected: "verified-nonce\n" }, + { kind: "artifact_exact", name: "extended-nonce.txt", expected: "verified-nonce\n", mimeType: "text/plain" }, + ])); + } + }); +}); diff --git a/tests/runner-e2e/launch.ts b/tests/runner-e2e/launch.ts index 36cfa5cd231..3a278d79dbe 100644 --- a/tests/runner-e2e/launch.ts +++ b/tests/runner-e2e/launch.ts @@ -5,7 +5,7 @@ import { readRecoveredFeedback } from "./feedback-recovery.js"; import { readInterruptedAttemptTrace } from "./trace-recovery.js"; import { EVAL_OBSERVABILITY_SECRET_NAMES, evalBraintrustConfig, finalizeEvalAttemptTrace } from "./braintrust.js"; import { finalizeEvalAttemptThroughRelay } from "./eval-finalization-relay.js"; -import { runnerE2EPlaywrightInvocation, runnerE2EPlaywrightShutdownSignal } from "./web-server-command.js"; +import { runnerE2EPlaywrightCaseTimeoutMs, runnerE2EPlaywrightInvocation, runnerE2EPlaywrightShutdownSignal, runnerE2EPlaywrightWatchdogMs } from "./web-server-command.js"; import { verifyInstalledDaytonaPlugin } from "./installed-daytona-plugin.js"; import { assertInstalledCliSelection, assertInstalledStartupOnly, verifyInstalledCli } from "./installed-cli.js"; import { createProcessTreeOwner, stopOwnedProcessTree } from "./process-tree-owner.js"; @@ -33,7 +33,8 @@ import { } from "node:fs/promises"; import { isImmutableDaytonaImage, runnerMatrix } from "./catalog.js"; import { renderRunnerE2EDashboard } from "./dashboard.js"; -import { packageEvidence } from "./evidence.js"; +import { packageEvidence, publishRunnerE2EAttemptEvidence } from "./evidence.js"; +import { prepareRunnerE2ECampaignDirectory, resolveRunnerE2EResultsRoot } from "./results-root.js"; import { privateCellEvidenceOwner } from "./manual-private-report-assets.js"; import { readManualPrivateHandoff, sealManualResultEvidence, preserveOriginalFrameworkResult, sealStandardOriginalEvidence, type ManualPrivateHandoff } from "./manual-private-handoff.js"; import { requiresPrivateManualEvidence } from "./manual-private-retention.js"; @@ -485,6 +486,7 @@ async function runAttempt(input: { ); } if (process.env.GITHUB_ACTIONS === "true" && !input.operations && executions.some(candidate => requiresPrivateManualEvidence(candidate.id)) && !manualPrivateHandoff) throw new Error("Sensitive manual evidence requires verified restricted retention before setup"); + await prepareRunnerE2ECampaignDirectory(resultsRoot, campaignId); const upgrades = await readUpgradeArtifacts(process.env[upgradeArtifactKey]); if (upgrades) { // All attempts and the browser use A's installed fixture admission. The @@ -588,9 +590,9 @@ async function runAttempt(input: { Math.max( ...executions.map( (candidate) => - candidate.task.attemptTimeoutMs[candidate.environment.id], + runnerE2EPlaywrightCaseTimeoutMs(candidate.task.flow, candidate.task.attemptTimeoutMs[candidate.environment.id]), ), - ) + 90_000, + ), ), PAPERCLIP_HOME: paperclipHome, PAPERCLIP_INSTANCE_ID: instanceId, @@ -622,14 +624,7 @@ async function runAttempt(input: { const watchdog = options.ui || options.debug ? null - : executions.reduce( - (total, candidate) => - total + - candidate.task.attemptTimeoutMs[candidate.environment.id] + - 90_000, - 0, - ) + - 5 * 60_000; + : runnerE2EPlaywrightWatchdogMs(executions); const processResult = await runProcess( runnerE2EPlaywrightInvocation(repositoryRoot, playwrightArgs), childEnv, @@ -754,7 +749,7 @@ async function runAttempt(input: { } } for (const [index, candidate] of executions.entries()) { - let result = { + let result: RunnerE2EResult = { ...results[index], accountingIdentity: { campaignId: accountingCampaignId, attemptId: accountingAttemptId }, }; @@ -812,7 +807,6 @@ async function runAttempt(input: { } } const casePrivateDir = path.join(privateDir, "cases", candidate.task.id); - const resultPath = path.join(casePrivateDir, "result.json"); const uploadDir = path.join( resultsRoot, campaignId, @@ -834,48 +828,59 @@ async function runAttempt(input: { originalResult: { status: original.status, failureClass: original.failureClass, error: original.error }, }, credentials), null, 2)}\n`, { mode: 0o600 }); } - if (manualPrivateHandoff) await preserveOriginalFrameworkResult(casePrivateDir); - await writeFile( - resultPath, - `${JSON.stringify(sanitizeJson(result, credentials), null, 2)}\n`, - "utf8", - ); - let evidence = await packageEvidence({ - privateDir: casePrivateDir, - uploadDir, - secrets: credentials, - expectPassScreenshot: result.status === "passed", - ...(manualPrivateHandoff ? { privateCell: { owner: await privateCellEvidenceOwner(manualPrivateHandoff, candidate.id), layout: "original" as const } } : {}), - }); - const originalEvidence = evidence, originalFeaturePassed = result.status === "passed"; - if (evidence.leaks.length > 0 || evidence.missing.length > 0) { - result = { - ...result, - status: "failed", - failureClass: - evidence.leaks.length > 0 - ? "secret_leak" - : "permanent_infrastructure", - error: - evidence.leaks.length > 0 - ? `Secret leak rejected from evidence: ${evidence.leaks.map((leak) => leak.file).join(", ")}` - : `Required evidence missing: ${evidence.missing.join(", ")}`, - }; + const resultPath = path.join(casePrivateDir, "result.json"); + if (manualPrivateHandoff) { + await preserveOriginalFrameworkResult(casePrivateDir); await writeFile( resultPath, `${JSON.stringify(sanitizeJson(result, credentials), null, 2)}\n`, "utf8", ); - evidence = await packageEvidence({ + let evidence = await packageEvidence({ + privateDir: casePrivateDir, + uploadDir, + secrets: credentials, + expectPassScreenshot: result.status === "passed", + privateCell: { owner: await privateCellEvidenceOwner(manualPrivateHandoff, candidate.id), layout: "original" as const }, + }); + const originalEvidence = evidence, originalFeaturePassed = result.status === "passed"; + if (evidence.leaks.length > 0 || evidence.missing.length > 0) { + result = { + ...result, + status: "failed", + failureClass: + evidence.leaks.length > 0 + ? "secret_leak" + : "permanent_infrastructure", + error: + evidence.leaks.length > 0 + ? `Secret leak rejected from evidence: ${evidence.leaks.map((leak) => leak.file).join(", ")}` + : `Required evidence missing: ${evidence.missing.join(", ")}`, + }; + await writeFile( + resultPath, + `${JSON.stringify(sanitizeJson(result, credentials), null, 2)}\n`, + "utf8", + ); + evidence = await packageEvidence({ + privateDir: casePrivateDir, + uploadDir, + secrets: credentials, + expectPassScreenshot: false, + privateCell: { owner: await privateCellEvidenceOwner(manualPrivateHandoff, candidate.id), layout: "original" as const }, + }); + } + await sealStandardOriginalEvidence({ context: manualPrivateHandoff, + executionId: candidate.id, source: casePrivateDir, evidence: originalEvidence, secrets: credentials, expectPassScreenshot: originalFeaturePassed }); + } else { + result = await publishRunnerE2EAttemptEvidence({ + result, + resultsRoot, privateDir: casePrivateDir, uploadDir, secrets: credentials, - expectPassScreenshot: false, - ...(manualPrivateHandoff ? { privateCell: { owner: await privateCellEvidenceOwner(manualPrivateHandoff, candidate.id), layout: "original" as const } } : {}), }); } - if (manualPrivateHandoff) await sealStandardOriginalEvidence({ context: manualPrivateHandoff, - executionId: candidate.id, source: casePrivateDir, evidence: originalEvidence, secrets: credentials, expectPassScreenshot: originalFeaturePassed }); console.log( `${result.status === "passed" ? "PASS" : "FAIL"} ${candidate.id} attempt ${attempt} -> ${uploadDir}`, ); @@ -1071,7 +1076,7 @@ async function main() { } const executions = selectRunnerExecutions(options, runnerMatrix); manualPrivateHandoff = await readManualPrivateHandoff(process.env, executions.map(execution => execution.id)); - if (manualPrivateHandoff) resultsRoot = path.join(manualPrivateHandoff.root, "results"); + resultsRoot = manualPrivateHandoff ? path.join(manualPrivateHandoff.root, "results") : resolveRunnerE2EResultsRoot(repositoryRoot); if (options.installedStartupOnly) { assertInstalledStartupOnly(process.env, executions, options); assertRunnerE2EPrerequisites(executions); @@ -1128,8 +1133,9 @@ async function main() { process.env.PAPERCLIP_E2E_CAMPAIGN_ID ?? `local-${new Date().toISOString().replace(/[:.]/g, "-")}`, ); - const summaryDir = path.join(resultsRoot, campaignId); - await mkdir(summaryDir, { recursive: true }); + await loadLocalEnvironment(process.env); + resultsRoot = manualPrivateHandoff ? path.join(manualPrivateHandoff.root, "results") : resolveRunnerE2EResultsRoot(repositoryRoot); + const summaryDir = await prepareRunnerE2ECampaignDirectory(resultsRoot, campaignId); if (executions.some(execution => execution.suite.id === "native-completion")) { process.env[NATIVE_COMPLETION_PREFLIGHT_ENV] = prepareNativeCompletionPreflight(summaryDir); } @@ -1140,7 +1146,6 @@ async function main() { process.env[STOCK_PREFLIGHT_ENV] = prepareStockHarnessPreflight(summaryDir); } - await loadLocalEnvironment(process.env); assertRemoteNativeEvidencePrerequisites(executions, process.env); const missingCredentials = process.env.PAPERCLIP_E2E_OPERATIONS_MANIFEST ? [] : [ ...new Set( diff --git a/tests/runner-e2e/live-fixtures.test.ts b/tests/runner-e2e/live-fixtures.test.ts index fa8aaa1201a..dd618c3da2f 100644 --- a/tests/runner-e2e/live-fixtures.test.ts +++ b/tests/runner-e2e/live-fixtures.test.ts @@ -1,3 +1,6 @@ +import { readFileSync } from "node:fs"; +import { transformSync } from "esbuild"; +import * as fixtureReadiness from "./live-fixtures.js"; import { describe, expect, it, vi } from "vitest"; import { mkdir, mkdtemp, realpath, rm } from "node:fs/promises"; import { tmpdir } from "node:os"; @@ -5,9 +8,47 @@ import path from "node:path"; import type { RunnerApi } from "./api.js"; import { buildRuntimeUsage } from "./billing.js"; import { runnerMatrix } from "./catalog.js"; -import { setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage } from "./live-fixtures.js"; +import { deleteDaytonaEnvironment, setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage } from "./live-fixtures.js"; describe("live runner fixtures", () => { + it.each([ + ["daytona", "native-write-permission-stop", false], + ["daytona", "native-write-provider-loss", false], + ["daytona", "message-marker", true], + ["local", "native-write-permission-stop", null], + ["local", "native-write-provider-loss", null], + ] as const)("produces the Kimi lease lifecycle for %s %s", async (environment, task, reusable) => { + const execution = runnerMatrix.find(row => row.id === `kimi-qualification.runner-acpx-kimi.${environment}.${task}`)!; + let environmentBody: any; + let agentBody: any; + const api = { + async get() { return [{ id: "local", driver: "local", config: {} }]; }, + async postSensitive(_url: string, data: any) { return { id: `secret-${data.key}` }; }, + async post(url: string, data: any) { + if (url === "/api/plugins/install") return { id: "plugin", status: "ready" }; + if (url === "/api/companies") return { id: "company", name: "Fixture" }; + if (url.endsWith("/environments")) { environmentBody = data; return { id: "environment", driver: "sandbox", config: data.config }; } + if (url.endsWith("/agents")) { agentBody = data; return { id: "agent", companyId: "company", ...data }; } + throw new Error(`Unexpected POST ${url}`); + }, + async delete() {}, + } as unknown as RunnerApi; + const fixtures = await setupLiveFixtures({ api, execution, executionNonce: "lifecycle-contract", workspacePath: "/tmp/fixture", + credentials: { KIMI_API_KEY: "synthetic-kimi-key", DAYTONA_API_KEY: "synthetic-daytona-key" }, + daytonaImage: `registry.example.test/runtime@sha256:${"a".repeat(64)}` }); + try { + if (reusable === null) { + expect(environmentBody).toBeUndefined(); + expect(fixtures.environment).toEqual({ id: "local", driver: "local", config: {} }); + } else { + expect(environmentBody.config).toMatchObject({ reuseLease: reusable, runnerLifecycleMode: reusable ? "warm" : "per_turn", + apiKey: { type: "secret_ref", secretId: "secret-DAYTONA_API_KEY", version: "latest" } }); + expect(fixtures.environment.config).toEqual(environmentBody.config); + } + expect(agentBody.adapterConfig).toMatchObject({ provider: "acpx", acpxAgent: "kimi", model: "kimi-for-coding" }); + } finally { await fixtures.teardown(); } + }); + it.each([ ["receipt-smoke", 100], ["same-task-roundtrip", 1_000], ] as const)("bounds %s before creating its initial subscription agent", async (taskId, budget) => { @@ -473,3 +514,121 @@ describe("managed setup runtime cost coverage", () => { expect(incomplete.leaseCount).toBe(3); }); }); + +describe("Daytona fixture readiness after candidate startup", () => { + const companyId = "11111111-2222-4333-8444-555555555555"; + const pluginId = "22222222-2222-4333-8444-555555555555"; + const pluginKey = "paperclip.daytona-sandbox-provider"; + const waitReady = () => (fixtureReadiness as unknown as { + waitForFixtureDaytonaProvider: (api: Pick, input: { companyId: string; deadlineAt: number }) => Promise; + }).waitForFixtureDaytonaProvider; + function candidateCompanyCallback(execution: unknown, api: unknown, restart: () => Promise, deadlineAt: number) { + const source = readFileSync(new URL("./runner.spec.ts", import.meta.url), "utf8"); + const start = source.indexOf(" const onCompanyCreated = async (id: string) => {"); + const end = source.indexOf(" const api = new RunnerApi(request);", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const js = transformSync(source.slice(start, end), { loader: "ts", format: "cjs" }).code; + return new Function("kimiCandidateAdmissionForExecution", "execution", "api", "nonce", "startedAtMs", "deadlineMs", "trace", "saveTrace", "restartIsolatedPaperclipServer", "waitForFixtureDaytonaProvider", js + ";return onCompanyCreated;")( + () => ({}), execution, api, "fixture", 0, deadlineAt, undefined, async () => {}, restart, waitReady(), + ) as (id: string) => Promise; + } + function apiAt(readiness: () => boolean, mismatch = false) { + const calls: string[] = []; + return { + calls, + async get(route: string): Promise { + calls.push(route); + if (route.endsWith("/dashboard")) return { pluginId, worker: { status: readiness() ? "running" : "starting" } } as T; + return { sandboxProviders: { daytona: readiness() ? { status: "supported", source: "plugin", pluginKey, pluginId: mismatch ? "foreign" : pluginId, supportsRunExecution: true } : { status: "unsupported" } } } as T; + }, + async post() { calls.push("ENV_POST"); if (!readiness()) throw new Error("environment POST422: plugin worker starting"); return { id: "environment" }; }, + }; + } + it("awaits real public provider readiness before the environment POST after HTTP restart ACK", async () => { + vi.useFakeTimers(); vi.setSystemTime(0); + try { + let running = false; setTimeout(() => { running = true; }, 1_000); + const api = apiAt(() => running); + const callback = candidateCompanyCallback({ environment: { id: "daytona" } }, api, async () => {}, 5_000); + const outcome = (async () => { await callback(companyId); return api.post(); })().then(value => ({ ok: true, value }), error => ({ ok: false, error: error.message })); + await vi.advanceTimersByTimeAsync(1_250); + expect(await outcome).toMatchObject({ ok: true, value: { id: "environment" } }); + expect(api.calls.filter(x => x === "ENV_POST")).toHaveLength(1); + expect(api.calls.some(x => x.endsWith("/dashboard"))).toBe(true); + } finally { vi.useRealTimers(); } + }); + it("leaves local candidate restart sequencing free of Daytona readiness requests", async () => { + const api = apiAt(() => false); + await candidateCompanyCallback({ environment: { id: "local" } }, api, async () => {}, Date.now() + 5_000)(companyId); + expect(api.calls).toEqual([]); + }); + it("does not admit a DB-ready provider whose live worker never runs", async () => { + vi.useFakeTimers(); vi.setSystemTime(0); + try { + const api = apiAt(() => false); + const result = waitReady()(api, { companyId, deadlineAt: 1_000 }).then(() => null, error => error); + await vi.advanceTimersByTimeAsync(1_100); + const error = await result; + expect(error).toMatchObject({ failureClass: "candidate_failure" }); + expect(api.calls).not.toContain("ENV_POST"); + } finally { vi.useRealTimers(); } + }); + it("rejects a different advertised plugin identity before creating an environment", async () => { + const api = apiAt(() => true, true); + await expect(waitReady()(api, { companyId, deadlineAt: Date.now() + 5_000 })).rejects.toThrow(/plugin identity/); + expect(api.calls).not.toContain("ENV_POST"); + }); +}); + + +describe("absolute Daytona fixture cleanup deadline", () => { + it("waits for an ordinary pending cleanup to succeed beyond the default120s without extending the original Kimi window", async () => { + vi.useFakeTimers(); vi.setSystemTime(0); + const times: number[] = []; + const remove = vi.fn(async (_path: string, options?: { allowNotFound?: boolean; timeout?: number }) => { + times.push(Date.now()); + expect(options?.allowNotFound).toBe(true); + expect(options!.timeout).toBeLessThanOrEqual(660_000 - Date.now()); + if (Date.now() < 300_000) throw new Error("DELETE409 pending_cleanup"); + }); + try { + const completed = deleteDaytonaEnvironment({ delete: remove }, "environment", 660_000); + await vi.advanceTimersByTimeAsync(300_000); + await completed; + expect(times.at(-1)).toBe(300_000); + expect(times.length).toBe(101); + } finally { vi.useRealTimers(); } + }); + + it("stops at the absolute deadline and clamps final request/sleep time", async () => { + vi.useFakeTimers(); vi.setSystemTime(0); + const times: number[] = [], timeouts: number[] = []; + const remove = vi.fn(async (_path: string, options?: { timeout?: number }) => { + times.push(Date.now()); timeouts.push(options!.timeout!); + throw new Error("DELETE409 pending_cleanup"); + }); + try { + const completed = deleteDaytonaEnvironment({ delete: remove }, "environment", 3_500).catch(error => error); + await vi.advanceTimersByTimeAsync(3_500); + expect(await completed).toMatchObject({ message: expect.stringContaining("Daytona lease cleanup failed") }); + expect(times).toEqual([0, 3_000]); + expect(timeouts).toEqual([3_500, 500]); + await expect(deleteDaytonaEnvironment({ delete: remove }, "environment", 3_500)).rejects.toThrow("cleanup failed"); + expect(remove).toHaveBeenCalledTimes(2); + vi.setSystemTime(0); + const lateSuccess = vi.fn(async () => { vi.setSystemTime(3_500); }); + await expect(deleteDaytonaEnvironment({ delete: lateSuccess }, "environment", 3_500)).rejects.toThrow("deadline expired"); + } finally { vi.useRealTimers(); } + }); + + it("preserves the120s default for callers without an absolute cleanup deadline", async () => { + vi.useFakeTimers(); vi.setSystemTime(0); + const remove = vi.fn(async () => { throw new Error("DELETE409 pending_cleanup"); }); + try { + const completed = deleteDaytonaEnvironment({ delete: remove }, "environment").catch(error => error); + await vi.advanceTimersByTimeAsync(120_000); + expect(await completed).toMatchObject({ message: expect.stringContaining("cleanup failed") }); + expect(remove).toHaveBeenCalledTimes(40); + } finally { vi.useRealTimers(); } + }); +}); diff --git a/tests/runner-e2e/live-fixtures.ts b/tests/runner-e2e/live-fixtures.ts index 29cdb4ff214..8eb9a7d8aeb 100644 --- a/tests/runner-e2e/live-fixtures.ts +++ b/tests/runner-e2e/live-fixtures.ts @@ -12,7 +12,7 @@ import { CHAT_FEATURE_SUITE, CHAT_FEATURE_BUDGET_CENTS } from "./chat-feature-ca import { codexSubscriptionAgentConfig, stageCodexSubscriptionFixture } from "./codex-subscription-fixture.js"; import { caseReceiptTask, CASE_RECEIPT_BUDGET_CENTS, CASE_RECEIPT_RUN_TIMEOUT_MS } from "./case-receipt-cases.js"; import { stageGrokSubscriptionFixture } from "./grok-subscription-fixture.js"; -import type { RunnerApi } from "./api.js"; +import { pollUntil, RunnerApiHttpError, type RunnerApi } from "./api.js"; import type { CredentialName, MatrixExecution, @@ -105,21 +105,28 @@ export function fixtureRuntimeCostCoverage(usage: RunnerE2ERuntimeUsage, fixture return { ...known, costStatus: "unavailable", costSource: "provider_cost_unavailable" }; } -export async function deleteDaytonaEnvironment(api: Pick, environmentId: string) { - const deadlineAt = Date.now() + 120_000; +export async function deleteDaytonaEnvironment(api: Pick, environmentId: string, cleanupDeadlineAt?: number) { + const deadlineAt = cleanupDeadlineAt ?? Date.now() + 120_000; + if (!Number.isFinite(deadlineAt)) throw new Error("Daytona cleanup deadline must be finite"); let lastError: unknown; while (Date.now() < deadlineAt) { + const remaining = deadlineAt - Date.now(); + if (remaining <= 0) break; try { await api.delete( `/api/environments/${environmentId}?destroyReusableSandboxLeases=true`, { allowNotFound: true, + timeout: Math.min(30_000, remaining), }, ); + if (Date.now() >= deadlineAt) throw new Error("Daytona cleanup deadline expired"); return; } catch (error) { lastError = error; - await new Promise((resolve) => setTimeout(resolve, 3_000)); + const remaining = deadlineAt - Date.now(); + if (remaining <= 0) break; + await new Promise((resolve) => setTimeout(resolve, Math.min(3_000, remaining))); } } throw new Error( @@ -135,6 +142,42 @@ export async function installFixtureDaytonaProvider(api: Pick }, { timeout: 120_000 }); } +/** HTTP liveness after a restart does not imply the provider worker is ready. */ +export async function waitForFixtureDaytonaProvider( + api: Pick, + input: { companyId: string; deadlineAt: number }, +): Promise { + const pluginKey = "paperclip.daytona-sandbox-provider"; + const timeout = () => { + const remaining = input.deadlineAt - Date.now(); + if (remaining <= 0) throw new Error("Daytona fixture readiness deadline expired"); + return Math.min(5_000, remaining); + }; + await pollUntil({ + label: "Daytona fixture provider readiness", + deadlineAt: input.deadlineAt, + intervalMs: 250, + retryOnError: error => error instanceof RunnerApiHttpError && [404, 502, 503, 504].includes(error.status), + load: async () => { + const dashboard = await api.get<{ pluginId: string; worker: { status: string } | null }>( + `/api/plugins/${pluginKey}/dashboard`, { timeout: timeout() }, + ); + const capabilities = await api.get<{ sandboxProviders?: Record }>(`/api/companies/${input.companyId}/environments/capabilities`, { timeout: timeout() }); + return { dashboard, provider: capabilities.sandboxProviders?.daytona }; + }, + accept: ({ dashboard, provider }) => dashboard.worker?.status === "running" + && typeof dashboard.pluginId === "string" && dashboard.pluginId.length > 0 + && provider?.status === "supported" && provider.source === "plugin" + && provider.pluginKey === pluginKey && provider.pluginId === dashboard.pluginId + && provider.supportsRunExecution === true, + reject: ({ dashboard, provider }) => provider?.status === "supported" + && (provider.source !== "plugin" || provider.pluginKey !== pluginKey || provider.pluginId !== dashboard.pluginId) + ? "advertised plugin identity does not match the running provider" : undefined, + }); +} + export async function setupLiveFixtures(input: { api: RunnerApi; execution: MatrixExecution; @@ -142,6 +185,7 @@ export async function setupLiveFixtures(input: { workspacePath: string; credentials: Partial>; daytonaImage?: string; + cleanupDeadlineAt?: number; onCompanyCreated?: (companyId: string) => Promise; }): Promise { const { api, execution } = input; @@ -271,7 +315,7 @@ export async function setupLiveFixtures(input: { }, async teardown(environment) { if (execution.environment.id === "daytona") { - await deleteDaytonaEnvironment(api, environment.id); + await deleteDaytonaEnvironment(api, environment.id, input.cleanupDeadlineAt); } }, }); @@ -407,7 +451,7 @@ export async function setupLiveFixtures(input: { || execution.task.flow === "finalization_recovery" || execution.task.flow === "followup_stop_resume" || execution.task.flow === "warm_three_turn" - || (execution.suite.id === "extended-harnesses" && execution.task.id === "file-edit-validate")) { + || (["extended-harnesses", "kimi-qualification"].includes(execution.suite.id) && execution.task.id === "file-edit-validate")) { registry.register({ id: "project", dependencies: ["company", "environment"], diff --git a/tests/runner-e2e/manual-private-cell-report-assets.test.ts b/tests/runner-e2e/manual-private-cell-report-assets.test.ts index e44b126986b..52bb2afb8a4 100644 --- a/tests/runner-e2e/manual-private-cell-report-assets.test.ts +++ b/tests/runner-e2e/manual-private-cell-report-assets.test.ts @@ -19,13 +19,15 @@ vi.mock("node:fs/promises", async original => { }; }); import { copySharedEvidence } from "./shared-evidence.js"; -import { packageEvidence, isStandardEvidencePath } from "./evidence.js"; -import { readManualPrivateHandoff, sealStandardOriginalEvidence, sealManualResultEvidence, type ManualPrivateHandoff } from "./manual-private-handoff.js"; +import { packageEvidence, publishRunnerE2EAttemptEvidence, isStandardEvidencePath } from "./evidence.js"; +import { readManualPrivateHandoff, sealStandardOriginalEvidence, sealManualResultEvidence, preserveOriginalFrameworkResult, type ManualPrivateHandoff } from "./manual-private-handoff.js"; import { canonicalGeneratedReportAsset, cellTraceAsset, inspectCellReportProducer, privateCellEvidenceOwner, readPrivateCellTraceAsset, type CellReportProducerIdentity } from "./manual-private-report-assets.js"; import { validateSealedInventory, validateSealedOriginals } from "./manual-private-transfer.js"; import { MANUAL_PRIVATE_EXECUTIONS, MANUAL_PRIVATE_PROTOCOL, manualCellKind } from "./manual-private-retention.js"; import { archivePrivateEvidence, retainPrivateTransfer } from "./private-history.js"; import { BoundedHistoryPublisher } from "./history-budget.js"; +import { sanitizeJson } from "./redaction.js"; +import type { RunnerE2EResult } from "./types.js"; const hash = (value: string | Buffer) => createHash("sha256").update(value).digest("hex"); const roots: string[] = []; const temporaryRoot = await realpath(os.tmpdir()); @@ -269,6 +271,41 @@ it.each([0, 1])("actual launcher packaging call %s retains the verified private const f = await fixture(); const packaged = await actualCall(f.context, { id: f.id }, f.cell, f.upload, [], { status: "failed" }); for (const name of ["playwright-logo.svg", "manifest.webmanifest", "index.html"]) expect(packaged.files).toContain(`html-report/trace/${name}`); - const publicPackage = await actualCall(undefined, { id: f.id }, f.cell, path.join(f.root, "nonprivate-packaged"), [], { status: "failed" }); - expect(publicPackage.files).not.toContain("html-report/trace/playwright-logo.svg"); +}); + +async function actualLauncherEvidenceBranch(packageCall = packageEvidence) { + const source = await readFile(new URL("./launch.ts", import.meta.url), "utf8"); + const start = source.indexOf(" if (manualPrivateHandoff) {\n await preserveOriginalFrameworkResult(casePrivateDir);"); + const end = source.indexOf(" console.log(", start); + expect(start).toBeGreaterThan(0); expect(end).toBeGreaterThan(start); + const code = stripTypeScriptTypes(`async function actualBranch(manualPrivateHandoff, candidate, casePrivateDir, resultsRoot, uploadDir, credentials, result, resultPath) { ${source.slice(start, end)} return result; }`, { mode: "strip" }); + const publisher = vi.fn(publishRunnerE2EAttemptEvidence); + const branch = Function("preserveOriginalFrameworkResult", "writeFile", "sanitizeJson", "packageEvidence", "privateCellEvidenceOwner", "sealStandardOriginalEvidence", "publishRunnerE2EAttemptEvidence", `${code}; return actualBranch;`)( + preserveOriginalFrameworkResult, writeFile, sanitizeJson, packageCall, privateCellEvidenceOwner, sealStandardOriginalEvidence, publisher, + ) as (context: ManualPrivateHandoff | undefined, candidate: { id: string }, source: string, root: string, target: string, secrets: string[], result: RunnerE2EResult, resultPath: string) => Promise; + return { branch, publisher }; +} +function failedLauncherResult(executionId: string): RunnerE2EResult { + return { schema: "paperclip.runner-e2e.result/v2", executionId, suiteId: "chat-features", profileId: "runner-opencode", environmentId: "local", caseId: "activity.agent-feedback", + attempt: 1, provider: "fixture", model: "fixture", runtimeMode: "native", status: "failed", cleanup: "not_started", failureClass: "candidate_failure", + startedAt: "2026-10-29T00:00:00Z", finishedAt: "2026-10-29T00:00:01Z", durationMs: 1000 }; +} +it("actual launcher public branch uses its normal publisher without retaining private-only trace assets", async () => { + const f = await fixture(), resultsRoot = path.join(f.root, "ordinary-results"), uploadDir = path.join(resultsRoot, "campaign/attempt-1"); + const privatePackage = vi.fn(packageEvidence), { branch, publisher } = await actualLauncherEvidenceBranch(privatePackage); + const result = await branch(undefined, { id: f.id }, f.cell, resultsRoot, uploadDir, [], failedLauncherResult(f.id), path.join(f.cell, "result.json")); + expect(result.status).toBe("failed"); expect(publisher).toHaveBeenCalledTimes(1); expect(privatePackage).not.toHaveBeenCalled(); + const evidence = JSON.parse(await readFile(path.join(uploadDir, "evidence-manifest.json"), "utf8")); + expect(evidence.files).not.toContain("html-report/trace/playwright-logo.svg"); + await expect(readFile(path.join(uploadDir, "html-report/trace/playwright-logo.svg"))).rejects.toMatchObject({ code: "ENOENT" }); + await expect(readFile(path.join(f.cell, "snapshots/manual-original-framework-result.json"))).rejects.toMatchObject({ code: "ENOENT" }); +}); +it("actual launcher private package failure preserves original bytes and never falls back to ordinary publication", async () => { + const f = await fixture(), original = await readFile(path.join(f.cell, "result.json")); + const fail = vi.fn().mockRejectedValue(new Error("synthetic private packaging failure")); + const { branch, publisher } = await actualLauncherEvidenceBranch(fail); + await expect(branch(f.context, { id: f.id }, f.cell, path.join(f.root, "results"), f.upload, [], failedLauncherResult(f.id), path.join(f.cell, "result.json"))).rejects.toThrow("synthetic private packaging failure"); + expect(fail).toHaveBeenCalledTimes(1); expect(publisher).not.toHaveBeenCalled(); + expect(await readFile(path.join(f.cell, "snapshots/manual-original-framework-result.json"))).toEqual(original); + await expect(readFile(path.join(f.upload, "result.json"))).rejects.toMatchObject({ code: "ENOENT" }); }); diff --git a/tests/runner-e2e/manual-private-policy.d.mts b/tests/runner-e2e/manual-private-policy.d.mts index d9dadc7bb8c..778ddfa73ca 100644 --- a/tests/runner-e2e/manual-private-policy.d.mts +++ b/tests/runner-e2e/manual-private-policy.d.mts @@ -1,4 +1,3 @@ -/** Public contract for the credential-free manual evidence authorization script. */ export function manualEvidenceRouting(env: Record): { retention: "recurring" | "actions" | "private"; executionIds: string[]; }; diff --git a/tests/runner-e2e/non-execution-wake.test.ts b/tests/runner-e2e/non-execution-wake.test.ts index 370fc1acaed..5800fe97d45 100644 --- a/tests/runner-e2e/non-execution-wake.test.ts +++ b/tests/runner-e2e/non-execution-wake.test.ts @@ -1,5 +1,7 @@ -import { expect, it } from "vitest"; -import { isBlockedUnstartedWake, isTerminalUnstartedWake } from "./non-execution-wake.js"; +import { createHash } from "node:crypto"; +import { expect, it, vi } from "vitest"; +import { assertChatFlowRunCount, collectChatRunEvidence } from "./chat-flow.js"; +import { chatProviderRuns, classifyPreDispatchExecutionWait, isBlockedUnstartedWake, isTerminalUnstartedWake, verifyChatAdmissionWaitCoverage } from "./non-execution-wake.js"; it("recognizes only explicitly suppressed wakes that never started execution", () => { const run = { status: "cancelled", errorCode: "issue_dependencies_blocked", startedAt: null }; expect(isBlockedUnstartedWake(run)).toBe(true); @@ -21,3 +23,168 @@ it("requires an unstarted terminal suppression on the same terminal task", () => expect(isTerminalUnstartedWake(run, [])).toBe(false); expect(isTerminalUnstartedWake(run, [{ id: "child", status: "in_progress" }])).toBe(false); }); + + +const identity = { + companyId: "10000000-0000-4000-8000-000000000001", agentId: "10000000-0000-4000-8000-000000000002", + issueId: "10000000-0000-4000-8000-000000000003", commentId: "10000000-0000-4000-8000-000000000004", + actionId: "10000000-0000-4000-8000-000000000005", waitId: "10000000-0000-4000-8000-000000000006", + stopId: "10000000-0000-4000-8000-000000000007", wakeId: "10000000-0000-4000-8000-000000000008", +}; +function admissionFixture() { + const row: Record = { + id: identity.waitId, companyId: identity.companyId, agentId: identity.agentId, issueId: identity.issueId, + wakeupRequestId: identity.wakeId, invocationSource: "on_demand", triggerDetail: "manual", responsibleUserId: "board", + status: "cancelled", runtimeMode: "legacy", errorCode: "execution_reconciliation_required", error: "Recovering saved work…", + runnerProfileJson: null, nextEventSeq: 2, logCompressed: false, + createdAt: "2026-10-30T00:00:03Z", finishedAt: "2026-10-30T00:00:04Z", + contextSnapshot: { issueId: identity.issueId, source: "issue.comment", wakeReason: "issue_commented", + wakeCommentId: identity.commentId, wakeCommentIds: [identity.commentId] }, + resultJson: { stopReason: "execution_reconciliation_required", timeoutSource: "stale_queued_run_gate", + timeoutFired: false, timeoutConfigured: false, effectiveTimeoutSec: 0, + executionWait: { issueId: identity.issueId, recoveryActionId: identity.actionId } }, + }; + for (const key of ["startedAt", "runtimeModeResolvedAt", "runtimeModeReason", "runtimeModeResolverVersion", + "runnerInstanceId", "nativeSessionId", "nativeIssueId", "driverKind", "driverVersion", "completionContractId", + "completionContractSha256", "nativePhase", "sessionIdBefore", "sessionIdAfter", "controllerBootId", + "controllerLeaseExpiresAt", "executionStage", "processPid", "processGroupId", "processStartedAt", "logStore", + "logRef", "logBytes", "logSha256", "stdoutExcerpt", "stderrExcerpt", "usageJson", "exitCode", "signal", "externalRunId"]) row[key] = null; + const event: Record = { seq: 1, runId: row.id, companyId: row.companyId, agentId: row.agentId, + eventType: "lifecycle", stream: "system", level: "warn", message: row.error, payload: { ...row.resultJson.executionWait } }; + for (const key of ["sourceInstanceId", "sourceEventId", "sourceSeq", "sourcePayloadSha256", "protocolSchemaVersion"]) event[key] = null; + const stopped = { id: identity.stopId, companyId: identity.companyId, agentId: identity.agentId, + nativeIssueId: identity.issueId, contextSnapshot: { issueId: identity.issueId }, runtimeMode: "native", status: "cancelled", finishedAt: "2026-10-30T00:00:01Z", + resultJson: { nativeCancellation: { schema: "paperclip.native-cancellation.v1", runId: identity.stopId, + companyId: identity.companyId, issueId: identity.issueId, scope: "run", reasonCode: "cancellation_run_only", + dispatchState: "acknowledged", dispatched: true } } }; + const seed = { ...stopped, id: "10000000-0000-4000-8000-000000000009", status: "succeeded", resultJson: {} }; + const reset = { id: "10000000-0000-4000-8000-000000000010", companyId: identity.companyId, agentId: identity.agentId, + runtimeMode: "legacy", status: "succeeded", startedAt: "2026-10-30T00:00:05Z", finishedAt: "2026-10-30T00:00:06Z", + contextSnapshot: { conversationReset: true, wakeCommentId: identity.commentId, commentId: identity.commentId, + conversationSessionGeneration: 1 }, resultJson: { conversationReset: true } }; + const fresh = { ...seed, id: "10000000-0000-4000-8000-000000000011", startedAt: "2026-10-30T00:00:07Z", + sessionIdBefore: null, contextSnapshot: { issueId: identity.issueId, conversationSessionGeneration: 1 } }; + const action = { id: identity.actionId, companyId: identity.companyId, sourceIssueId: identity.issueId, + status: "resolved", outcome: "cancelled", resolvedAt: "2026-10-30T00:00:05Z", fingerprint: `native-workspace:${identity.stopId}`, evidence: { runId: identity.stopId, + automaticRecovery: { policy: "preserve_without_replay_v1", replay: "explicit_user_continuation", runId: identity.stopId }, + explicitUserContinuation: { previousRunId: identity.stopId, commentId: identity.commentId, actorId: "board", + runId: reset.id, queuedCommentRequestId: identity.wakeId, commentBodyHash: createHash("sha256").update("/new").digest("hex") }, + systemRecovery: { schema: "paperclip.system-recovery.v1", sourceRunId: identity.stopId, state: "complete", + phase: "reconcile_effects", preservationOperationIds: ["10000000-0000-4000-8000-000000000012"] }, + workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: identity.stopId, status: "recovered" } } }; + return { row, event, final: { runs: [seed, stopped, row, reset, fresh], + runEvidence: [{ runId: row.id, log: null, logOmissionReason: "pre_dispatch_execution_wait", events: [event] }], + companyId: identity.companyId, agentId: identity.agentId, stoppedRunId: identity.stopId, + issue: { id: identity.issueId, companyId: identity.companyId, assigneeAgentId: identity.agentId, + conversationSessionGeneration: 1, conversationBoundaryCommentId: identity.commentId }, + comments: [{ id: identity.commentId, companyId: identity.companyId, issueId: identity.issueId, body: "/new", + authorType: "user", deletedAt: null, authorUserId: "board", authorAgentId: null, createdByRunId: null, createdAt: "2026-10-30T00:00:02Z" }], + recoverySources: [{ runId: identity.stopId, recoveryActionId: identity.actionId, response: { actions: [action], + sourceInspection: { schema: "paperclip.workspace-source-inspection.v1", actionId: identity.actionId, + runId: identity.stopId, scope: "recorded_source_identity", status: "verified" } } }] } }; +} + +it("counts only exact pre-dispatch waits provisionally without requiring a future reset", () => { + const { row, event } = admissionFixture(); + const receipt = classifyPreDispatchExecutionWait(row, [event]); + expect(receipt).toMatchObject({ runId: row.id, kind: "pre_dispatch_execution_wait" }); + expect(chatProviderRuns([{ id: "provider" }, row as any], [receipt!])).toEqual([{ id: "provider" }]); + expect(row.status).toBe("cancelled"); +}); + +it("requires complete receipt and rejects provider authority rather than filtering cancelled or legacy rows", () => { + for (const patch of [{ runtimeMode: "native" }, { status: "failed" }, { errorCode: "operator_cancelled" }, + { startedAt: "2026-10-30" }, { startedAt: undefined }, { processPid: 12 }, { controllerBootId: identity.wakeId }, + { runnerProfileJson: { nativeExecutionInput: {} } }, { nativeSessionId: identity.wakeId }, { logRef: "log" }, + { nextEventSeq: 3 }, { wakeupRequestId: null }, { resultJson: { stopReason: "execution_reconciliation_required" } }]) { + const { row, event } = admissionFixture(); + expect(classifyPreDispatchExecutionWait({ ...row, ...patch }, [event]), JSON.stringify(patch)).toBeNull(); + } + for (const patch of [{ stream: "stdout" }, { sourceEventId: "native-event" }, { seq: 2 }, { runId: identity.stopId }, + { companyId: identity.stopId }, { eventType: "provider.event" }, { payload: { issueId: identity.issueId, recoveryActionId: identity.stopId } }]) { + const { row, event } = admissionFixture(); + expect(classifyPreDispatchExecutionWait(row, [{ ...event, ...patch }]), JSON.stringify(patch)).toBeNull(); + } + const { row, event } = admissionFixture(); + expect(classifyPreDispatchExecutionWait(row, [])).toBeNull(); + expect(classifyPreDispatchExecutionWait(row, [event, { ...event, seq: 2, eventType: "provider.event" }])).toBeNull(); +}); + +it("requires preserved source, saved human reset and all three actual native providers at final grading", () => { + const { final } = admissionFixture(); + const original = structuredClone(final); + const waits = verifyChatAdmissionWaitCoverage(final); + expect(waits).toHaveLength(1); + expect(chatProviderRuns(final.runs as any, waits)).toHaveLength(3); + expect(final.runs).toHaveLength(5); + expect(final).toEqual(original); +}); + +it("rejects incomplete, cross-scope or unrecovered final proof without treating an admission wait as a reset", () => { + const mutations: Array<(f: any) => void> = [ + f => { f.comments[0].body = "/new unrelated"; }, + f => { f.comments[0].authorType = "system"; }, f => { f.comments[0].deletedAt = "2026-10-30T00:00:03Z"; }, f => { f.comments[0].authorAgentId = identity.agentId; }, + f => { f.comments[0].createdByRunId = identity.stopId; }, f => { f.comments[0].issueId = identity.stopId; }, + f => { f.comments[0].authorUserId = "other"; }, f => { f.stoppedRunId = identity.waitId; }, + f => { f.runs[1].resultJson.nativeCancellation.dispatchState = "pending"; }, + f => { f.runs[1].resultJson.nativeCancellation.runId = identity.waitId; }, + f => { f.runs[3].contextSnapshot.wakeCommentId = identity.wakeId; }, f => { f.runs[3].status = "cancelled"; }, + f => { f.runs[4].contextSnapshot.conversationSessionGeneration = 0; }, f => { f.runs[4].runtimeMode = "legacy"; }, + f => { f.runs[4].sessionIdBefore = "old"; }, f => { f.runs.push({ ...f.runs[4], id: identity.wakeId }); }, + f => { f.runs.push(structuredClone(f.runs[2])); }, f => { f.runEvidence[0].log = {}; }, + f => { f.recoverySources = []; }, f => { f.recoverySources[0].runId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].companyId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].sourceIssueId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].evidence.runId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].evidence.systemRecovery.state = "awaiting_evidence"; }, + f => { f.recoverySources[0].response.actions[0].evidence.automaticRecovery.replay = "blocked"; }, + f => { f.recoverySources[0].response.actions[0].evidence.explicitUserContinuation.runId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].evidence.explicitUserContinuation.queuedCommentRequestId = identity.waitId; }, + f => { f.recoverySources[0].response.actions[0].evidence.explicitUserContinuation.commentBodyHash = "0".repeat(64); }, + f => { f.recoverySources[0].response.actions[0].evidence.systemRecovery.preservationOperationIds = []; }, + f => { f.recoverySources[0].response.actions[0].evidence.workspaceRecovery.status = "blocked"; }, + f => { f.recoverySources[0].response.sourceInspection.status = "blocked"; }, + f => { f.recoverySources[0].response.sourceInspection.actionId = identity.waitId; }, + ]; + for (const mutate of mutations) { + const { final } = admissionFixture(); mutate(final); + expect(() => verifyChatAdmissionWaitCoverage(final)).toThrow("Chat admission wait:"); + } +}); + + +it("keeps the actual chat provider-count assertion at three and preserves raw admission rows", () => { + const { final } = admissionFixture(); + const wait = verifyChatAdmissionWaitCoverage(final); + expect(() => assertChatFlowRunCount({ suiteId: "kimi-qualification", task: { expectedRunCount: 3 }, + runs: final.runs.filter(row => row.runtimeMode !== "legacy" || row.status === "cancelled") as any })).toThrow("Declared total provider-run bounds"); + expect(() => assertChatFlowRunCount({ suiteId: "kimi-qualification", task: { expectedRunCount: 3 }, + runs: chatProviderRuns(final.runs as any, wait) })).not.toThrow(); + expect(final.runs).toHaveLength(5); +}); + +it("captures the actual complete public wait stream without requesting a nonexistent provider log", async () => { + const { row, event } = admissionFixture(); + const get = vi.fn(async (path: string) => { + if (path.includes("/events?")) return [event]; + throw new Error("provider log must not be omitted for unverified evidence"); + }); + expect(await collectChatRunEvidence({ get } as any, row as any, true)).toEqual({ + runId: row.id, log: null, logOmissionReason: "pre_dispatch_execution_wait", events: [event], + }); + expect(get).toHaveBeenCalledTimes(1); + await expect(collectChatRunEvidence({ get } as any, { ...row, processPid: 99 } as any, true)).rejects.toThrow("provider log must not be omitted"); + await expect(collectChatRunEvidence({ get } as any, row as any)).rejects.toThrow("provider log must not be omitted"); +}); + + +it("permits only a predecessor-correlated historical before-session hint at final grading", () => { + const { row, event, final } = admissionFixture(); + row.sessionIdBefore = "session_historical"; + (final.runs[0] as Record).sessionIdAfter = "session_historical"; + (final.runs[1] as Record).sessionIdBefore = "session_historical"; + expect(classifyPreDispatchExecutionWait(row, [event])).toBeTruthy(); + expect(verifyChatAdmissionWaitCoverage(final)).toHaveLength(1); + row.sessionIdBefore = "session_foreign"; + expect(() => verifyChatAdmissionWaitCoverage(final)).toThrow("historical_session_hint"); +}); diff --git a/tests/runner-e2e/non-execution-wake.ts b/tests/runner-e2e/non-execution-wake.ts index 4518ff391ac..48d02ccf5ea 100644 --- a/tests/runner-e2e/non-execution-wake.ts +++ b/tests/runner-e2e/non-execution-wake.ts @@ -1,3 +1,5 @@ +import { createHash } from "node:crypto"; + /** A queued parent wake can be suppressed while its child still runs. This is * not a provider failure: admission never started. Keep it in the evidence. */ export function isBlockedUnstartedWake(run: Record) { @@ -26,3 +28,153 @@ export function isTerminalUnstartedWake( task.id === issueId && ["done", "cancelled"].includes(String(task.status)), ); } + +type Row = Record; +const object = (value: unknown): Row | null => value !== null && typeof value === "object" && !Array.isArray(value) + ? value as Row : null; +const uuid = (value: unknown): value is string => typeof value === "string" && + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i.test(value); +const time = (value: unknown) => typeof value === "string" ? Date.parse(value) : Number.NaN; +const keys = (value: Row, expected: string[]) => Object.keys(value).sort().join(",") === expected.sort().join(","); +const one = (value: unknown, expected: unknown) => Array.isArray(value) && value.length === 1 && value[0] === expected; + +export interface ChatAdmissionWait { + runId: string; + kind: "pre_dispatch_execution_wait"; + companyId: string; + agentId: string; + issueId: string; + recoveryActionId: string; + wakeupRequestId: string; + commentId: string; +} + +export function classifyPreDispatchExecutionWait(run: unknown, events: unknown): ChatAdmissionWait | null { + const row = object(run), context = object(row?.contextSnapshot), result = object(row?.resultJson); + const wait = object(result?.executionWait); + if (!row || !context || !result || !wait || !Array.isArray(events) || events.length !== 1) return null; + for (const field of ["id", "companyId", "agentId", "wakeupRequestId"]) if (!uuid(row[field])) return null; + if (!uuid(context.issueId) || !uuid(context.wakeCommentId) || !uuid(wait.recoveryActionId) || + wait.issueId !== context.issueId || row.issueId !== context.issueId || + context.source !== "issue.comment" || context.wakeReason !== "issue_commented" || + !one(context.wakeCommentIds, context.wakeCommentId)) return null; + if (row.runtimeMode !== "legacy" || row.status !== "cancelled" || + row.invocationSource !== "on_demand" || row.triggerDetail !== "manual" || + row.errorCode !== "execution_reconciliation_required" || row.runnerProfileJson !== null || + !keys(result, ["stopReason", "timeoutSource", "timeoutFired", "timeoutConfigured", "effectiveTimeoutSec", "executionWait"]) || + !keys(wait, ["issueId", "recoveryActionId"]) || result.stopReason !== row.errorCode || + result.timeoutSource !== "stale_queued_run_gate" || result.timeoutFired !== false || + result.timeoutConfigured !== false || result.effectiveTimeoutSec !== 0 || + row.nextEventSeq !== 2 || row.logCompressed !== false || + !(row.sessionIdBefore === null || (typeof row.sessionIdBefore === "string" && row.sessionIdBefore.length > 0)) || + !Number.isFinite(time(row.createdAt)) || !Number.isFinite(time(row.finishedAt)) || + time(row.finishedAt) < time(row.createdAt)) return null; + for (const field of ["startedAt", "runtimeModeResolvedAt", "runtimeModeReason", "runtimeModeResolverVersion", + "runnerInstanceId", "nativeSessionId", "nativeIssueId", "driverKind", "driverVersion", "completionContractId", + "completionContractSha256", "nativePhase", "sessionIdAfter", "controllerBootId", + "controllerLeaseExpiresAt", "executionStage", "processPid", "processGroupId", "processStartedAt", "logStore", + "logRef", "logBytes", "logSha256", "stdoutExcerpt", "stderrExcerpt", "usageJson", "exitCode", "signal", "externalRunId"]) + if (row[field] !== null) return null; + const event = object(events[0]), payload = object(event?.payload); + if (!event || !payload || event.seq !== 1 || event.runId !== row.id || event.companyId !== row.companyId || + event.agentId !== row.agentId || event.eventType !== "lifecycle" || event.stream !== "system" || event.level !== "warn" || + typeof row.error !== "string" || !row.error || event.message !== row.error || + !keys(payload, ["issueId", "recoveryActionId"]) || payload.issueId !== wait.issueId || + payload.recoveryActionId !== wait.recoveryActionId) return null; + for (const field of ["sourceInstanceId", "sourceEventId", "sourceSeq", "sourcePayloadSha256", "protocolSchemaVersion"]) + if (event[field] !== null) return null; + return { runId: String(row.id), kind: "pre_dispatch_execution_wait", companyId: String(row.companyId), + agentId: String(row.agentId), issueId: context.issueId, recoveryActionId: wait.recoveryActionId, + wakeupRequestId: String(row.wakeupRequestId), commentId: context.wakeCommentId }; +} + +export function chatProviderRuns; resultJson?: Record }>( + runs: T[], admissionWaits: readonly ChatAdmissionWait[] = [], +) { + const waiting = new Set(admissionWaits.map(wait => wait.runId)); + return runs.filter(run => run.contextSnapshot?.conversationReset !== true && + run.resultJson?.conversationReset !== true && !waiting.has(run.id)); +} + +export function verifyChatAdmissionWaitCoverage(input: { + runs: unknown[]; runEvidence: unknown[]; comments: unknown[]; issue: unknown; + companyId: string; agentId: string; stoppedRunId: string | undefined; + recoverySources: Array<{ runId: string; recoveryActionId: string; response: unknown }>; +}): ChatAdmissionWait[] { + function require(condition: unknown, code: string): asserts condition { + if (!condition) throw new Error(`Chat admission wait: ${code}`); + } + const rows = input.runs.map(object), issue = object(input.issue); + require(rows.every(Boolean) && issue, "missing_rows"); + require(new Set(rows.map(row => row!.id)).size === rows.length, "duplicate_rows"); + const candidates = rows.filter(row => row!.errorCode === "execution_reconciliation_required"); + require(candidates.length <= 1, "repeated_admission"); + if (!candidates.length) return []; + require(issue.companyId === input.companyId && issue.assigneeAgentId === input.agentId && + issue.conversationSessionGeneration === 1, "issue_scope"); + const stopped = rows.find(row => row!.id === input.stoppedRunId), cancellation = object(object(stopped?.resultJson)?.nativeCancellation); + require(stopped && stopped.runtimeMode === "native" && stopped.status === "cancelled" && + stopped.companyId === input.companyId && stopped.agentId === input.agentId && stopped.nativeIssueId === issue.id && + cancellation?.schema === "paperclip.native-cancellation.v1" && cancellation.runId === stopped.id && + cancellation.companyId === input.companyId && cancellation.issueId === issue.id && cancellation.scope === "run" && + cancellation.reasonCode === "cancellation_run_only" && cancellation.dispatchState === "acknowledged" && + cancellation.dispatched === true, "stopped_native_source"); + const waits: ChatAdmissionWait[] = []; + for (const row of candidates) { + const evidence = input.runEvidence.map(object).filter(value => value?.runId === row!.id); + require(evidence.length === 1 && evidence[0]!.log === null && + evidence[0]!.logOmissionReason === "pre_dispatch_execution_wait", "complete_event_receipt"); + const wait = classifyPreDispatchExecutionWait(row, evidence[0]!.events); + require(wait && wait.companyId === input.companyId && wait.agentId === input.agentId && wait.issueId === issue.id, "wait_scope"); + const comment = input.comments.map(object).filter(value => value?.id === wait.commentId); + require(comment.length === 1 && comment[0]!.companyId === input.companyId && comment[0]!.issueId === issue.id && + comment[0]!.body === "/new" && comment[0]!.authorType === "user" && comment[0]!.deletedAt === null && + typeof comment[0]!.authorUserId === "string" && comment[0]!.authorUserId && + comment[0]!.authorAgentId === null && comment[0]!.createdByRunId === null && + row!.responsibleUserId === comment[0]!.authorUserId && time(stopped.finishedAt) < time(comment[0]!.createdAt) && + time(comment[0]!.createdAt) <= time(row!.createdAt), "saved_human_new"); + if (row!.sessionIdBefore !== null) { + const seed = rows.filter(value => value!.runtimeMode === "native" && value!.status === "succeeded" && + object(value!.contextSnapshot)?.conversationSessionGeneration !== 1); + require(seed.length === 1 && seed[0]!.sessionIdAfter === row!.sessionIdBefore && + stopped.sessionIdBefore === row!.sessionIdBefore, "historical_session_hint"); + } + const reset = rows.filter(value => object(value!.contextSnapshot)?.conversationReset === true); + require(reset.length === 1, "single_authored_reset"); + const resetContext = object(reset[0]!.contextSnapshot)!; + require(reset[0]!.status === "succeeded" && resetContext.wakeCommentId === wait.commentId && + resetContext.commentId === wait.commentId && resetContext.conversationSessionGeneration === 1 && + object(reset[0]!.resultJson)?.conversationReset === true && issue.conversationBoundaryCommentId === wait.commentId && + time(row!.finishedAt) <= time(reset[0]!.startedAt), "same_saved_reset"); + const sources = input.recoverySources.filter(value => value.runId === stopped.id && value.recoveryActionId === wait.recoveryActionId); + require(sources.length === 1, "selected_recovery"); + const response = object(sources[0]!.response), actions = response?.actions; + require(Array.isArray(actions) && actions.length === 1, "exact_recovery_action"); + const action = object(actions[0]), facts = object(action?.evidence), recovery = object(facts?.systemRecovery), + preservation = object(facts?.workspaceRecovery), automatic = object(facts?.automaticRecovery), + authorization = object(facts?.explicitUserContinuation), inspection = object(response?.sourceInspection); + require(action?.id === wait.recoveryActionId && action.companyId === input.companyId && action.sourceIssueId === issue.id && + action.status === "resolved" && action.outcome === "cancelled" && Number.isFinite(time(action.resolvedAt)) && + action.fingerprint === `native-workspace:${stopped.id}` && facts?.runId === stopped.id && + automatic?.policy === "preserve_without_replay_v1" && automatic.replay === "explicit_user_continuation" && automatic.runId === stopped.id && + authorization && authorization.previousRunId === stopped.id && authorization.commentId === wait.commentId && + authorization.actorId === comment[0]!.authorUserId && authorization.runId === reset[0]!.id && + authorization.queuedCommentRequestId === wait.wakeupRequestId && + authorization.commentBodyHash === createHash("sha256").update("/new").digest("hex") && + recovery?.schema === "paperclip.system-recovery.v1" && recovery.sourceRunId === stopped.id && recovery.state === "complete" && + recovery.phase === "reconcile_effects" && Array.isArray(recovery.preservationOperationIds) && recovery.preservationOperationIds.length > 0 && + recovery.preservationOperationIds.every(uuid) && new Set(recovery.preservationOperationIds).size === recovery.preservationOperationIds.length && + preservation?.schema === "paperclip.workspace-recovery.v1" && preservation.runId === stopped.id && preservation.status === "recovered" && + inspection?.schema === "paperclip.workspace-source-inspection.v1" && inspection.actionId === wait.recoveryActionId && + inspection.runId === stopped.id && inspection.scope === "recorded_source_identity" && inspection.status === "verified", "preserved_native_source"); + waits.push(wait); + } + const providers = chatProviderRuns(rows as Array, waits); + require(providers.length === 3 && providers.every(row => row.runtimeMode === "native" && row.companyId === input.companyId && row.agentId === input.agentId && + row.nativeIssueId === issue.id && object(row.contextSnapshot)?.issueId === issue.id) && + providers.filter(row => row.status === "cancelled").length === 1 && providers.filter(row => row.status === "succeeded").length === 2, "three_native_providers"); + const fresh = providers.filter(row => row.status === "succeeded" && object(row.contextSnapshot)?.conversationSessionGeneration === 1); + require(fresh.length === 1 && fresh[0]!.sessionIdBefore === null && + time(fresh[0]!.startedAt) >= time(rows.find(row => object(row!.contextSnapshot)?.conversationReset === true)!.finishedAt), "fresh_native_after_reset"); + return waits; +} diff --git a/tests/runner-e2e/playwright-support.config.ts b/tests/runner-e2e/playwright-support.config.ts index 8f7b252da09..000c7d502c6 100644 --- a/tests/runner-e2e/playwright-support.config.ts +++ b/tests/runner-e2e/playwright-support.config.ts @@ -3,7 +3,7 @@ import { defineConfig } from "@playwright/test"; /** Browser-only harness regressions: no Paperclip instance or provider credentials. */ export default defineConfig({ testDir: ".", - testMatch: ["clarification-readback.spec.ts", "project-context-readback.spec.ts", "skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts", "task-search-filter.spec.ts"], + testMatch: ["clarification-readback.spec.ts", "project-context-readback.spec.ts", "skill-studio-flow.spec.ts", "native-task-reply.spec.ts", "screenshot-readiness.spec.ts", "service-worker-reload.spec.ts", "lost-send.spec.ts", "chat-restart.spec.ts", "settings-toggle.spec.ts", "browser-bootstrap-diagnostics.spec.ts", "service-worker-module-loading.spec.ts", "browser-bootstrap-recovery.spec.ts", "blocker-input.spec.ts", "native-blocker-visible.spec.ts", "copilot-permission-card.spec.ts", "kimi-permission-card.spec.ts", "github-review-check.spec.ts", "antigravity-journey-browser.spec.ts", "task-search-filter.spec.ts"], workers: 1, retries: 0, timeout: 10_000, diff --git a/tests/runner-e2e/prerequisites.ts b/tests/runner-e2e/prerequisites.ts index e9d5430a15e..807e568c1a6 100644 --- a/tests/runner-e2e/prerequisites.ts +++ b/tests/runner-e2e/prerequisites.ts @@ -1,4 +1,5 @@ import type { MatrixExecution } from "./types.js"; +import { supportsKimiLocalProviderFault } from "./kimi-local-provider-fault.js"; /** Explicit-only profiles blocked until complete identity/auth/session/billing qualification. */ export const PENDING_PROFILE_PREREQUISITES = { @@ -17,7 +18,7 @@ export function assertRemoteNativeEvidencePrerequisites( environment: NodeJS.ProcessEnv, ): void { if (!executions.some(execution => execution.environment.id === "daytona" - && ["cursor_native", "pi_native", "pi_controls", "copilot_protection", "native_active_stop", "native_provider_loss"].includes(execution.task.flow))) return; + && ["cursor_native", "pi_native", "pi_controls", "copilot_protection", "native_active_stop", "native_provider_loss", "kimi_native_lifecycle"].includes(execution.task.flow))) return; const names = ["PAPERCLIP_E2E_DAYTONA_NODE_SHA256", "PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256"]; const invalid = names.filter(name => !/^sha256:[a-f0-9]{64}$/u.test(environment[name] ?? "")); if (invalid.length) throw new Error(`Native Daytona evidence requires exact image executable digests: ${invalid.join(", ")}`); @@ -25,7 +26,16 @@ export function assertRemoteNativeEvidencePrerequisites( export function assertRunnerE2EPrerequisites( executions: readonly MatrixExecution[], + host: Pick = process, ): void { + if (!supportsKimiLocalProviderFault(host) && executions.some(execution => + execution.environment.id === "local" && execution.profile.qualificationCandidate === "kimi" + && execution.task.id === "native-write-provider-loss")) { + throw new Error( + `Local native Kimi provider-loss qualification requires darwin-arm64; selected host is ${host.platform}-${host.arch}. ` + + "Use its qualified local host or the Linux Daytona cell. No provider credentials were loaded or sent, and no execution resources were acquired.", + ); + } const blocked = [...new Set( executions .map((execution) => execution.profile.id) diff --git a/tests/runner-e2e/remote-native-fixtures.test.ts b/tests/runner-e2e/remote-native-fixtures.test.ts index 0209318959f..ca5b8cebccf 100644 --- a/tests/runner-e2e/remote-native-fixtures.test.ts +++ b/tests/runner-e2e/remote-native-fixtures.test.ts @@ -12,12 +12,17 @@ import { REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS, bindRemoteNativeFixture, validatePi import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js"; import { createRemoteNativeBootstrap } from "./remote-native-bootstrap.js"; +import { kimiFaultRuntimeContract } from "./kimi-provider-fault-contract.js"; const hash = (s: string) => `sha256:${createHash("sha256").update(s).digest("hex")}`; const bootId = "12345678-1234-1234-1234-123456789abc"; const authority = { companyId: "company", environmentId: "environment", runId: "run", leaseId: "lease", sandboxId: "sandbox", image: `registry/image@sha256:${"a".repeat(64)}` }; const binding = { ...authority, remoteCwd: "/workspace" }; const root = { pid: 21, ppid: 1, startTicks: "100", bootId }; +function kimiWorker(overrides: Record = {}) { + return { schema: "paperclip.kimi11.remote-fault-diagnostic.v1", qualification: false, status: "FAIL", stage: "inspect_before", failingGuard: "parent_bootstrap", errno: null, + signalAttempted: false, signalDispatched: false, receipt: null, externalModelCalls: 0, ...overrides }; +} function snapshot(): RemoteNativeSnapshot { return { binding, observedAtMs: 1000, observedMonotonicNs: "10000", receivedAtMs: 1000, complete: true, workspace: { "existing.txt": hash("original") }, targets: { "result.txt": { absent: true, sha256: null, parent: { dev: "1", ino: "2" }, mutationCount: 0, complete: true } }, @@ -68,6 +73,10 @@ describe("remote process exit during proc reads", () => { }); describe("remote native lease admission", () => { + it("keeps the actual observer installation command below the native Linux argv string bound", async () => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options), install = h.calls.find(call => call.request.op === "install"); + expect(install).toBeDefined(); expect(Buffer.byteLength(install!.command)).toBeLessThan(131072); await fixture.close(); + }); it.each(["companyId", "environmentId", "heartbeatRunId", "providerLeaseId", "provider", "status"])("rejects wrong %s before executing any remote command", async key => { const h = harness(); h.setLease({ ...h.lease(), [key]: "foreign" }); await expect(bindRemoteNativeFixture(h.options)).rejects.toThrow("lease_scope"); expect(h.executeCommand).not.toHaveBeenCalled(); @@ -380,6 +389,61 @@ describe("remote native lease admission", () => { }); }); +describe("closed bootstrap timeout diagnostics", () => { + it("reports the unchanged 10s initial lease-readmission deadline without remote work", async () => { + vi.useFakeTimers(); vi.setSystemTime(1_000_000); + try { + const h = harness(); h.apiGet.mockImplementation(() => new Promise(() => {})); + let settled = false; + const outcome = bindRemoteNativeFixture(h.options).catch(error => { settled = true; return error; }); + await vi.advanceTimersByTimeAsync(9999); expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(1); const error = await outcome; + expect(error.message).toBe("remote_native_fixture:remote_command_failed_or_deadline"); + expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "lease-readmission", code: "rpc_deadline" }]); + expect(h.get).not.toHaveBeenCalled(); expect(h.executeCommand).not.toHaveBeenCalled(); + } finally { vi.useRealTimers(); } + }); + it("reports the unchanged 12s runtime-ready deadline without installing or publishing", async () => { + vi.useFakeTimers(); vi.setSystemTime(1_000_000); + try { + const h = harness(); h.override(request => request.op === "runtime-ready" ? new Promise(() => {}) : undefined); + let settled = false; + const outcome = bindRemoteNativeFixture(h.options).catch(error => { settled = true; return error; }); + await vi.advanceTimersByTimeAsync(11999); expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(1); const error = await outcome; + expect(error.message).toBe("remote_native_fixture:remote_command_failed_or_deadline"); + expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "runtime-ready", code: "rpc_deadline" }]); + expect(h.calls.map(call => call.request.op)).toEqual(["runtime-ready"]); + } finally { vi.useRealTimers(); } + }); + it("keeps a timed-out Kimi fault signal-unknown and consumes its one-shot capability", async () => { + vi.useFakeTimers(); vi.setSystemTime(1_000_000); + try { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); + await fixture.publishAction("action.txt", "action"); + h.override(request => request.op === "kimi-provider-death" ? new Promise(() => {}) : undefined); + let settled = false; + const outcome = fixture.terminateKimiProvider!("workspace-id").catch(error => { settled = true; return error; }); + await vi.advanceTimersByTimeAsync(11999); expect(settled).toBe(false); + await vi.advanceTimersByTimeAsync(1); const error = await outcome; + expect(error.message).toBe("remote_native_fixture:remote_command_failed_or_deadline"); + expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "kimi-provider-death", code: "rpc_deadline", + fault: { stage: "worker_response", code: "invalid_response", signalDispatched: null } }]); + await expect(fixture.terminateKimiProvider!("workspace-id")).rejects.toThrow("kimi_fault_admission"); + expect(h.calls.filter(call => call.request.op === "kimi-provider-death")).toHaveLength(1); + await fixture.close(); h.rejectTerminal(new Error("late synthetic rejection")); await Promise.resolve(); + } finally { vi.useRealTimers(); } + }); + it("keeps an SDK rejection distinct from a deadline without retaining its private text", async () => { + const h = harness(); h.executeCommand.mockRejectedValueOnce(new Error("PRIVATE SDK rejection")); + const error = await bindRemoteNativeFixture(h.options).catch(error => error); + expect(error.message).toBe("remote_native_fixture:remote_command_failed_or_deadline"); + expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "runtime-ready", code: "transport_failure" }]); + expect(JSON.stringify(remoteNativeFixtureDiagnostics(error))).not.toContain("PRIVATE"); + expect(h.executeCommand).toHaveBeenCalledTimes(1); + }); +}); + describe("cell deadline and cleanup bounds", () => { it("expires finish with 15s reserved, bounds SDK/socket/observer clocks, and handles late SDK failure", async () => { vi.useFakeTimers(); vi.setSystemTime(1_000_000); @@ -576,6 +640,53 @@ describe("actual generated observer state machine", () => { const reply = o.request("snapshot")[0]; expect(reply.ok && reply.result.complete).toBe(false); }); + it("enforces Kimi observer authority before the exact one-shot native-child fault", async () => { + const o = await observerHarness(), contract = kimiFaultRuntimeContract("linux-x64"); o.request("snapshot"); + const request = { runtimeEnvironmentLeaseId: "workspace-id", contract, bootstrapTemplate: "reviewed bootstrap", guardTemplate: "reviewed guard" }; + o.faultSpawn.mockReturnValue({ status: 0, stdout: JSON.stringify(kimiWorker({ status: "PASS", stage: "complete", failingGuard: null, signalAttempted: true, signalDispatched: true, receipt: piFaultReceipt() })) }); + expect(o.request("kimi-provider-death", request)[0]).toEqual({ ok: false, error: "observer_evidence_incomplete", diagnostic: { stage: "observer_admission", code: "observer_authority", signalDispatched: false } }); expect(o.faultSpawn).not.toHaveBeenCalled(); + o.request("publish", { path: "action.txt", text: "Use the native read tool" }); + expect(o.request("kimi-provider-death", { ...request, runtimeEnvironmentLeaseId: "foreign" })[0].ok).toBe(false); + expect(o.faultSpawn).not.toHaveBeenCalled(); + expect(o.request("kimi-provider-death", request)[0].ok).toBe(true); + const [program, args, options] = o.faultSpawn.mock.calls[0] as unknown as [string, string[], Record]; + expect(program).toBe("/usr/bin/python3"); expect(args.slice(0, 2)).toEqual(["-I", "-c"]); + expect(args[2]).toContain("signal.pidfd_send_signal"); expect(Buffer.from(args[3]!, "base64").toString()).toContain("kimi_provider_identity_or_signal_failed"); + expect(JSON.parse(Buffer.from(args[4]!, "base64").toString())).toEqual({ root, binding, runtimeEnvironmentLeaseId: "workspace-id", runnerdSha256: hash("runnerd"), closureSha256: contract.closureSha256, nodeSha256: contract.nodeSha256, guardLoaderSha256: contract.guardLoaderSha256, bootstrapTemplate: request.bootstrapTemplate, guardTemplate: request.guardTemplate }); + expect(options.env).toEqual({ PATH: "/usr/bin:/bin" }); expect(o.context.process.kill).not.toHaveBeenCalled(); + expect(o.request("kimi-provider-death", request)[0].ok).toBe(false); expect(o.faultSpawn).toHaveBeenCalledOnce(); + }); + it("records no signal before a rejected observer nonce without executing the worker", async () => { + const o = await observerHarness(); o.request("snapshot"); o.request("publish", { path: "action.txt", text: "action" }); + expect(o.request("kimi-provider-death", { nonce: "foreign" })[0]).toEqual({ ok: false, error: "observer_evidence_incomplete", diagnostic: { stage: "observer_admission", code: "observer_authority", signalDispatched: false } }); expect(o.faultSpawn).not.toHaveBeenCalled(); expect(o.context.process.kill).not.toHaveBeenCalled(); + }); + it.each([false, null, true])("retains a closed failed worker acknowledgement %s without retry", async signalDispatched => { + const o = await observerHarness(); o.request("snapshot"); o.request("publish", { path: "action.txt", text: "action" }); + o.faultSpawn.mockReturnValue({ status: 1, stdout: JSON.stringify(kimiWorker({ stage: signalDispatched === false ? "inspect_before" : signalDispatched === null ? "signal" : "receipt", failingGuard: signalDispatched === false ? "parent_bootstrap" : null, signalAttempted: signalDispatched !== false, signalDispatched })) }); + const reply = o.request("kimi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id", contract: kimiFaultRuntimeContract("linux-x64"), bootstrapTemplate: "reviewed", guardTemplate: "reviewed" })[0]; + expect(reply).toEqual({ ok: false, error: "observer_evidence_incomplete", diagnostic: { stage: signalDispatched === false ? "inspect_before" : signalDispatched === null ? "signal" : "receipt", code: signalDispatched === false ? "parent_bootstrap" : "worker_failure", signalDispatched } }); + expect(JSON.stringify(reply)).not.toContain("private"); + expect(o.request("kimi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id" })[0].ok).toBe(false); expect(o.faultSpawn).toHaveBeenCalledOnce(); + }); + it.each([ + { status: null, signal: "SIGKILL", stdout: "" }, + { status: 1, stdout: "private exception text" }, + { status: 1, stdout: "null" }, + { status: 1, stdout: JSON.stringify(kimiWorker({ status: "PASS", stage: "complete", failingGuard: null, signalAttempted: true, signalDispatched: true })) }, + { status: 0, stdout: JSON.stringify(kimiWorker()) }, + { status: 1, stdout: JSON.stringify(kimiWorker({ signalAttempted: true })) }, + { status: null, signal: "SIGKILL", stdout: JSON.stringify(kimiWorker({ status: "PASS", stage: "complete", failingGuard: null, signalAttempted: true, signalDispatched: true })) }, + { status: 1, stdout: JSON.stringify(kimiWorker({ stage: "private_stage", failingGuard: "private_path" })) }, + { status: 0, stdout: JSON.stringify(kimiWorker({ status: "PASS", stage: "complete", failingGuard: null })) }, + { status: 1, stdout: JSON.stringify(kimiWorker({ stage: "terminate", signalAttempted: true, signalDispatched: true })) }, + { status: 1, stdout: JSON.stringify(kimiWorker({ private: "secret/path" })) }, + { status: 1, error: new Error("private spawn error"), stdout: JSON.stringify(kimiWorker()) }, + { status: 1, signal: "SIGKILL", stdout: JSON.stringify(kimiWorker()) }, + ])("keeps an unproven worker outcome unknown", async outcome => { + const o = await observerHarness(); o.request("snapshot"); o.request("publish", { path: "action.txt", text: "action" }); o.faultSpawn.mockReturnValue(outcome as never); + const reply = o.request("kimi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id", contract: kimiFaultRuntimeContract("linux-x64"), bootstrapTemplate: "reviewed", guardTemplate: "reviewed" })[0]; + expect(reply.ok).toBe(false); expect(reply.diagnostic.signalDispatched).toBeNull(); expect(JSON.stringify(reply)).not.toContain("private"); expect(o.context.process.kill).not.toHaveBeenCalled(); + }); it("executes the actual observer fault phase once with a closed environment and exact identity", async () => { for (const published of [false, true]) { const rejected = await observerHarness(); @@ -932,3 +1043,66 @@ describe("exact Pi-child fault capability", () => { await fixture.close(); }); }); + +describe("exact Kimi-child fault capability", () => { + it.each(["parent_descriptor_changed", "parent_descriptor_shape", "parent_memfd_required", "parent_memfd_seals"])("retains the reviewed memfd guard %s", async code => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); await fixture.publishAction("action.txt", "action"); + h.override(request => request.op === "kimi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: false, diagnostic: { stage: "inspect_before", code, signalDispatched: false } }) } : undefined); + const error = await fixture.terminateKimiProvider!("workspace-id").catch(error => error); + expect(remoteNativeFixtureDiagnostics(error)[0]?.fault).toEqual({ stage: "inspect_before", code, signalDispatched: false }); await fixture.close(); + }); + it.each([false, null, true])("propagates a validated worker signal acknowledgement %s through the public diagnostic projector", async signalDispatched => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); await fixture.publishAction("action.txt", "action"); + h.override(request => request.op === "kimi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: false, error: "observer_evidence_incomplete", diagnostic: { stage: signalDispatched === false ? "inspect_before" : signalDispatched === null ? "signal" : "receipt", code: signalDispatched === false ? "native_guard_bytes" : "worker_failure", signalDispatched } }) } : undefined); + const error = await fixture.terminateKimiProvider!("workspace-id").catch(error => error); + expect(remoteNativeFixtureDiagnostics(error)).toEqual([{ phase: "kimi-provider-death", code: "kimi_provider_identity_or_signal_failed", fault: { stage: signalDispatched === false ? "inspect_before" : signalDispatched === null ? "signal" : "receipt", code: signalDispatched === false ? "native_guard_bytes" : "worker_failure", signalDispatched } }]); await fixture.close(); + }); + it("keeps a transport exception signal-unknown without leaking SDK text", async () => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); await fixture.publishAction("action.txt", "action"); + h.override(request => { if (request.op === "kimi-provider-death") throw new Error("private SDK credential/path"); return undefined; }); + const error = await fixture.terminateKimiProvider!("workspace-id").catch(error => error), diagnostics = remoteNativeFixtureDiagnostics(error); + expect(diagnostics).toEqual([{ phase: "kimi-provider-death", code: "transport_failure", fault: { stage: "worker_response", code: "invalid_response", signalDispatched: null } }]); expect(JSON.stringify(diagnostics)).not.toContain("private"); await fixture.close(); + }); + it.each([ + { exitCode: 1, result: "private stderr" }, + { exitCode: 0, result: "not json" }, + { exitCode: 0, result: JSON.stringify({ ok: false, diagnostic: { stage: "private", code: "private", signalDispatched: false } }) }, + ])("does not infer no signal from an untyped RPC failure", async response => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); await fixture.publishAction("action.txt", "action"); h.override(request => request.op === "kimi-provider-death" ? response : undefined); + const error = await fixture.terminateKimiProvider!("workspace-id").catch(error => error), diagnostics = remoteNativeFixtureDiagnostics(error); + expect(diagnostics).toHaveLength(1); expect(diagnostics[0]?.fault).toEqual({ stage: "worker_response", code: "invalid_response", signalDispatched: null }); expect(JSON.stringify(diagnostics)).not.toContain("private"); await fixture.close(); + }); + it("requires publication and consumes uncertain one-shot delivery without a worker-loss fallback", async () => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); + await expect(fixture.terminateKimiProvider!("workspace-id")).rejects.toThrow(); + await fixture.publishAction("action.txt", "Use the native read tool"); + h.override(request => request.op === "kimi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: false, error: "kimi_provider_identity_or_signal_failed" }) } : undefined); + await expect(fixture.terminateKimiProvider!("workspace-id")).rejects.toThrow(); + await expect(fixture.terminateKimiProvider!("workspace-id")).rejects.toThrow(); + const calls = h.calls.filter(call => call.request.op === "kimi-provider-death"); + expect(calls).toHaveLength(1); + expect(calls[0]!.request.contract).toEqual(kimiFaultRuntimeContract("linux-x64")); + expect(calls[0]!.request.bootstrapTemplate).toContain("/__paperclip_kimi_fault_snapshot__/distribution"); + expect(calls[0]!.request.guardTemplate).toContain("const digests={};"); + expect(h.calls.some(call => call.request.op === "inject-loss")).toBe(false); + await fixture.close(); + }); + it("returns the bound native-child receipt once and rejects a foreign lease", async () => { + const h = harness(), fixture = await bindRemoteNativeFixture(h.options); + await fixture.publishAction("action.txt", "Use the native read tool"); + const parent = { pid: 22, ppid: 21, startTicks: "101", bootId }, target = { pid: 23, ppid: 22, startTicks: "102", bootId }; + const receipt = { schema: "paperclip.e2e.kimi-provider-death.v1", binding, runtimeEnvironmentLeaseId: "workspace-id", target, ancestry: [target, parent, root], + ...kimiFaultRuntimeContract("linux-x64"), entrypointSha256: hash("entry"), guardSha256: hash("guard"), bootstrapSha256: hash("bootstrap"), + entrypointAttribution: "pinned_native_bootstrap_parent", originalChildArgvAvailable: false, observedChildTitle: "kimi-code", signalled: true, signal: "SIGKILL", targetKind: "kimi_native_child", workerSignalled: false }; + h.override(request => request.op === "kimi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: true, result: receipt }) } : undefined); + expect(await fixture.terminateKimiProvider!("workspace-id")).toEqual(receipt); + await expect(fixture.terminateKimiProvider!("workspace-id")).rejects.toThrow(); + expect(h.calls.filter(call => call.request.op === "kimi-provider-death")).toHaveLength(1); + await fixture.close(); + const foreign = harness(), other = await bindRemoteNativeFixture(foreign.options); + await other.publishAction("action.txt", "Use the native read tool"); + foreign.override(request => request.op === "kimi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: true, result: receipt }) } : undefined); + await expect(other.terminateKimiProvider!("foreign-lease")).rejects.toThrow("receipt_identity"); + await other.close(); + }); +}); diff --git a/tests/runner-e2e/remote-native-fixtures.ts b/tests/runner-e2e/remote-native-fixtures.ts index f965528d3f9..44602354b26 100644 --- a/tests/runner-e2e/remote-native-fixtures.ts +++ b/tests/runner-e2e/remote-native-fixtures.ts @@ -1,10 +1,13 @@ import { canonicalRemoteCopilotCommand, copilotDeathArguments, selectOwnedCopilotProcess } from "./copilot-provider-death.js"; +import { expectedNativeBootstrap, expectedNativeModuleGuard, KIMI_FAULT_SNAPSHOT, kimiFaultRuntimeContract, validateKimiProviderDeathReceipt, type KimiProviderDeathReceipt } from "./kimi-provider-fault-contract.js"; import { createHash, randomBytes } from "node:crypto"; import { watch, lstatSync, readFileSync } from "node:fs"; import { posix } from "node:path"; import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js"; const PI_FAULT_SOURCE = readFileSync(new URL("./pi-provider-fault.py", import.meta.url), "utf8"); +const KIMI_FAULT_SOURCE = readFileSync(new URL("./kimi-provider-fault.py", import.meta.url), "utf8"); +const KIMI_FAULT_DIAGNOSTIC_SOURCE = readFileSync(new URL("./kimi-provider-fault-diagnostic.py", import.meta.url), "utf8"); export const REMOTE_FIXTURE_DAYTONA_SDK_VERSION = "0.203.0"; const NODE = "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node"; @@ -34,18 +37,32 @@ function relative(value: string): string { } // Only closed diagnostic enums cross the remote boundary; SDK errors and output // are never retained. These diagnostics explain failed evidence, not qualification. -const RPC_PHASES = ["runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "release-attached", "read", "inject-loss", "provider-death", "pi-provider-death"] as const; -const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline", "terminal_evidence_incomplete", "pi_provider_identity_or_signal_failed"] as const; +const RPC_PHASES = ["lease-readmission", "runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "release-attached", "read", "inject-loss", "provider-death", "pi-provider-death", "kimi-provider-death"] as const; +const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline", "terminal_evidence_incomplete", "pi_provider_identity_or_signal_failed", "kimi_provider_identity_or_signal_failed"] as const; +const KIMI_FAULT_STAGES = ["observer_admission", "observer_sample", "worker_start", "worker_response", "input", "inspect", "inspect_before", "pidfd_open", "inspect_held", "pidfd_identity", "signal", "receipt", "inspected", "terminate", "complete"] as const; +const KIMI_FAULT_CODES = ["ancestry","ancestry_birth","ancestry_changed","argv_bound","boot_changed","bootstrap_template","closure_bound","closure_inventory","closure_metadata","closure_path","closure_pin","closure_shape","config_keys","cwd","dead_or_invalid_process","digest_shape","executable","file_bound","file_changed","file_realpath","file_shape","file_writable","guard_template","identity_changed_after_pidfd","input_bound","invalid_response","native_bootstrap_bytes","native_executable","native_guard_bytes","native_title","not_worker","observer_authority","observer_sample","parent_bootstrap","parent_descriptor_changed","parent_descriptor_shape","parent_directory_descriptor","parent_memfd_required","parent_memfd_seals","parent_node","parent_node_descriptor","parent_node_path","parent_source_pipe","pidfd_open","pidfd_required","pidfd_retired","pidfd_send_signal","process_bound","required_entries","root_binding","root_changed","root_flag","root_session","runner_changed","runner_executable","runner_shape","snapshot_digest","snapshot_path","snapshot_seal","stat_pid","unique_native_child","unknown_guard","worker_failure"] as const; +interface KimiFaultDiagnostic { stage: typeof KIMI_FAULT_STAGES[number]; code: typeof KIMI_FAULT_CODES[number]; signalDispatched: boolean | null } +function readKimiFaultDiagnostic(value: unknown): KimiFaultDiagnostic | undefined { + const row = value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; + if (Object.keys(row).sort().join() !== "code,signalDispatched,stage" || !KIMI_FAULT_STAGES.includes(row.stage as KimiFaultDiagnostic["stage"]) + || !KIMI_FAULT_CODES.includes(row.code as KimiFaultDiagnostic["code"]) || ![true, false, null].includes(row.signalDispatched as boolean | null)) return undefined; + const beforeSignal = ["observer_admission", "observer_sample", "input", "inspect", "inspect_before", "pidfd_open", "inspect_held", "pidfd_identity", "inspected", "terminate"].includes(row.stage as string); + const coherent = row.signalDispatched === false ? beforeSignal : row.signalDispatched === true + ? ["receipt", "complete"].includes(row.stage as string) : ["worker_start", "worker_response", "signal"].includes(row.stage as string); + if (!coherent || (row.signalDispatched === true && !["worker_failure", "unknown_guard"].includes(row.code as string))) return undefined; + return { stage: row.stage as KimiFaultDiagnostic["stage"], code: row.code as KimiFaultDiagnostic["code"], signalDispatched: row.signalDispatched as boolean | null }; +} +const unknownKimiFaultDiagnostic = (): KimiFaultDiagnostic => ({ stage: "worker_response", code: "invalid_response", signalDispatched: null }); type RpcPhase = typeof RPC_PHASES[number]; type RpcCode = typeof RPC_CODES[number]; -interface RpcDiagnostic { phase: RpcPhase; code: RpcCode } +interface RpcDiagnostic { phase: RpcPhase; code: RpcCode; fault?: KimiFaultDiagnostic } class RemoteFixtureError extends Error { constructor(message: string, readonly diagnostic: RpcDiagnostic, options?: ErrorOptions) { super(message, options); } } export function remoteNativeFixtureDiagnostics(error: unknown): RpcDiagnostic[] { const result: RpcDiagnostic[] = []; for (let current = error, depth = 0; current instanceof Error && depth < 3; current = current.cause, depth++) { - if (current instanceof RemoteFixtureError) result.push({ ...current.diagnostic }); + if (current instanceof RemoteFixtureError) result.push({ ...current.diagnostic, ...(current.diagnostic.fault ? { fault: { ...current.diagnostic.fault } } : {}) }); } return result; } @@ -162,6 +179,8 @@ const OBSERVER = String.raw` const fs=require('node:fs'),path=require('node:path'),crypto=require('node:crypto'),net=require('node:net'),cp=require('node:child_process'); const config=JSON.parse(Buffer.from(process.argv[2],'base64').toString()); const parseStat=PARSE_STAT;const runRoot=RUN_ROOT;const vanished=PROC_ENTRY_DISAPPEARED;const watchTarget=WATCH_TARGET;const PI_FAULT_SOURCE=PI_FAULT_PROGRAM,PI_CLOSURE_PIN=PI_CLOSURE_DIGEST; +const KIMI_FAULT_SOURCE=KIMI_FAULT_PROGRAM,KIMI_FAULT_DIAGNOSTIC_SOURCE=KIMI_FAULT_DIAGNOSTIC_PROGRAM; +const KIMI_FAULT_STAGES=KIMI_STAGES,KIMI_FAULT_CODES=KIMI_CODES,KIMI_WORKER_GUARDS=["ancestry","ancestry_birth","ancestry_changed","argv_bound","boot_changed","bootstrap_template","closure_bound","closure_inventory","closure_metadata","closure_path","closure_pin","closure_shape","config_keys","cwd","dead_or_invalid_process","digest_shape","executable","file_bound","file_changed","file_realpath","file_shape","file_writable","guard_template","identity_changed_after_pidfd","input_bound","native_bootstrap_bytes","native_executable","native_guard_bytes","native_title","not_worker","parent_bootstrap","parent_descriptor_changed","parent_descriptor_shape","parent_directory_descriptor","parent_memfd_required","parent_memfd_seals","parent_node","parent_node_descriptor","parent_node_path","parent_source_pipe","pidfd_open","pidfd_required","pidfd_retired","pidfd_send_signal","process_bound","required_entries","root_binding","root_changed","root_flag","root_session","runner_changed","runner_executable","runner_shape","snapshot_digest","snapshot_path","snapshot_seal","stat_pid","unique_native_child","unknown_guard"],readKimiFaultDiagnostic=KIMI_DIAGNOSTIC_READER; const selectCopilot=SELECT_COPILOT;const canonicalCopilot=CANONICAL_COPILOT;const copilotArgs=COPILOT_ARGS;let providerDeathDispatched=false; const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex'); const cwdStat=fs.lstatSync(config.binding.remoteCwd,{bigint:true}),rootStat=fs.lstatSync(config.root,{bigint:true}),scriptStat=fs.lstatSync(__filename,{bigint:true}),scriptHash=hash(fs.readFileSync(__filename)); @@ -169,7 +188,7 @@ const runtimeRoot=path.join(config.binding.remoteCwd,config.runtimeRelative),run const boot=fs.readFileSync('/proc/sys/kernel/random/boot_id','utf8').trim(); const targets=new Map();let complete=true,sealed=false,root=null,attached=null,child=null,client=null,childTimer=null; const incompleteReasons=new Set(),failureCodes=new Set();function incomplete(reason){complete=false;incompleteReasons.add(reason);failureCodes.add(reason)}function markIncomplete(reason){incomplete(reason)} -const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false,piFaultAttempted=false; +const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false,piFaultAttempted=false,kimiFaultAttempted=false; function identity(pid){return parseStat(pid,fs.readFileSync('/proc/'+pid+'/stat','utf8'),boot)} function deathCommand(pid){const command=fs.readFileSync('/proc/'+pid+'/cmdline').toString().split('\0').filter(Boolean);if(!/^\/proc\/self\/fd\/(?:3|7)$/.test(command[0]??''))return command;const executable=fs.realpathSync('/proc/'+pid+'/exe'),s=fs.statSync('/proc/'+pid+'/exe',{bigint:true}),fd='/proc/'+pid+'/fd/'+command[0].split('/').at(-1),d=fs.statSync(fd,{bigint:true});return canonicalCopilot(command,executable,{dev:String(s.dev),ino:String(s.ino)},{path:fs.realpathSync(fd),dev:String(d.dev),ino:String(d.ino)})} function table(){const ids=fs.readdirSync('/proc').filter(x=>/^\d+$/.test(x)&&Number(x)>1);if(ids.length>4096)throw Error('process_bound');return ids.flatMap(x=>{try{return [identity(Number(x))]}catch(e){if(vanished(e,()=>fs.lstatSync('/proc/'+x)))return [];throw e}})} @@ -208,7 +227,7 @@ function publicSnapshot(){const result=snapshot();if(result.attached)result.atta function seal(){if(sealed)return;sealed=true;workspaceWatch.close();for(const t of targets.values())t.watch.close();clearInterval(observer);finalReceipt=publicSnapshot();finalReceipt.files={};for(const [name,t] of targets){if(!file(t.path).absent)finalReceipt.files[name]=readSafe(t.path).toString('base64');}if(Buffer.byteLength(JSON.stringify(finalReceipt))>250000){finalReceipt.complete=false;finalReceipt.incompleteReasons.push('receipt_byte_bound');finalReceipt.files={};}if(child&&child.exitCode===null&&child.signalCode===null){finalReceipt.complete=false;finalReceipt.incompleteReasons.push('attached_process_live')};for(const socket of waiters)socket.end(JSON.stringify({ok:true,result:finalReceipt})+'\n');waiters.clear();setTimeout(()=>shutdown(null),250);} function sampleFailure(e){incomplete('process_sample_failed');if(e?.code)incomplete('process_read_io');else if(['invalid_proc_identity','invalid_proc_fields'].includes(e?.message))incomplete('process_stat_shape');else if(['process_prp_identity_shape','process_runtime_binding','process_prp_identity_changed','runner_binary_identity'].includes(e?.message))incomplete('process_runtime_binding')} const observer=setInterval(()=>{try{const p=sample();if(p.captured&&p.live.length===0&&!retiring){retiring=true;setTimeout(()=>{let end;try{end=sample()}catch(e){sampleFailure(e);return}if(end.live.length!==0){retiring=false;return}try{seal()}catch{incomplete('process_terminal_seal_failed')}},100)}}catch(e){sampleFailure(e)}},25); -const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=>sockets.delete(socket));socket.on('error',()=>{});let buffer='';socket.on('data',data=>{buffer+=data;if(Buffer.byteLength(buffer)>8192){socket.destroy();incomplete('control_request_bound');return}if(!buffer.includes('\n'))return;socket.removeAllListeners('data');try{const r=JSON.parse(buffer);if(r.nonce!==config.nonce)throw Error('control_identity');guard();let result; +const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=>sockets.delete(socket));socket.on('error',()=>{});let buffer='';socket.on('data',data=>{buffer+=data;if(Buffer.byteLength(buffer)>8192){socket.destroy();incomplete('control_request_bound');return}if(!buffer.includes('\n'))return;socket.removeAllListeners('data');let faultDiagnostic=null;try{const r=JSON.parse(buffer);if(r.op==='kimi-provider-death')faultDiagnostic={stage:'observer_admission',code:'observer_authority',signalDispatched:false};if(r.nonce!==config.nonce)throw Error('control_identity');guard();let result; if(r.op==='snapshot')result=finalReceipt??publicSnapshot(); else if(r.op==='wait'){if(finalReceipt)result=finalReceipt;else{if(waiters.size)throw Error('duplicate_receipt_channel');waiters.add(socket);for(const arm of armWaiters)arm.end(JSON.stringify({ok:true,result:{armed:true,sealed:false}})+'\n');armWaiters.clear();socket.on('close',()=>waiters.delete(socket));return}} else if(r.op==='arm'){if(waiters.size)result={armed:true,sealed};else{armWaiters.add(socket);socket.on('close',()=>armWaiters.delete(socket));return}} @@ -221,6 +240,24 @@ const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()= const childResult=cp.spawnSync('/usr/bin/python3',['-I','-c',PI_FAULT_SOURCE,Buffer.from(JSON.stringify(request)).toString('base64')],{env:{PATH:'/usr/bin:/bin'},timeout:8000,maxBuffer:32768,encoding:'utf8'}); if(childResult.status!==0||childResult.error)throw Error('pi_provider_identity_or_signal_failed');result=JSON.parse(childResult.stdout); } + else if(r.op==='kimi-provider-death'){ + try{ + if(kimiFaultAttempted||sealed||!publishedHash||!root||!complete||typeof r.runtimeEnvironmentLeaseId!=='string'||r.runtimeEnvironmentLeaseId!==observedPrpEnvironmentLeaseId)throw Error('kimi_provider_identity_or_signal_failed'); + kimiFaultAttempted=true;faultDiagnostic={stage:'observer_sample',code:'observer_sample',signalDispatched:false};const before=sample();if(!before.captured||!before.live.includes(root.pid))throw Error('kimi_provider_identity_or_signal_failed'); + const request={root:{pid:root.pid,ppid:root.ppid,startTicks:root.startTicks,bootId:root.bootId},binding:config.binding,runtimeEnvironmentLeaseId:r.runtimeEnvironmentLeaseId,runnerdSha256:config.runnerdSha256,closureSha256:r.contract.closureSha256,nodeSha256:r.contract.nodeSha256,guardLoaderSha256:r.contract.guardLoaderSha256,bootstrapTemplate:r.bootstrapTemplate,guardTemplate:r.guardTemplate}; + faultDiagnostic={stage:'worker_start',code:'worker_failure',signalDispatched:null}; + const childResult=cp.spawnSync('/usr/bin/python3',['-I','-c',KIMI_FAULT_DIAGNOSTIC_SOURCE,Buffer.from(KIMI_FAULT_SOURCE).toString('base64'),Buffer.from(JSON.stringify(request)).toString('base64')],{env:{PATH:'/usr/bin:/bin'},timeout:8000,maxBuffer:32768,encoding:'utf8'}); + faultDiagnostic={stage:'worker_response',code:'invalid_response',signalDispatched:null};let worker;try{worker=JSON.parse(childResult.stdout)}catch{faultDiagnostic={stage:'worker_response',code:'invalid_response',signalDispatched:null};throw Error('kimi_provider_identity_or_signal_failed')} + const observed=readKimiFaultDiagnostic({stage:worker?.stage,code:worker?.failingGuard??'worker_failure',signalDispatched:worker?.signalDispatched}); + if(!worker||typeof worker!=='object'||Array.isArray(worker)||Object.keys(worker).sort().join()!=='errno,externalModelCalls,failingGuard,qualification,receipt,schema,signalAttempted,signalDispatched,stage,status' + ||worker.schema!=='paperclip.kimi11.remote-fault-diagnostic.v1'||worker.qualification!==false||!['PASS','FAIL'].includes(worker.status)||!observed||worker.externalModelCalls!==0 + ||(worker.errno!==null&&(!Number.isInteger(worker.errno)||worker.errno<1||worker.errno>4095))||(worker.failingGuard!==null&&!KIMI_WORKER_GUARDS.includes(worker.failingGuard)) + ||typeof worker.signalAttempted!=='boolean'||worker.signalAttempted!==(observed.signalDispatched!==false)||childResult.error||childResult.signal||childResult.status!==(worker.status==='PASS'?0:1) + ||(worker.status==='PASS'&&(worker.stage!=='complete'||worker.failingGuard!==null||worker.errno!==null))||(worker.status==='FAIL'&&(worker.stage==='complete'||worker.receipt!==null)) + ||(observed.signalDispatched===true&&worker.failingGuard!==null&&worker.failingGuard!=='unknown_guard')){faultDiagnostic={stage:'worker_response',code:'invalid_response',signalDispatched:null};throw Error('kimi_provider_identity_or_signal_failed')} + faultDiagnostic=observed;if(childResult.status!==0||childResult.error||worker.status!=='PASS')throw Error('kimi_provider_identity_or_signal_failed');if(observed.signalDispatched!==true){faultDiagnostic={stage:'worker_response',code:'invalid_response',signalDispatched:null};throw Error('kimi_provider_identity_or_signal_failed')}result=worker.receipt; + }catch{socket.end(JSON.stringify({ok:false,error:'observer_evidence_incomplete',diagnostic:faultDiagnostic})+'\n');return;} + } else if(r.op==='attached'){if(attached||sealed)throw Error('attached_already_configured');if(!config.targets.includes(r.marker)||!Number.isInteger(r.delayMs)||r.delayMs<100||r.delayMs>8000||(r.waitForFinishAttempt!==undefined&&typeof r.waitForFinishAttempt!=='boolean'))throw Error('attached_bounds'); attached={waitForFinishAttempt:r.waitForFinishAttempt===true,releasedAtMs:null,connections:0,failure:null,commandExit:null,markerWrittenAtMs:null,markerWrittenMonotonicNs:null,clientExitedAtMs:null,clientExitedMonotonicNs:null}; const clientScript=path.join(config.root,'client.cjs'),clientSocket=path.join(config.root,'attached.sock'); @@ -249,13 +286,13 @@ const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()= // Socket handles never cross the evidence boundary. if(result?.attached)result={...result,attached:{connections:attached.connections,failure:attached.failure,commandExit:attached.commandExit,markerWrittenAtMs:attached.markerWrittenAtMs,markerWrittenMonotonicNs:attached.markerWrittenMonotonicNs,clientExitedAtMs:attached.clientExitedAtMs,clientExitedMonotonicNs:attached.clientExitedMonotonicNs}}; socket.end(JSON.stringify({ok:true,result})+'\n'); - }catch{socket.end(JSON.stringify({ok:false,error:'observer_evidence_incomplete'})+'\n')}})}); + }catch{socket.end(JSON.stringify({ok:false,error:'observer_evidence_incomplete',...(faultDiagnostic?{diagnostic:faultDiagnostic}:{})})+'\n')}})}); let closing=false;async function shutdown(reply){if(closing){reply?.end(JSON.stringify({ok:false,error:'closing'})+'\n');return}closing=true;sealed=true;workspaceWatch.close();for(const t of targets.values())t.watch.close();clearInterval(observer);let settled=true;try{if(child&&child.exitCode===null&&child.signalCode===null){child.kill('SIGTERM');const exited=new Promise(resolve=>child.once('exit',resolve));await Promise.race([exited,new Promise(resolve=>setTimeout(resolve,2000))]);if(child.exitCode===null&&child.signalCode===null){child.kill('SIGKILL');await Promise.race([exited,new Promise(resolve=>setTimeout(resolve,2000))]);}settled=child.exitCode!==null||child.signalCode!==null;}if(reply)reply.end(JSON.stringify({ok:settled,result:{closed:settled}})+'\n');}finally{for(const s of sockets)if(s!==reply)s.destroy();server.close();if(attached?.server)attached.server.close();setTimeout(()=>{reply?.destroy();if(settled){const current=fs.lstatSync(config.root,{bigint:true});if(current.dev===rootStat.dev&¤t.ino===rootStat.ino&&!current.isSymbolicLink())fs.rmSync(config.root,{recursive:true,force:true});else settled=false;}process.exit(settled?0:2)},100)}} server.listen(path.join(config.root,'control.sock'));setTimeout(()=>{incomplete('observer_ttl');shutdown(null)},config.observerTtlMs).unref(); `; const ATTACHED_CLIENT = String.raw`const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\n'));s.on('data',x=>{b+=x;if(b.length>1024)process.exit(6);if(b.includes('\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5)}});`; function observerSource() { - return OBSERVER.replace("PI_FAULT_PROGRAM", () => JSON.stringify(PI_FAULT_SOURCE)).replace("PI_CLOSURE_DIGEST", () => JSON.stringify(PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"])).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString()) + return OBSERVER.replace("KIMI_FAULT_DIAGNOSTIC_PROGRAM", () => JSON.stringify(KIMI_FAULT_DIAGNOSTIC_SOURCE)).replace("KIMI_STAGES", () => JSON.stringify(KIMI_FAULT_STAGES)).replace("KIMI_CODES", () => JSON.stringify(KIMI_FAULT_CODES)).replace("KIMI_DIAGNOSTIC_READER", () => readKimiFaultDiagnostic.toString()).replace("KIMI_FAULT_PROGRAM", () => JSON.stringify(KIMI_FAULT_SOURCE)).replace("PI_FAULT_PROGRAM", () => JSON.stringify(PI_FAULT_SOURCE)).replace("PI_CLOSURE_DIGEST", () => JSON.stringify(PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"])).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString()) .replace("SELECT_COPILOT", () => selectOwnedCopilotProcess.toString()).replace("CANONICAL_COPILOT", () => canonicalRemoteCopilotCommand.toString()).replace("COPILOT_ARGS", () => JSON.stringify(copilotDeathArguments)).replace("PROC_ENTRY_DISAPPEARED", () => remoteProcEntryDisappeared.toString()).replace("WATCH_TARGET", () => createRemoteTargetWatch.toString()).replace("ATTACHED_CLIENT", () => JSON.stringify(ATTACHED_CLIENT)); } const RPC = String.raw`const fs=require('node:fs'),net=require('node:net'),cp=require('node:child_process'),crypto=require('node:crypto');const r=JSON.parse(Buffer.from(process.argv[1],'base64').toString());const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex'); @@ -303,6 +340,7 @@ export interface RemoteNativeFixture { snapshot(label: string): Promise; /** One-shot fault against the exact admitted Pi child; never a worker kill. */ terminatePiProvider?(runtimeEnvironmentLeaseId: string): Promise; + terminateKimiProvider?(runtimeEnvironmentLeaseId: string): Promise; readFile(path: string): Promise; publishAction(path: string, text: string): Promise; injectOwnedRunLoss(): Promise>; @@ -420,14 +458,14 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption binding = current.binding; sentinel = current.sentinel; identityKey = key; return sandbox; } - async function bounded(operation: () => Promise, budgetMs: number): Promise { + async function bounded(operation: () => Promise, budgetMs: number, phase: RpcPhase): Promise { fail(budgetMs >= 1, "receipt_deadline"); let timer: ReturnType | undefined; try { return await Promise.race([ operation(), new Promise((_resolve, reject) => { - timer = setTimeout(() => reject(new Error("remote_native_fixture:remote_command_failed_or_deadline")), budgetMs); + timer = setTimeout(() => reject(new RemoteFixtureError("remote_native_fixture:remote_command_failed_or_deadline", { phase, code: "rpc_deadline", ...(phase === "kimi-provider-death" ? { fault: unknownKimiFaultDiagnostic() } : {}) })), budgetMs); timer.unref(); }), ]); @@ -452,24 +490,28 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption const command = `/usr/bin/env -i PATH=/usr/bin:/bin ${quote(NODE)} -e ${quote(rpcSource())} ${quote(payload)}`; let response: { exitCode: number; result: string }; try { response = await sandbox.process.executeCommand(command, binding!.remoteCwd, {}, timeoutMs / 1000); } - catch { throw new RemoteFixtureError("remote_native_fixture:remote_command_failed_or_deadline", { phase: request.op as RpcPhase, code: "transport_failure" }); } - if (typeof response.result !== "string" || Buffer.byteLength(response.result) > MAX_OUTPUT) throw new RemoteFixtureError("remote_native_fixture:command_failed_or_output_bound", { phase: request.op as RpcPhase, code: "output_bound" }); + catch { throw new RemoteFixtureError("remote_native_fixture:remote_command_failed_or_deadline", { phase: request.op as RpcPhase, code: "transport_failure", ...(request.op === "kimi-provider-death" ? { fault: unknownKimiFaultDiagnostic() } : {}) }); } + if (typeof response.result !== "string" || Buffer.byteLength(response.result) > MAX_OUTPUT) throw new RemoteFixtureError("remote_native_fixture:command_failed_or_output_bound", { phase: request.op as RpcPhase, code: "output_bound", ...(request.op === "kimi-provider-death" ? { fault: unknownKimiFaultDiagnostic() } : {}) }); if (response.exitCode !== 0) { - let diagnostic: RpcDiagnostic = { phase: request.op as RpcPhase, code: "invalid_response" }; + let diagnostic: RpcDiagnostic = { phase: request.op as RpcPhase, code: "invalid_response", ...(request.op === "kimi-provider-death" ? { fault: unknownKimiFaultDiagnostic() } : {}) }; try { const value = record(JSON.parse(response.result)), d = record(value.diagnostic); if (value.ok === false && Object.keys(value).sort().join() === "diagnostic,ok" && Object.keys(d).sort().join() === "code,phase" - && d.phase === request.op && RPC_CODES.includes(d.code as RpcCode)) diagnostic = { phase: d.phase as RpcPhase, code: d.code as RpcCode }; + && d.phase === request.op && RPC_CODES.includes(d.code as RpcCode)) diagnostic = { phase: d.phase as RpcPhase, code: d.code as RpcCode, ...(request.op === "kimi-provider-death" ? { fault: unknownKimiFaultDiagnostic() } : {}) }; } catch { /* Never retain remote text. */ } throw new RemoteFixtureError(`remote_native_fixture:command_failed_or_output_bound:${diagnostic.phase}:${diagnostic.code}`, diagnostic); } let parsed: Record; - try { parsed = record(JSON.parse(response.result)); } catch { throw new Error("remote_native_fixture:invalid_observer_json"); } + try { parsed = record(JSON.parse(response.result)); } catch { + if (request.op === "kimi-provider-death") throw new RemoteFixtureError("remote_native_fixture:invalid_observer_json", { phase: "kimi-provider-death", code: "invalid_response", fault: unknownKimiFaultDiagnostic() }); + throw new Error("remote_native_fixture:invalid_observer_json"); + } + if (request.op === "kimi-provider-death" && parsed.ok !== true) throw new RemoteFixtureError("remote_native_fixture:observer_incomplete", { phase: "kimi-provider-death", code: "kimi_provider_identity_or_signal_failed", fault: readKimiFaultDiagnostic(parsed.diagnostic) ?? unknownKimiFaultDiagnostic() }); fail(parsed.ok === true, "observer_incomplete"); return parsed.result; - }, budgetMs); + }, budgetMs, request.op as RpcPhase); } - const sandbox = await bounded(admittedSandbox, Math.min(LEASE_READMISSION_BUDGET_MS, receiptDeadlineAt - Date.now())); + const sandbox = await bounded(admittedSandbox, Math.min(LEASE_READMISSION_BUDGET_MS, receiptDeadlineAt - Date.now()), "lease-readmission"); const names = [...targets, ...(options.crossRoot ? ["@cross-root"] : [])]; const config = { root, nonce, binding, sentinel, targets, actionFile, crossRoot: options.crossRoot, runtimeRelative: RUNTIME_RELATIVE, runnerdSha256: options.runnerdSha256 }; // Lease activation precedes runner artifact staging. Observe the exact pinned @@ -523,6 +565,7 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption throw error; } let piFaultAttempted = false; + let kimiFaultAttempted = false; let closed = false, published = false, finished: RemoteNativeSnapshot | undefined; let retiredTerminal: RemoteNativeSnapshot | undefined; const retainedFiles = new Map(); @@ -539,6 +582,14 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption const receipt = await rpc({ op: "pi-provider-death", runtimeEnvironmentLeaseId }); return validatePiProviderDeathReceipt(receipt, binding!, baseline.processes.root!, runtimeEnvironmentLeaseId); }, + async terminateKimiProvider(runtimeEnvironmentLeaseId) { + fail(!closed && !finished && published && !kimiFaultAttempted && /^[A-Za-z0-9._:-]{1,240}$/u.test(runtimeEnvironmentLeaseId), "kimi_fault_admission"); + kimiFaultAttempted = true; + const contract = kimiFaultRuntimeContract("linux-x64"); + const receipt = await rpc({ op: "kimi-provider-death", runtimeEnvironmentLeaseId, contract, + bootstrapTemplate: expectedNativeBootstrap(KIMI_FAULT_SNAPSHOT, contract.nodeSha256), guardTemplate: expectedNativeModuleGuard(KIMI_FAULT_SNAPSHOT, []) }); + return validateKimiProviderDeathReceipt(receipt, binding!, baseline.processes.root!, runtimeEnvironmentLeaseId); + }, async readFile(path) { fail(!closed && names.includes(path), "unregistered_read"); if (finished) { const bytes = retainedFiles.get(path); fail(bytes, "terminal_file_missing"); return Buffer.from(bytes!); } diff --git a/tests/runner-e2e/results-root.test.ts b/tests/runner-e2e/results-root.test.ts new file mode 100644 index 00000000000..efdd0d6afca --- /dev/null +++ b/tests/runner-e2e/results-root.test.ts @@ -0,0 +1,259 @@ +import { execFile, execFileSync } from "node:child_process"; +import { mkdir, mkdtemp, readFile, readdir, realpath, rm, stat, symlink, writeFile } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; +import { promisify } from "node:util"; +import { afterEach, expect, it } from "vitest"; +import { runnerMatrix } from "./catalog.js"; +import { publishRunnerE2EAttemptEvidence } from "./evidence.js"; +import { prepareRunnerE2ECampaignDirectory, resolveRunnerE2EResultsRoot, RUNNER_E2E_RESULTS_ROOT } from "./results-root.js"; +import type { RunnerE2EResult } from "./types.js"; +import { MANUAL_PRIVATE_PROTOCOL } from "./manual-private-retention.js"; +import { inspectCellReportProducer } from "./manual-private-report-assets.js"; + +const roots: string[] = []; +afterEach(async () => { + await Promise.all(roots.splice(0).map(root => rm(root, { recursive: true, force: true }))); +}); + +async function fixture() { + const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "runner-output-test-"))); + roots.push(root); + const repository = path.join(root, "checkout"); + const privateDir = path.join(repository, "temporary", "case"); + await mkdir(privateDir, { recursive: true }); + const resultsRoot = resolveRunnerE2EResultsRoot(repository, { [RUNNER_E2E_RESULTS_ROOT]: path.join(root, "durable") }); + const campaignDir = await prepareRunnerE2ECampaignDirectory(resultsRoot, "unique-campaign"); + const uploadDir = path.join(campaignDir, "suite", "profile", "local", "case", "attempt-1"); + return { root, repository, privateDir, resultsRoot, uploadDir }; +} + +function result(status: "passed" | "failed" = "passed"): RunnerE2EResult { + return { + schema: "paperclip.runner-e2e.result/v2", executionId: "suite.profile.local.case", + suiteId: "suite", profileId: "profile", environmentId: "local", caseId: "case", + attempt: 1, provider: "fixture", model: "fixture", runtimeMode: "native", status, + startedAt: "2026-10-29T00:00:00Z", finishedAt: "2026-10-29T00:00:01Z", + durationMs: 1000, cleanup: status === "passed" ? "passed" : "not_started", + ...(status === "failed" ? { failureClass: "permanent_infrastructure", error: "startup failed" } : {}), + }; +} + +async function completeEvidence(privateDir: string) { + for (const directory of ["snapshots", "html-report", "blob-report"]) + await mkdir(path.join(privateDir, directory)); + await writeFile(path.join(privateDir, "final-state.png"), "synthetic-test-png"); + await writeFile(path.join(privateDir, "junit.xml"), "\n"); + await writeFile(path.join(privateDir, "html-report/index.html"), "

fixture

"); + await writeFile(path.join(privateDir, "snapshots/fixtures.json"), "{}\n"); + await writeFile(path.join(privateDir, "snapshots/api-state.json"), "{}\n"); + const blob = path.join(privateDir, "blob-report"); + await writeFile(path.join(blob, "report.txt"), "fixture report\n"); + execFileSync("zip", ["-q", "report.zip", "report.txt"], { cwd: blob }); +} + +it("publishes the real final evidence package outside the checkout and survives its removal", async () => { + const f = await fixture(); + await completeEvidence(f.privateDir); + expect(await publishRunnerE2EAttemptEvidence({ ...f, result: result(), secrets: [] })).toMatchObject({ status: "passed", cleanup: "passed" }); + await rm(f.repository, { recursive: true, force: true }); + expect(JSON.parse(await readFile(path.join(f.uploadDir, "result.json"), "utf8"))).toMatchObject({ status: "passed", cleanup: "passed" }); + const manifest = JSON.parse(await readFile(path.join(f.uploadDir, "evidence-manifest.json"), "utf8")); + expect(manifest.missing).toEqual([]); + expect(manifest.files).toEqual(expect.arrayContaining(["final-state.png", "snapshots/fixtures.json", "blob-report/report.zip"])); + expect((await stat(f.resultsRoot)).mode & 0o777).toBe(0o700); +}); + +it("retains an early failure without requiring successful-flow evidence", async () => { + const f = await fixture(); + await writeFile(path.join(f.privateDir, "server.log"), "startup failed\n"); + await publishRunnerE2EAttemptEvidence({ ...f, result: result("failed"), secrets: [] }); + await rm(f.repository, { recursive: true, force: true }); + const text = await readFile(path.join(f.uploadDir, "result.json"), "utf8"); + expect(JSON.parse(text)).toMatchObject({ status: "failed", failureClass: "permanent_infrastructure", cleanup: "not_started" }); + expect(await readFile(path.join(f.uploadDir, "server.log"), "utf8")).toBe("startup failed\n"); +}); + +it("still rejects and redacts leaked diagnostics at the external destination", async () => { + const f = await fixture(), secret = "synthetic-private-fixture-credential"; + await writeFile(path.join(f.privateDir, "server.log"), `startup failed: ${secret}\n`); + await publishRunnerE2EAttemptEvidence({ ...f, result: { ...result("failed"), error: `startup failed: ${secret}` }, secrets: [secret] }); + const text = await readFile(path.join(f.uploadDir, "result.json"), "utf8"); + expect(JSON.parse(text)).toMatchObject({ status: "failed", failureClass: "secret_leak", cleanup: "not_started" }); + expect(text).not.toContain(secret); + expect(await readFile(path.join(f.uploadDir, "server.log"), "utf8")).toContain("[REDACTED]"); +}); + +it("still fails a nominal pass when required evidence is missing at the external destination", async () => { + const f = await fixture(); + await completeEvidence(f.privateDir); + await rm(path.join(f.privateDir, "final-state.png")); + const final = await publishRunnerE2EAttemptEvidence({ ...f, result: result(), secrets: [] }); + expect(final).toMatchObject({ status: "failed", failureClass: "permanent_infrastructure", error: "Required evidence missing: final-state.png" }); + expect(JSON.parse(await readFile(path.join(f.uploadDir, "result.json"), "utf8"))).toEqual(final); +}); + +it("preserves the default and rejects unsafe override roots before any output or provider admission", async () => { + const f = await fixture(); + expect(resolveRunnerE2EResultsRoot(f.repository, {})).toBe(path.join(f.repository, "tests/runner-e2e/results")); + const alias = path.join(f.root, "alias"); + await symlink(f.resultsRoot, alias); + const file = path.join(f.root, "file"); + await writeFile(file, "not a directory"); + for (const unsafe of ["", "relative", `${f.root}/missing/../other`, path.parse(f.root).root, + os.homedir(), f.repository, path.join(f.repository, "source"), path.dirname(f.repository), alias, path.join(alias, "child"), file]) { + expect(() => resolveRunnerE2EResultsRoot(f.repository, { [RUNNER_E2E_RESULTS_ROOT]: unsafe })).toThrow(RUNNER_E2E_RESULTS_ROOT); + } +}); + +it("denies traversal and symlink destinations before deleting or writing final evidence", async () => { + const f = await fixture(); + for (const unsafe of [".", "..", "../outside", "one/two", ""]) + await expect(prepareRunnerE2ECampaignDirectory(f.resultsRoot, unsafe)).rejects.toThrow("safe path segment"); + const outside = path.join(f.root, "outside"); + await mkdir(path.join(outside, "attempt-1"), { recursive: true }); + const sentinel = path.join(outside, "attempt-1", "sentinel.txt"); + await writeFile(sentinel, "preserve me"); + await symlink(outside, path.join(f.resultsRoot, "alias-campaign")); + await expect(prepareRunnerE2ECampaignDirectory(f.resultsRoot, "alias-campaign")).rejects.toThrow("physical directories"); + await symlink(outside, path.join(f.resultsRoot, "unique-campaign", "alias-profile")); + const uploadDir = path.join(f.resultsRoot, "unique-campaign", "alias-profile", "attempt-1"); + await expect(publishRunnerE2EAttemptEvidence({ ...f, uploadDir, result: result("failed"), secrets: [] })).rejects.toThrow("physical directories"); + await expect(publishRunnerE2EAttemptEvidence({ ...f, uploadDir: outside, result: result("failed"), secrets: [] })).rejects.toThrow("beneath its results root"); + expect(await readFile(sentinel, "utf8")).toBe("preserve me"); +}); + +it("allows a normal retry in the same physical campaign", async () => { + const f = await fixture(); + await completeEvidence(f.privateDir); + await publishRunnerE2EAttemptEvidence({ ...f, result: result(), secrets: [] }); + expect(await prepareRunnerE2ECampaignDirectory(f.resultsRoot, "unique-campaign")).toBe(path.join(f.resultsRoot, "unique-campaign")); + await publishRunnerE2EAttemptEvidence({ ...f, result: result("failed"), secrets: [] }); + expect(JSON.parse(await readFile(path.join(f.uploadDir, "result.json"), "utf8"))).toMatchObject({ status: "failed", error: "startup failed" }); +}); + +async function shadowLauncher(dotenv: string, host?: { platform: NodeJS.Platform; arch: string }) { + const root = await realpath(await mkdtemp(path.join(os.tmpdir(), "runner-dotenv-output-"))); + roots.push(root); + const repository = path.join(root, "checkout"), directory = path.join(repository, "tests/runner-e2e"); + await mkdir(directory, { recursive: true }); + await writeFile(path.join(repository, "package.json"), '{"type":"module"}'); + let source = await readFile(path.join(import.meta.dirname, "launch.ts"), "utf8"); + if (host) source = source.replaceAll("assertRunnerE2EPrerequisites(executions);", + `assertRunnerE2EPrerequisites(executions, ${JSON.stringify(host)});`); + await writeFile(path.join(directory, "launch.ts"), source); + for (const entry of await readdir(import.meta.dirname, { withFileTypes: true })) { + if (["launch.ts", "results"].includes(entry.name)) continue; + await symlink(path.join(import.meta.dirname, entry.name), path.join(directory, entry.name), entry.isDirectory() ? "dir" : "file"); + } + const output = path.join(root, "dotenv-reports"), home = path.join(root, "empty-home"); + await mkdir(home); + await writeFile(path.join(repository, ".env.runner-e2e.local"), dotenv.replaceAll("OUTPUT_ROOT", output)); + async function launch(args: string[], environment: NodeJS.ProcessEnv = {}) { + try { + return { code: 0, ...await promisify(execFile)(process.execPath, + ["--import", path.resolve(import.meta.dirname, "../../cli/node_modules/tsx/dist/loader.mjs"), path.join(directory, "launch.ts"), ...args], + { cwd: repository, timeout: 20_000, env: { PATH: process.env.PATH, HOME: home, PAPERCLIP_E2E_CAMPAIGN_ID: "dotenv-probe", ...environment } }) }; + } catch (error) { + const result = error as { code: number; stdout: string; stderr: string }; + return { code: result.code, stdout: result.stdout, stderr: result.stderr }; + } + } + return { root, repository, output, launch }; +} + +it("rejects unsupported local Kimi provider loss before the actual launcher reads dotenv or creates resources", async () => { + const f = await shadowLauncher("KIMI_API_KEY=synthetic-never-read\n", { platform: "linux", arch: "x64" }); + const probe = path.join(f.root, "deny-dotenv-read.mjs"), marker = path.join(f.root, "dotenv-read-attempted"); + await writeFile(probe, `import fs from "node:fs"; +import { syncBuiltinESMExports } from "node:module"; +const original = fs.promises.readFile; +fs.promises.readFile = async (...args) => { + if (String(args[0]) === process.env.TEST_DOTENV_PATH) { + await fs.promises.writeFile(process.env.TEST_DOTENV_MARKER, "read attempted\\n"); + throw new Error("forbidden dotenv credential read"); + } + return original(...args); +}; +syncBuiltinESMExports(); +`); + const result = await f.launch(["--id", "kimi-qualification.runner-acpx-kimi.local.native-write-provider-loss"], { + NODE_OPTIONS: `--import=${probe}`, TEST_DOTENV_PATH: path.join(f.repository, ".env.runner-e2e.local"), TEST_DOTENV_MARKER: marker, + }); + expect(result.code).toBe(1); + expect(result.stderr).toContain("Local native Kimi provider-loss"); + expect(result.stderr).toContain("No provider credentials were loaded or sent"); + expect(result.stderr).not.toContain("forbidden dotenv credential read"); + await expect(stat(marker)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(stat(f.output)).rejects.toMatchObject({ code: "ENOENT" }); + await expect(stat(path.join(f.repository, "tests/runner-e2e/results"))).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("uses the actual dotenv output root before a normal native campaign fails credential admission", async () => { + const f = await shadowLauncher("export PAPERCLIP_RUNNER_E2E_RESULTS_ROOT='OUTPUT_ROOT'\n"); + const result = await f.launch(["--id", "kimi-qualification.runner-acpx-kimi.local.message-marker"]); + expect(result.code).toBe(1); + expect(result.stderr).toContain("Missing runner E2E credentials: KIMI_API_KEY"); + expect((await stat(path.join(f.output, "dotenv-probe"))).isDirectory()).toBe(true); + await expect(stat(path.join(f.repository, "tests/runner-e2e/results"))).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it.each(["list", "pending", "installed-startup"])("does not load dotenv in the %s admission path", async mode => { + const f = await shadowLauncher("invalid dotenv assignment\n"); + const pending = runnerMatrix.find(row => row.profile.id === "legacy-kimi-cli")!; + const args = mode === "list" ? ["--list", "--id", "kimi-qualification.runner-acpx-kimi.local.message-marker"] + : mode === "pending" ? ["--id", pending.id] + : ["--installed-startup-only", "--id", "extended-harnesses.runner-acpx-pi.local.hello-complete", "--max-automatic-retries", "0"]; + const result = await f.launch(args); + expect(result.stderr).not.toContain("Invalid .env.runner-e2e.local"); + if (mode === "list") { + expect(result.code).toBe(0); + expect(result.stdout).toContain("kimi-qualification.runner-acpx-kimi.local.message-marker"); + } else { + expect(result.code).toBe(1); + expect(result.stderr).toContain(mode === "pending" ? "prerequisite(s) are pending" : "Installed startup-only requires"); + } + await expect(stat(f.output)).rejects.toMatchObject({ code: "ENOENT" }); +}); + +async function verifiedPrivateLaunch(f: Awaited>, executionId: string) { + const privateRoot = path.join(f.root, "verified-private"), file = path.join(privateRoot, "preflight.json"); + await mkdir(privateRoot, { mode: 0o700 }); + const identity = { workflowSha: "a".repeat(40), targetSha: "b".repeat(40), runId: "123", attempt: "1" }; + await writeFile(file, JSON.stringify({ protocol: MANUAL_PRIVATE_PROTOCOL, status: "verified", private: true, + root: privateRoot, ...identity, executionIds: [executionId], cellReportProducer: await inspectCellReportProducer() }), { mode: 0o600 }); + return { privateRoot, environment: { PAPERCLIP_RUNNER_E2E_PRIVATE_PREFLIGHT: file, RUNNER_TEMP: f.root, + GITHUB_SHA: identity.workflowSha, GITHUB_RUN_ID: identity.runId, GITHUB_RUN_ATTEMPT: identity.attempt, + PAPERCLIP_RUNNER_E2E_SOURCE_SHA: identity.targetSha } }; +} + +it.each([ + ["process", "conflicting"], ["process", "invalid"], ["dotenv", "conflicting"], ["dotenv", "invalid"], +] as const)("actual launcher keeps verified private custody over %s %s output settings", async (origin, kind) => { + const setting = kind === "invalid" ? "relative-invalid-root" : "OUTPUT_ROOT"; + const f = await shadowLauncher(origin === "dotenv" ? `${RUNNER_E2E_RESULTS_ROOT}='${setting}'\n` : ""); + const executionId = "kimi-qualification.runner-acpx-kimi.local.message-marker"; + const custody = await verifiedPrivateLaunch(f, executionId); + const environment = { ...custody.environment, + ...(origin === "process" ? { [RUNNER_E2E_RESULTS_ROOT]: kind === "invalid" ? setting : f.output } : {}) }; + const result = await f.launch(["--id", executionId], environment); + expect(result.code).toBe(1); + expect(result.stderr).toContain("Missing runner E2E credentials: KIMI_API_KEY"); + expect(result.stderr).not.toContain("requires a normalized absolute directory"); + expect((await stat(path.join(custody.privateRoot, "results/dotenv-probe"))).isDirectory()).toBe(true); + expect(result.stderr).toContain("Private evidence finalization failed; no public fallback"); + for (const output of [f.output, path.join(f.repository, "tests/runner-e2e/results")]) + await expect(stat(output)).rejects.toMatchObject({ code: "ENOENT" }); +}); + +it("actual launcher rejects foreign private custody before reading dotenv or creating fallback output", async () => { + const f = await shadowLauncher("invalid dotenv assignment\n"); + const executionId = "kimi-qualification.runner-acpx-kimi.local.message-marker"; + const custody = await verifiedPrivateLaunch(f, executionId); + const result = await f.launch(["--id", executionId], { ...custody.environment, GITHUB_RUN_ID: "124" }); + expect(result.code).toBe(1); + expect(result.stderr).toContain("Private evidence preflight does not match this exact attempt"); + expect(result.stderr).not.toContain("Invalid .env.runner-e2e.local"); + for (const output of [path.join(custody.privateRoot, "results"), f.output, path.join(f.repository, "tests/runner-e2e/results")]) + await expect(stat(output)).rejects.toMatchObject({ code: "ENOENT" }); +}); diff --git a/tests/runner-e2e/results-root.ts b/tests/runner-e2e/results-root.ts new file mode 100644 index 00000000000..158add45275 --- /dev/null +++ b/tests/runner-e2e/results-root.ts @@ -0,0 +1,71 @@ +import { lstatSync, realpathSync } from "node:fs"; +import { chmod, lstat, mkdir, realpath } from "node:fs/promises"; +import os from "node:os"; +import path from "node:path"; + +export const RUNNER_E2E_RESULTS_ROOT = "PAPERCLIP_RUNNER_E2E_RESULTS_ROOT"; + +function contains(parent: string, child: string) { + return child === parent || child.startsWith(`${parent}${path.sep}`); +} + +export function resolveRunnerE2EResultsRoot( + repositoryRoot: string, + environment: NodeJS.ProcessEnv = process.env, +): string { + const configured = environment[RUNNER_E2E_RESULTS_ROOT]; + if (configured === undefined) + return path.join(repositoryRoot, "tests/runner-e2e/results"); + if (!configured || configured.trim() !== configured || /[\u0000-\u001f]/u.test(configured) + || !path.isAbsolute(configured) || path.normalize(configured) !== configured) + throw new Error(`${RUNNER_E2E_RESULTS_ROOT} requires a normalized absolute directory`); + const repository = path.resolve(repositoryRoot); + if (path.dirname(configured) === configured || contains(configured, repository) + || contains(repository, configured) || contains(configured, os.homedir())) + throw new Error(`${RUNNER_E2E_RESULTS_ROOT} requires a dedicated directory outside the checkout and home roots`); + + let ancestor = configured; + while (true) { + try { + const metadata = lstatSync(ancestor); + if (!metadata.isDirectory() || realpathSync(ancestor) !== ancestor) + throw new Error(`${RUNNER_E2E_RESULTS_ROOT} must not use files or symbolic-link paths`); + break; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + ancestor = path.dirname(ancestor); + } + } + return configured; +} + +export async function prepareRunnerE2ECampaignDirectory(resultsRoot: string, campaignId: string) { + if (!/^[A-Za-z0-9_.-]+$/u.test(campaignId) || campaignId === "." || campaignId === "..") + throw new Error("Runner E2E campaign requires one safe path segment"); + const directory = path.join(resultsRoot, campaignId); + await assertRunnerE2EResultsPath(resultsRoot, directory); + await mkdir(resultsRoot, { recursive: true, mode: 0o700 }); + await chmod(resultsRoot, 0o700); + await mkdir(directory, { recursive: true, mode: 0o700 }); + await assertRunnerE2EResultsPath(resultsRoot, directory); + return directory; +} + +export async function assertRunnerE2EResultsPath(resultsRoot: string, directory: string) { + if (!path.isAbsolute(resultsRoot) || path.normalize(resultsRoot) !== resultsRoot + || !path.isAbsolute(directory) || path.normalize(directory) !== directory + || directory === resultsRoot || !contains(resultsRoot, directory)) + throw new Error("Runner E2E output must remain beneath its results root"); + let ancestor = directory; + while (true) { + try { + const metadata = await lstat(ancestor); + if (!metadata.isDirectory() || await realpath(ancestor) !== ancestor) + throw new Error("Runner E2E output must use physical directories without symbolic links"); + return; + } catch (error) { + if ((error as NodeJS.ErrnoException).code !== "ENOENT") throw error; + ancestor = path.dirname(ancestor); + } + } +} diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index a18da0c05e3..988ce11cb53 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,4 @@ +import { kimiCandidateAdmissionForExecution } from "./server-restart-request.js"; import { runPlanRevisionAcceptance, isPendingPlanConfirmation, normalizePlanMarkdown, expectPlanStageVisible } from "./plan-revision-flow.js"; import { createWorkerTiming } from "./worker-timing.js"; import { captureTaskChat, type TaskChatEvidence } from "./task-chat.js"; @@ -17,6 +18,7 @@ import { runNativeActiveStopFlow as runCopilotActiveStopFlow } from "./copilot-a import { nativeActiveStopTasks as copilotActiveStopTasks } from "./copilot-active-stop-tasks.js"; import type { RestartRunnerIdentity } from "./process-tree-owner.js"; import { runNativeActiveStopFlow } from "./native-active-stop-flow.js"; +import { runKimiLifecycleFlow } from "./kimi-lifecycle.js"; import { runPiControlsFlow } from "./pi-controls-flow.js"; import { runCursorNativeFlow } from "./cursor-native-flow.js"; import { createRemoteNativeBootstrap, createRemoteFixtureClient } from "./remote-native-bootstrap.js"; @@ -65,14 +67,14 @@ import { expect, test, type Page } from "@playwright/test"; import { RunnerApi, pollUntil } from "./api.js"; import { buildRuntimeUsage, summarizeExecutionBilling } from "./billing.js"; import { collectAttemptFeedback, emptyFeedbackSummary } from "./feedback.js"; -import { finishBudgetedEvalSetup } from "./budgeted-setup.js"; +import { createKimiLifecycleTiming, finishBudgetedEvalSetup, waitForEvalAgentReady } from "./budgeted-setup.js"; import { establishPublicMcpSession, runPublicMcpFlow } from "./public-mcp-flow.js"; import { assistantUsage } from "./public-mcp-model.js"; import { runnerExecutionById } from "./catalog.js"; import { classifyFailure } from "./failure-classifier.js"; import { runnerE2EServerControlPaths } from "./harness-env.js"; import { setupConnectionReview } from "./connection-reviews.js"; -import { setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage, type LiveFixtureValues } from "./live-fixtures.js"; +import { setupLiveFixtures, selectFixtureRuntimeLeases, fixtureRuntimeCostCoverage, waitForFixtureDaytonaProvider, type LiveFixtureValues } from "./live-fixtures.js"; import { evaluateMatcher, persistedFinalRunMessage, type MatcherResult } from "./matchers.js"; import { readRegisteredArtifacts } from "./registered-artifact.js"; import { @@ -264,6 +266,7 @@ function sortRunsChronologically(runs: readonly RunRecord[]): RunRecord[] { async function restartIsolatedPaperclipServer(input: { api: RunnerApi; requestId: string; + kimiCandidateAdmission?: import("../../packages/paperclip-runner/src/drivers/acpx/candidate-admission.js").AcpxCandidateAdmission; deadlineAt: number; preserveRunner?: RestartRunnerIdentity; discardControllerCache?: boolean; @@ -282,7 +285,7 @@ async function restartIsolatedPaperclipServer(input: { const temporaryRequestPath = `${requestPath}.${process.pid}.${input.requestId}.tmp`; await writeFile( temporaryRequestPath, - JSON.stringify({ requestId: input.requestId, ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true, + JSON.stringify({ requestId: input.requestId, ...(input.kimiCandidateAdmission ? { kimiCandidateAdmission: input.kimiCandidateAdmission } : {}), ...(input.controllerDowntimeMs !== undefined && !input.controllerOutage ? { controllerDowntimeMs: input.controllerDowntimeMs } : {}), discardControllerCache: input.discardControllerCache === true, ...(input.controllerOutage ? { controllerOutage: input.controllerOutage } : {}), ...(input.recoveryBinding ? { recoveryBinding: input.recoveryBinding } : {}), ...(input.preserveRunner ? { preserveRunner: input.preserveRunner } : {}), @@ -567,10 +570,12 @@ for (const execution of executions) { test.skip(execution.suite.id === "first-task" && execution.profile.id === "runner-acpx-claude" && (process.platform !== "darwin" || process.arch !== "arm64"), "Ordinary native Claude onboarding is qualified on Apple Silicon only in this delivery."); + const kimiLifecycleTiming = execution.task.flow === "kimi_native_lifecycle" + ? createKimiLifecycleTiming({ startedAt: Date.now(), deadlineMs }) : undefined; test.setTimeout( - execution.task.flow === "warm_three_turn" + kimiLifecycleTiming?.playwrightTimeoutMs ?? (execution.task.flow === "warm_three_turn" ? deadlineMs - : deadlineMs + 90_000, + : deadlineMs + 90_000), ); // Remote allocation uncertainty survives a worker crash before publication. if (mayAllocateRemoteResources(execution.environment.id)) { @@ -607,10 +612,18 @@ for (const execution of executions) { }) : undefined; const saveTrace = async () => { if (trace) await writeSanitizedJson(snapshotsDir, "braintrust-trace.json", trace.snapshot(), secrets); }; const onCompanyCreated = async (id: string) => { + const admission = kimiCandidateAdmissionForExecution(execution, id); + if (admission) { + await restartIsolatedPaperclipServer({ api, requestId: `kimi-startup-${nonce}`, + kimiCandidateAdmission: admission, deadlineAt: startedAtMs + deadlineMs }); + if (execution.environment.id === "daytona") { + await waitForFixtureDaytonaProvider(api, { companyId: id, deadlineAt: startedAtMs + deadlineMs }); + } + } if (trace) { try { await trace.configureCompany(api, id); } finally { await saveTrace(); } } }; const api = new RunnerApi(request); - const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "repository_skills", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss", "public_mcp", "case_handoff"].includes(execution.task.flow); + const companyRunFlow = execution.suite.id === "task-titles" || ["plan_task_guidance", "blocker_guidance", "continuation_accounting", "continuation", "context_integrity", "repository_skills", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss", "kimi_native_lifecycle", "public_mcp", "case_handoff"].includes(execution.task.flow); const publicMcpUsage = execution.task.flow === "public_mcp" ? assistantUsage(execution.profile.provider === "claude" ? "anthropic" : "openai", execution.profile.model) : undefined; let publicMcpUserId = ""; const consoleDiagnostics: Array> = []; @@ -931,11 +944,21 @@ for (const execution of executions) { workspacePath, credentials, daytonaImage: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE, + cleanupDeadlineAt: kimiLifecycleTiming?.cleanupDeadlineAt, onCompanyCreated, }); await finishBudgetedEvalSetup({ executionId: execution.id, environmentId: execution.environment.id, agentId: fixtures.agent.id, companyId: fixtures.company.id, deadlineAt: startedAtMs + deadlineMs, api, observe: observations => writeSanitizedJson(snapshotsDir, "budgeted-setup.json", { observations }, secrets) }); + const kimiWorkDeadlineAt = kimiLifecycleTiming ? await (async () => { + await waitForEvalAgentReady({ environmentId: execution.environment.id, agentId: fixtures.agent.id, companyId: fixtures.company.id, + deadlineAt: kimiLifecycleTiming.setupDeadlineAt, api, + observe: observations => writeSanitizedJson(snapshotsDir, "kimi-lifecycle-setup.json", { observations }, secrets) }); + const readyAt = Date.now(), workDeadlineAt = kimiLifecycleTiming.beginWork(readyAt); + await writeSanitizedJson(snapshotsDir, "kimi-lifecycle-timing.json", { setupDeadlineAt: kimiLifecycleTiming.setupDeadlineAt, + readyAt, workDeadlineAt, setupBudgetMs: deadlineMs, workBudgetMs: deadlineMs, playwrightTimeoutMs: kimiLifecycleTiming.playwrightTimeoutMs }, secrets); + return workDeadlineAt; + })() : undefined; await workerTiming.enter("execution"); trace?.phase("execution"); @@ -951,14 +974,14 @@ for (const execution of executions) { nativeInitial = { issueIds: issues.map(value => value.id), agentIds: agents.map(value => value.id), workspaceDigest }; } const remoteBootstrap = execution.environment.id === "daytona" - && ["pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss"].includes(execution.task.flow) + && ["pi_native", "pi_controls", "copilot_protection", "cursor_native", "native_active_stop", "native_provider_loss", "kimi_native_lifecycle"].includes(execution.task.flow) ? createRemoteNativeBootstrap({ api, daytona: await createRemoteFixtureClient(credentials.DAYTONA_API_KEY ?? ""), companyId: fixtures.company.id, environmentId: fixtures.environment.id, agentId: fixtures.agent.id, image: process.env.PAPERCLIP_E2E_DAYTONA_IMAGE ?? "", nodeSha256: process.env.PAPERCLIP_E2E_DAYTONA_NODE_SHA256 ?? "", runnerdSha256: process.env.PAPERCLIP_E2E_DAYTONA_RUNNERD_SHA256 ?? "", - deadlineAt: startedAtMs + deadlineMs, + deadlineAt: kimiWorkDeadlineAt ?? startedAtMs + deadlineMs, requireCopilotTaskContext: execution.profile.qualificationCandidate === "copilot", evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), }) : undefined; @@ -1144,6 +1167,15 @@ for (const execution of executions) { }); issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piControls.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); + } else if (execution.task.flow === "kimi_native_lifecycle") { + const story = await runKimiLifecycleFlow({ + page, api, fixtures, execution, nonce, workspacePath, deadlineAt: kimiWorkDeadlineAt!, + observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; }, + capture: captureScreenshot, evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), + remoteBootstrap, registerCleanupAssertion, registerBeforeEnvironmentTeardownAssertion, + }); + issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; + matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `kimiNativeLifecycle.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); } else if (execution.task.flow === "native_provider_loss") { const story = await runNativeProviderLossFlow({ page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, @@ -1267,6 +1299,7 @@ for (const execution of executions) { } else if (execution.task.flow === "agent_chat") { const chat = await runChatFlow({ page, api, fixtures, execution, nonce, workspacePath, + deadlineAt: startedAtMs + deadlineMs, restart: () => restartChatServer(page, () => restartIsolatedPaperclipServer({ api, requestId: `chat-${nonce}`, deadlineAt: startedAtMs + deadlineMs })), observe: (chatIssue, chatRuns) => { issue = chatIssue; selectedRuns = chatRuns; }, capture: captureScreenshot, @@ -2732,6 +2765,7 @@ for (const execution of executions) { { issue, run, + runs: selectedRuns, comments: terminal.comments, interactions: terminal.interactions, planLifecycleEvidence, diff --git a/tests/runner-e2e/server-candidate-startup.test.mjs b/tests/runner-e2e/server-candidate-startup.test.mjs new file mode 100644 index 00000000000..38c960c1bb9 --- /dev/null +++ b/tests/runner-e2e/server-candidate-startup.test.mjs @@ -0,0 +1,61 @@ +import assert from "node:assert/strict"; +import { readFileSync } from "node:fs"; +import { EventEmitter } from "node:events"; +import vm from "node:vm"; +import { transformSync } from "esbuild"; +import { it } from "vitest"; + +it("stages company authority only across the actual completed supervisor restart boundary", async () => { + const source = readFileSync(new URL('./server.ts', import.meta.url), 'utf8'); + const extract=(start,end)=>{const a=source.indexOf(start),b=source.indexOf(end,a);assert.ok(a>=0&&b>a);const part=source.slice(a,b);assert.ok(part.trimEnd().endsWith('}'));return part;}; + const functions=extract('async function startServer()', '\nfunction delay(')+'\n'+extract('async function restartServer(', '\nfor (const signal of'); + const controls=[]; + const grantA={schema:'paperclip.acpx-candidate-admission.v1',companyId:'11111111-1111-4111-8111-111111111111',agent:'kimi',model:'kimi-for-coding',platform:'darwin',architecture:'arm64',expiresAt:Date.now()+60000}; + const grantB={...grantA,companyId:'22222222-2222-4222-8222-222222222222'}; + function factory(code){ + const events=[],spawns=[];let pid=100,stopFails=false,shuttingDown=false; + const stop=async c=>{events.push('stop');if(stopFails)throw new Error('controlled stop failure');c.exitCode=0;events.push('stopped');}; + stop.watch=async()=>{};stop.forRestart=stop;stop.crashForRestart=stop; + const ctx=vm.createContext({Map,Set,JSON,Date,Error,Promise, + process:{execPath:'/controlled/pinned-node',env:{PAPERCLIP_ACPX_CANDIDATE_ADMISSION:'ambient-forgery'},stdout:{write(){}},stderr:{write(){}}}, + upgradeArtifacts:undefined,artifactIndex:0,selectedExecutions:['controlled-kimi'], + verifyInstalledCli:async()=>null,installedReleaseLaunch:()=>{throw Error('unexpected installed path');}, + definedServerEnvironment:{PATH:'/controlled/bin'},repositoryRoot:'/controlled/repo',paperclipCli:'/controlled/cli',runnerE2EServerDetached:true, + runnerE2ETypeScriptProcessArgs:()=>['controlled-cli','onboard','--yes','--run'], + shutdownRequested:()=>shuttingDown,shutdownSignal:null,expectedStops:new Set(),childErrors:new Map(),servingArtifacts:new Map(), + kimiCandidateAdmission:undefined,activeRestartRequestId:null,child:null,log:{write(){}},appendLog(){}, + stopServer:stop,waitForHealthToStop:async()=>events.push('health-stopped'),waitForHealth:async()=>events.push('healthy'),writeRestartAck:async()=>events.push('ack'), + spawn:(executable,args,options)=>{events.push('spawn');const c=new EventEmitter();c.pid=++pid;c.stdout=new EventEmitter();c.stderr=new EventEmitter();c.exitCode=null;c.signalCode=null;spawns.push({executable,args,options});return c;}, + }); + vm.runInContext(transformSync(code,{loader:'ts',target:'es2022',format:'cjs'}).code+'\nglobalThis.control = {startServer,restartServer};',ctx); + return {ctx,events,spawns,setStopFails:v=>stopFails=v,setShutdown:v=>shuttingDown=v,...ctx.control}; + } + const old=factory(functions.replace('if (stagedKimiAdmission) kimiCandidateAdmission = stagedKimiAdmission;','')); + await old.startServer();await old.restartServer({requestId:'counterfactual-missing-assignment',kimiCandidateAdmission:grantA}); + assert.equal(old.spawns.at(-1).options.env.PAPERCLIP_ACPX_CANDIDATE_ADMISSION,undefined); + controls.push({name:'removed-assignment counterfactual reaches real spawn without staged authority',result:'RED_REPRODUCED',counterfactualOnly:true}); + const current=factory(functions); + await current.startServer();assert.equal(current.spawns[0].options.env.PAPERCLIP_ACPX_CANDIDATE_ADMISSION,undefined); + controls.push({name:'initial start ignores ambient authority and has no supervisor-held record',result:'PASS'}); + current.events.length=0; + await current.restartServer({requestId:'stage-authority',kimiCandidateAdmission:grantA}); + assert.deepEqual(JSON.parse(current.spawns.at(-1).options.env.PAPERCLIP_ACPX_CANDIDATE_ADMISSION),grantA); + assert.ok(current.events.indexOf('stopped') { root = await mkdtemp(join(tmpdir(), "paperclip-supervisor-request-")); requestPath = join(root, "restart.json"); }); -afterEach(async () => { await rm(root, { recursive: true, force: true }); }); +afterEach(async () => { + vi.doUnmock(profilesModule); + vi.resetModules(); + await rm(root, { recursive: true, force: true }); +}); const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite => ["codex", "acpx"].map(provider => { @@ -58,3 +63,39 @@ it("ignores absent and incomplete atomic request files", async () => { await writeFile(requestPath, "{"); await expect(readRestartRequest(requestPath, [cells[0].id])).resolves.toBeNull(); }); + +it.each(["local", "daytona"])("uses ordinary released Kimi restart without candidate authority on %s", async environment => { + const execution = runnerMatrix.find(row => row.suite.id === "kimi-qualification" && row.environment.id === environment)!; + const admission = kimiCandidateAdmissionForExecution(execution, "11111111-1111-4111-8111-111111111111"); + expect(admission).toBeUndefined(); + await writeFile(requestPath, JSON.stringify({ requestId: "kimi-released", kimiCandidateAdmission: admission })); + const request = await readRestartRequest(requestPath, [execution.id]); + expect(request).toMatchObject({ requestId: "kimi-released" }); + expect(request).not.toHaveProperty("kimiCandidateAdmission"); +}); + +it.each(["local", "daytona"])("stages only exact pending Kimi company/model/profile/target via owned restart on %s", async environment => { + const actual = await vi.importActual(profilesModule); + vi.doMock(profilesModule, () => ({ ...actual, QUALIFIED_ACPX_PROFILES: { ...actual.QUALIFIED_ACPX_PROFILES, + kimi: { ...actual.QUALIFIED_ACPX_PROFILES.kimi, qualificationStatus: "pending" } } })); + vi.resetModules(); + const { readRestartRequest, kimiCandidateAdmissionForExecution } = await import("./server-restart-request.js"); + const execution = runnerMatrix.find(row => row.suite.id === "kimi-qualification" && row.environment.id === environment)!; + const admission = kimiCandidateAdmissionForExecution(execution, "11111111-1111-4111-8111-111111111111")!; + expect(admission).toBeDefined(); + await writeFile(requestPath, JSON.stringify({ requestId: "kimi-startup", kimiCandidateAdmission: admission })); + await expect(readRestartRequest(requestPath, [execution.id])).resolves.toMatchObject({ requestId: "kimi-startup", kimiCandidateAdmission: admission }); + expect(admission).not.toHaveProperty("runId"); + for (const change of [{ companyId: "foreign" }, { agent: "antigravity" }, { model: "different" }, + { profileDigest: `sha256:${"0".repeat(64)}` }, { nativeClosureDigest: `sha256:${"0".repeat(64)}` }, + { expiresAt: 0 }, { runId: "injected-run" }, { architecture: "arm64", platform: "linux" }]) { + await writeFile(requestPath, JSON.stringify({ requestId: "wrong-kimi", kimiCandidateAdmission: { ...admission, ...change } })); + await expect(readRestartRequest(requestPath, [execution.id])).rejects.toThrow(); + } + await writeFile(requestPath, JSON.stringify({ requestId: "unselected-kimi", kimiCandidateAdmission: admission })); + for (const ids of [[], [cells[0].id]]) await expect(readRestartRequest(requestPath, ids)).rejects.toThrow(); + for (const change of [{ controllerDowntimeMs: 1 }, { preserveRunner: {} }, { discardControllerCache: true }]) { + await writeFile(requestPath, JSON.stringify({ requestId: "mixed-kimi", kimiCandidateAdmission: admission, ...change })); + await expect(readRestartRequest(requestPath, [execution.id])).rejects.toThrow(); + } +}); diff --git a/tests/runner-e2e/server-restart-request.ts b/tests/runner-e2e/server-restart-request.ts index ab38d2bc590..fa246ee2ab1 100644 --- a/tests/runner-e2e/server-restart-request.ts +++ b/tests/runner-e2e/server-restart-request.ts @@ -1,5 +1,9 @@ import { readFile } from "node:fs/promises"; import { runnerMatrix } from "./catalog.js"; +import { parseAcpxCandidateAdmission, type AcpxCandidateAdmission } from "../../packages/paperclip-runner/src/drivers/acpx/candidate-admission.js"; +import { QUALIFIED_ACPX_PROFILES } from "../../packages/paperclip-runner/src/drivers/acpx/qualified-profiles.js"; +import { KIMI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/kimi-closure-pins.js"; +import type { MatrixExecution } from "./types.js"; import { controllerOutageForExecution, parseControllerOutage, type ControllerOutage } from "./controller-outage.js"; import { parseRecoveryBinding, type RecoveryBinding } from "./controller-recovery-observer.js"; import { parseFinalizationCrashBinding, type FinalizationCrashBinding } from "./finalization-crash-boundary.js"; @@ -7,6 +11,7 @@ import { parseRestartRunnerIdentity, type RestartRunnerIdentity } from "./proces export interface RestartRequest { requestId: string; + kimiCandidateAdmission?: AcpxCandidateAdmission; controllerDowntimeMs?: number; preserveRunner?: RestartRunnerIdentity; discardControllerCache?: boolean; @@ -15,6 +20,36 @@ export interface RestartRequest { recoveryBinding?: RecoveryBinding; } +export function kimiCandidateAdmissionForExecution(execution: MatrixExecution, companyId: string, now = Date.now()): AcpxCandidateAdmission | undefined { + if (execution.suite.id !== "kimi-qualification" || execution.profile.qualificationCandidate !== "kimi" + || QUALIFIED_ACPX_PROFILES.kimi.qualificationStatus !== "pending") return undefined; + if (!execution.suite.manualOnly || execution.profile.id !== "runner-acpx-kimi" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(companyId)) { + throw new Error("Kimi qualification requires an exact disposable company and manual profile"); + } + const target = execution.environment.expectedExecutionTarget.kind === "remote" ? "linux-x64" : `${process.platform}-${process.arch}`; + if (target !== "darwin-arm64" && target !== "linux-x64") throw new Error("Kimi fixture host is unsupported"); + const [platform, architecture] = target.split("-"); + return parseAcpxCandidateAdmission({ schema: "paperclip.acpx-candidate-admission.v1", companyId, agent: "kimi", + model: execution.profile.model, profileDigest: QUALIFIED_ACPX_PROFILES.kimi.commandDigest, + nativeClosureDigest: `sha256:${KIMI_DISTRIBUTION_CLOSURE_SHA256[target]}`, platform, architecture, + expiresAt: now + 60 * 60 * 1000 }, { companyId, agent: "kimi", model: execution.profile.model }, now); +} + +function selectedKimiCandidateAdmission(value: unknown, executionIds: readonly string[]): AcpxCandidateAdmission { + const executions = executionIds.map(id => runnerMatrix.find(row => row.id === id)); + if (!executions.length || executions.some(row => !row || row.suite.id !== "kimi-qualification" + || !row.suite.manualOnly || row.profile.id !== "runner-acpx-kimi" || row.profile.qualificationCandidate !== "kimi")) { + throw new Error("Kimi authority staging requires an explicit Kimi qualification selection"); + } + const record = value as Partial | null; + if (!record || typeof record.companyId !== "string" || !/^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(record.companyId)) throw new Error("Invalid Kimi fixture company"); + const admission = parseAcpxCandidateAdmission(value, { companyId: record.companyId, agent: "kimi", model: executions[0]!.profile.model }); + if (!executions.some(row => row!.environment.expectedExecutionTarget.kind === "remote" + ? admission.platform === "linux" && admission.architecture === "x64" + : admission.platform === process.platform && admission.architecture === process.arch)) throw new Error("Kimi fixture target does not match the selected execution"); + return admission; +} + export async function readRestartRequest(restartRequestPath: string, executionIds: readonly string[]): Promise { let encoded: string; try { @@ -38,6 +73,14 @@ export async function readRestartRequest(restartRequestPath: string, executionId ) { return null; } + const kimiAuthority = (value as { kimiCandidateAdmission?: unknown }).kimiCandidateAdmission; + const kimiCandidateAdmission = kimiAuthority === undefined ? undefined : selectedKimiCandidateAdmission(kimiAuthority, executionIds); + if (kimiCandidateAdmission && Object.keys(value).some(key => !["requestId", "kimiCandidateAdmission", "discardControllerCache"].includes(key))) { + throw new Error("Kimi startup authority staging cannot combine another restart operation"); + } + if (kimiCandidateAdmission && (value as { discardControllerCache?: unknown }).discardControllerCache === true) { + throw new Error("Kimi startup authority staging cannot replace controller state"); + } const preserveRunner = (value as { preserveRunner?: unknown }).preserveRunner; const controllerDowntimeMs = (value as { controllerDowntimeMs?: unknown }).controllerDowntimeMs; if (controllerDowntimeMs !== undefined && (!Number.isSafeInteger(controllerDowntimeMs) @@ -59,7 +102,7 @@ export async function readRestartRequest(restartRequestPath: string, executionId if (crashAfterResult !== undefined && (preserveRunner !== undefined || discardControllerCache)) { throw new Error("A finalization crash cannot preserve a running provider or discard its controller cache"); } - return { requestId, discardControllerCache, ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), + return { requestId, discardControllerCache, ...(kimiCandidateAdmission ? { kimiCandidateAdmission } : {}), ...(controllerDowntimeMs === undefined ? {} : { controllerDowntimeMs: Number(controllerDowntimeMs) }), ...(recoveryBinding === undefined ? {} : { recoveryBinding: parseRecoveryBinding(recoveryBinding) }), ...(controllerOutage === undefined ? {} : { controllerOutage: parseControllerOutage(controllerOutage) }), ...(preserveRunner === undefined ? {} : { preserveRunner: parseRestartRunnerIdentity(preserveRunner) }), diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index c67f29caa82..eba5deec9ff 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -42,6 +42,7 @@ const paperclipCli = path.join(repositoryRoot, "tests/runner-e2e/server-entry.ts const executionIds: string[] = JSON.parse(process.env.PAPERCLIP_RUNNER_E2E_EXECUTION_IDS ?? "[]"); const upgradeArtifacts = await readUpgradeArtifacts(process.env[upgradeArtifactKey]); let artifactIndex = 0; +let kimiCandidateAdmission: RestartRequest["kimiCandidateAdmission"]; const selectedExecutions = usesInstalledCli(process.env) || upgradeArtifacts ? executionIds.map(id => { const execution = runnerMatrix.find(row => row.id === id); if (!execution) throw new Error("Unknown installed CLI proof execution"); @@ -198,6 +199,7 @@ async function startServer() { const launch = installed ? installedReleaseLaunch(installed.entry) : null; // The serving process must report its baked-in revision, never a test-supplied SHA. const environment = { ...definedServerEnvironment }; + if (kimiCandidateAdmission) environment.PAPERCLIP_ACPX_CANDIDATE_ADMISSION = JSON.stringify(kimiCandidateAdmission); if (upgradeArtifacts) { delete environment.PAPERCLIP_BUILD_COMMIT; delete environment[upgradeArtifactKey]; @@ -342,7 +344,7 @@ async function writeRestartAck( await rename(temporaryAckPath, restartAckPath); } -async function restartServer({ requestId, preserveRunner, discardControllerCache, crashAfterResult, controllerOutage, recoveryBinding, controllerDowntimeMs = 0 }: RestartRequest) { +async function restartServer({ requestId, kimiCandidateAdmission: stagedKimiAdmission, preserveRunner, discardControllerCache, crashAfterResult, controllerOutage, recoveryBinding, controllerDowntimeMs = 0 }: RestartRequest) { activeRestartRequestId = requestId; appendLog(`\nRestart request ${requestId}: stopping Paperclip\n`); const previous = child; @@ -404,6 +406,7 @@ async function restartServer({ requestId, preserveRunner, discardControllerCache throw new Error("Wrapper shutdown interrupted the Paperclip restart"); } + if (stagedKimiAdmission) kimiCandidateAdmission = stagedKimiAdmission; appendLog(`Restart request ${requestId}: starting Paperclip\n`); const replacement = await startServer(); await waitForHealth(replacement); diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index 5180e7e8f4c..4dab61023ca 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -3,6 +3,7 @@ export const CREDENTIAL_NAMES = [ "ANTHROPIC_API_KEY", "OPENROUTER_API_KEY", "KIMI_MODEL_API_KEY", + "KIMI_API_KEY", "XAI_API_KEY", "GROK_AUTH_JSON", "CODEX_AUTH_JSON", @@ -48,6 +49,7 @@ export type RunnerTaskFlow = | "pi_native" | "pi_controls" | "native_active_stop" + | "kimi_native_lifecycle" | "copilot_protection" | "native_provider_loss" | "cursor_native"; @@ -92,7 +94,7 @@ export interface RunnerProfileFixture { | "openrouter_rankings_snapshot"; qualificationId: string; }; - qualificationCandidate?: "cursor" | "copilot" | "pi"; + qualificationCandidate?: "cursor" | "copilot" | "pi" | "kimi"; ranking?: { rank: number; canonicalModelId: string; @@ -208,6 +210,7 @@ export interface RunnerSuiteFixture { groups: readonly string[]; profiles: readonly RunnerProfileFixture[]; environments: readonly EnvironmentFixture[]; + environmentOverrides?: Readonly>>>; tasks: readonly RunnerTaskFixture[]; excludedExecutionIds?: readonly string[]; expectedMatrixSize: number; diff --git a/tests/runner-e2e/verified-provider-bundles.setup.mjs b/tests/runner-e2e/verified-provider-bundles.setup.mjs new file mode 100644 index 00000000000..cc034e4cd3f --- /dev/null +++ b/tests/runner-e2e/verified-provider-bundles.setup.mjs @@ -0,0 +1,12 @@ +import { spawnSync } from "node:child_process"; +import { fileURLToPath } from "node:url"; + +export default function setup() { + const root = fileURLToPath(new URL("../../", import.meta.url)); + const builder = fileURLToPath(new URL("../../packages/paperclip-runner/scripts/build-verified-provider-entrypoints.mjs", import.meta.url)); + const result = spawnSync(process.execPath, [builder], { cwd: root, stdio: "inherit" }); + if (result.error) throw new Error("Runner E2E provider bundle preparation failed", { cause: result.error }); + if (result.signal || result.status !== 0) { + throw new Error(`Runner E2E provider bundle preparation failed: ${result.signal ?? `exit ${result.status}`}`); + } +} diff --git a/tests/runner-e2e/vitest.config.ts b/tests/runner-e2e/vitest.config.ts index 831188aad0a..b77babcd5a8 100644 --- a/tests/runner-e2e/vitest.config.ts +++ b/tests/runner-e2e/vitest.config.ts @@ -4,6 +4,7 @@ export default defineConfig({ root: import.meta.dirname, test: { name: "runner-e2e-support", + globalSetup: ["./verified-provider-bundles.setup.mjs"], include: ["**/*.test.ts", "**/*.test.mjs"], // These use node:test and run separately in test:e2e:runner:unit. exclude: [...configDefaults.exclude, "native-completion-{checks,git-source,source-contract}.test.mjs", "artifact-budget.test.mjs", "chat-feature-testops-api.test.mjs", "chat-feature-testops-cli.test.mjs", "chat-feature-testops-results.test.mjs", "testops-chat-reporter.test.mjs"], diff --git a/tests/runner-e2e/web-server-command.ts b/tests/runner-e2e/web-server-command.ts index df8425695a1..14b184a2677 100644 --- a/tests/runner-e2e/web-server-command.ts +++ b/tests/runner-e2e/web-server-command.ts @@ -1,4 +1,16 @@ import path from "node:path"; +import type { MatrixExecution, RunnerTaskFlow } from "./types.js"; + +export function runnerE2EPlaywrightCaseTimeoutMs(flow: RunnerTaskFlow, deadlineMs: number) { + return (flow === "kimi_native_lifecycle" ? 2 * deadlineMs : deadlineMs) + 90_000; +} + +export function runnerE2EPlaywrightWatchdogMs(executions: readonly MatrixExecution[]) { + // Playwright starts its web server before any case timeout begins. + return executions.reduce((total, execution) => total + runnerE2EPlaywrightCaseTimeoutMs( + execution.task.flow, execution.task.attemptTimeoutMs[execution.environment.id], + ), 0) + 5 * 60_000; +} export const runnerE2EPlaywrightShutdownSignal = "SIGINT" as const; diff --git a/tests/runner-e2e/workflow-target.test.ts b/tests/runner-e2e/workflow-target.test.ts index 8abc9f95e3e..a4e1c57998b 100644 --- a/tests/runner-e2e/workflow-target.test.ts +++ b/tests/runner-e2e/workflow-target.test.ts @@ -65,3 +65,17 @@ describe("qualification image dependency correction", () => { } finally { await rm(directory, { recursive: true, force: true }); } }); }); + +describe("release image capacity prerequisite", () => { + it.each([34, 35])("checks the unchanged disk threshold for every image route: %s GiB", async available => { + const workflow = await readFile(new URL("../../.github/workflows/docker-runner-check.yml", import.meta.url), "utf8"); + const step = workflow.split(" - name: Prepare hosted image build capacity\n")[1]!.split("\n - ")[0]!; + expect(step).toContain("if: inputs.build_eval_image || inputs.publish_eval_image\n"); + expect(step).not.toContain("release_run_id"); + const script = step.split(" run: |\n")[1]!.replace(/^ {10}/gm, ""); + const result = spawnSync("bash", ["-c", `sudo() { test "$*" = "rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc"; }; df() { printf 'Avail\\n%sG\\n' "$AVAILABLE"; }; ${script}`], { + encoding: "utf8", env: { PATH: process.env.PATH, AVAILABLE: String(available) }, + }); + expect(result.status, result.stderr).toBe(available >= 35 ? 0 : 1); + }); +});