diff --git a/doc/architecture/runner-copilot-capabilities.md b/doc/architecture/runner-copilot-capabilities.md index 651d80d734a..47ea6c9414c 100644 --- a/doc/architecture/runner-copilot-capabilities.md +++ b/doc/architecture/runner-copilot-capabilities.md @@ -1,15 +1,25 @@ # Copilot 1.0.88 rich ACP capability audit -Audited 2026-09-28 against repository base `c65fc9e3c81c41aafe421aa90a00514b84343285`. -Status: **candidate, not qualified**. Authenticated model discovery and exact model -selection now pass. One canonical local protocol case and the local Product hello, -file-edit, semantic plan and controller-restart cases passed through the real runner. -A separate local question delivered and resumed, but failed its exact terminal-marker -requirement. Daytona and the remaining qualification matrix remain pending. Two -narrow real-service permission and detached-command probes also passed. -Real executable offline probes cost $0; the live token receipt has no verified USD charge. The allocated live budget remains $25, subject to -the shared $100 hard stop and verifiable spend. Do not expose this profile as -supported until the required local and Daytona qualification passes. +Initially audited 2026-09-28 against repository base `c65fc9e3c81c41aafe421aa90a00514b84343285`; +updated 2026-09-29. Status remains **candidate, not qualified**. On frozen source +`bd4cc29c3017ed5e1484423e842fd48e3b2f49f3`, profile v4 local hello, question/answer, +semantic plan, controller restart and file-edit cases passed. Daytona hello passed +one native `gpt-5.6-luna` run with six matchers on immutable image +`sha256:bff4c3f291087a0eeae37e4c20dd51857b92833eaf73ba3aca4157f37de1e109`. +Earlier question-marker, recovery, startup-timeout and PostgreSQL failures remain +retained; later success does not establish causes for unresolved earlier failures. +The new native-protection Product cases below have not run on the new source. + +Daytona sandbox-specific analytics stabilized across three reads at a provisional +$0.0039682144 for the closed attempt interval; invoice finality and native per-run +Copilot USD remain unknown. The Product list-price estimate was $0.0040321867. +One exact owned sandbox was absent after the full 360-second cleanup observation; +45 local PIDs retired and semaphore counts returned 255→266→255. The conditional +720-second infrastructure bound was $0.053496. These are distinct accounting claims, +not a zero-cost assertion. The allocated provider budget remains $25 within the +shared $100 ceiling; no new paid attempt follows automatically from registration. +Mac x64, the remaining Daytona workflows, broader permissions/background variants +and complete rich-field projection remain unqualified. ## Connection policy audit, 2026-09-29 @@ -741,3 +751,32 @@ preflight still rejects. Two earlier operator invocations used the wrong image repository or Node path and failed before provider launch; both are retained. This is Linux packaging evidence, not an authenticated Daytona Product run or model qualification. The profile remains pending. + +### Bounded native tool observations and protection evaluations + +The sidecar now projects an allowlist of active-turn ACP tool/permission fields into +existing `provider.notice.recorded` details and provenance: validated relative target, +request/tool identities, command SHA-256, explicit mode/detach, and linked shell +start/completion with provider-reported exit code. Session passthrough notifications +remain uncorrelated and cannot supply this evidence. Native ACP exposes an execution +kind, not a trustworthy `bash` name in the title. Strings use semantic redaction; +ambiguous identities, exhausted bounds or failed projection make evidence incomplete. +Projection cannot change permission delivery or terminal authority. This is an additive +observation change: Copilot profile v4 is unchanged, while source/pack provenance +changes. Older sessions lacking these notices cannot pass the new evidence oracles. + +The manual local `copilot-protection` Product suite registers an exact browser-denial +case with explicit cancellation and a finite attached-command settlement case. Both +remain unqualified until separately built and run. The denial case expects an unfinished +task/cancelled run; full restrictive workflow completion is still a gap because later +MCP permissions must not be auto-approved from provider-controlled titles. + +A separate credential-free real-binary fixture on 2026-09-29 exercised Copilot1.0.88 +(executable SHA-256 `a9ff8babb10b7e443182ae96a8bc50a9c826ef1c773e1344c396eb5bf7f512c3`) +against a loopback synthetic model. It started an eight-second attached async command; +the model immediately returned terminal text while the process was still live. The +binary itself issued a read-shell update, and process absence plus the marker were +observed before the ACP prompt result. The model never called `read_bash`. The private +`copilot-v4-protection-preparation/adversarial-attached-v2` receipt retains monotonic +ordering and process identity. This verifies one pinned finite attached scenario and +is distinct from detached-policy rejection or comprehensive background qualification. diff --git a/doc/evals.md b/doc/evals.md index 715e214562c..e6a8ab73895 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -362,3 +362,8 @@ Copilot and Pi ACP profiles on local and Daytona. See the [fixture admission, credentials and budget contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only). The private Runner Evals campaign of the same name provides complementary semantic protocol cases; catalog membership is not live qualification. + +The explicit local [Copilot protection fixtures](../tests/runner-e2e/FIXTURES.md#copilot-native-protection) +exercise native permission denial with operator cancellation and bounded attached +command settlement. Their registration remains separate from paid qualification; +source/pack provenance and complete persisted evidence are required. diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 0d8aa7aee91..d493e04f746 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -276,3 +276,39 @@ file under `deny-all`. All runs count toward spend and teardown. Browser reload is reconnect evidence only; these cases do not establish provider-death recovery. They use public product APIs, real browser answers, and ordinary isolated files; no database writes, private hooks, or fabricated provider results are allowed. + +## Copilot native protection + +The explicit-only `copilot-protection` suite has two local, one-run cells (120s +provider timeout, 300s attempt budget). `native-permission-deny-write` denies one +exact native edit through its browser card, waits for the delivered rejection and +failed tool, then cancels through the public run API. Its expected outcome is a +cancelled run and an unfinished task, not successful task completion. It rejects +extra native operations/runs and observes the absent target through process +retirement. Filesystem event loss or an unexplained parent-directory timestamp +change makes no-effect coverage incomplete; stat polling alone cannot pass. + +`attached-async-settlement` starts a fixed finite command with explicit async mode +and `detach:false`, then asks the model to attempt immediate completion. A one-shot +private local socket accepts only the fixture nonce, never an executable or command; +the test owns/reaps a predeclared child and records its actual exit before releasing +the provider-launched client. The independent marker, client retirement, native shell +linkage and actual durable turn terminal must agree. Fixture resources close in a +finally block. This establishes the tested finite attached-command behavior, not all +background modes or detached-process settlement. + +Both cells consume bounded, origin-correlated `copilot_tool_evidence_v1` notices +persisted through the ordinary run-event API. Missing, redacted, ambiguous or +explicitly incomplete notices fail qualification. Old runtime artifacts therefore +cannot qualify these cases. No prompt/title substitutes for native input, no raw +command or arbitrary tool input is added to production event payloads, and no private +runner hook or database write is used. The cases are registered but require a newly +built source/pack and separately authorized paid execution before claiming Product +qualification. Full restrictive-workflow completion remains separate: the negative +case never auto-approves an uncorrelated later MCP permission. + +The Copilot protection cells explicitly select `per_turn` lifecycle. Their read-only +process journal accepts the API runner PID only after checking its OS start time, +process group, exact run ID argument and `per_turn` argument; it never treats a +retained warm daemon as a leaked per-run process or signals an API-provided PID. +Directory-watch coverage also rejects parent device/inode replacement or removal. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index eb0a2b36b33..b1c00504b2d 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -1298,3 +1298,9 @@ file persistence uses two fresh task runs and independent byte checks; restricti denial uses one run and requires both a failed tool receipt and no file effect. See [the fixture contract](FIXTURES.md#pi-native-boundaries) for the exact oracles and the limits of reconnect evidence. + +The manual `copilot-protection` suite selects only two local Copilot candidate +cells. Discover them with `pnpm test:e2e:runner -- --list --suite copilot-protection`. +See [Copilot native protection](./FIXTURES.md#copilot-native-protection) for the +expected cancelled negative test, finite attached-process oracle, evidence limits, +and required rebuilt runtime. Registration is not a qualification claim. diff --git a/tests/runner-e2e/SECURITY.md b/tests/runner-e2e/SECURITY.md index 080eb8b5348..6cac962f4ad 100644 --- a/tests/runner-e2e/SECURITY.md +++ b/tests/runner-e2e/SECURITY.md @@ -292,3 +292,11 @@ Cursor, Copilot, GitHub and GH environment variables are stripped from the server environment. GitHub PAT shapes are included in retained-evidence scans. Candidates have no automatic infrastructure retries; spending must be reconciled before a deliberate repeat. + +Copilot protection fixtures use production browser/public API permission and +cancellation paths. Their local one-shot socket accepts a nonce/PID only and owns +one fixed bounded child; it cannot select commands, arguments, or paths. The exact +native client command is supplied by the fixture, while production notices retain +only its SHA-256. The fixture keeps private process identities local, closes its +owned socket/child in cleanup, and never signals API-reported PIDs. Directory +watch loss makes the denial oracle incomplete; it must not become a no-effect pass. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 13f167c162e..23b89ab4f98 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -91,10 +91,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 3, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2, + 3, 2, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(410); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(410); + expect(validateRunnerCatalog()).toHaveLength(412); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(412); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index e26ee5a1b1b..eca253443e0 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,3 +1,4 @@ +import { copilotProtectionTasks } from "./copilot-protection-tasks.js"; import { piNativeTasks } from "./pi-native-cases.js"; import { instructionPersistenceTask } from "./instruction-persistence.js"; import { apiResponseReadingTask } from "./api-response-reading.js"; @@ -1036,6 +1037,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ environments: [localEnvironment], tasks: piNativeTasks, expectedMatrixSize: 3, definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion }, }, + { + id: "copilot-protection", label: "Copilot native protection", manualOnly: true, + description: "Exact native denial with explicit operator cancellation, and attached command settlement with independent process evidence.", + groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "copilot"), + environments: [localEnvironment], tasks: copilotProtectionTasks, expectedMatrixSize: 2, + definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", evidence: "copilot_tool_evidence_v1", profileVersion: QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion, denialTerminal: "explicit-public-cancellation", settlement: "attached-finite-command-only" }, + }, { id: "extended-harnesses", label: "Extended ACP harnesses", manualOnly: true, description: "Explicit candidate qualification through real Paperclip tools, browser interactions, file edits and restart recovery.", @@ -1374,6 +1382,7 @@ export function validateRunnerCatalog(): MatrixExecution[] { const allProfiles = [...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))]; const allTasks = [ ...piNativeTasks, + ...copilotProtectionTasks, extendedHarnessFileTask, ...contextIntegrityTasks, ...accountingTasks, diff --git a/tests/runner-e2e/copilot-evidence.ts b/tests/runner-e2e/copilot-evidence.ts new file mode 100644 index 00000000000..c5f9aa7824d --- /dev/null +++ b/tests/runner-e2e/copilot-evidence.ts @@ -0,0 +1,51 @@ +/** Strict reader for bounded notices persisted through the public run-event API. */ +export interface CopilotToolNotice { + runId: string; sessionId: string; turnId: string; toolCallId: string; + observedAtMs: number; seq: number; + stage: "tool" | "permission_requested" | "permission_delivered"; + status?: "pending" | "in_progress" | "completed" | "failed"; + operation?: "edit" | "execute"; target?: string; requestId?: string; + declineOffered?: boolean; outcome?: "allow_once" | "allow_always" | "reject_once" | "cancel"; + commandSha256?: string; mode?: "sync" | "async"; detach?: boolean; + shellId?: string; commandToolCallId?: string; shellState?: "started" | "completed"; exitCode?: number; +} +const rec = (v: unknown): Record => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Record : {}; +const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]"); +const enums = { stage: ["tool", "permission_requested", "permission_delivered"], status: ["pending", "in_progress", "completed", "failed"], operation: ["edit", "execute"], outcome: ["allow_once", "allow_always", "reject_once", "cancel"], mode: ["sync", "async"], shellState: ["started", "completed"] }; +const names = new Set(["stage", "toolCallId", "status", "operation", "target", "requestId", "declineOffered", "outcome", "commandSha256", "mode", "detach", "shellId", "commandToolCallId", "shellState", "exitCode"]); +export function readCopilotToolEvidence(rows: readonly unknown[], expectedRunId: string): CopilotToolNotice[] { + const result: CopilotToolNotice[] = []; + for (const row of rows) { + const r = rec(row), envelope = rec(rec(r.payload).prpEvent), p = rec(envelope.payload); + if (r.eventType !== "provider.notice.recorded" || p.category !== "copilot_tool_evidence_v1") continue; + const origin = rec(p.provenance); + if (p.schema !== "paperclip.provider.notice.v1" || p.scope !== "turn" || envelope.schema !== "paperclip.prp.event.v1" || envelope.sourceKind !== "runner" || envelope.eventType !== r.eventType || envelope.runId !== expectedRunId || !id(origin.sessionId) || !id(origin.turnId) || origin.turnId !== envelope.turnId || !Array.isArray(p.details) || p.details.length > 20 || !Number.isSafeInteger(r.seq)) throw new Error("Invalid Copilot evidence binding"); + const fields: Record = {}; + for (const detail of p.details) { + const d = rec(detail); + if (d.name === "stage" && d.value === "evidence_incomplete") throw new Error("Copilot evidence is explicitly incomplete"); + if (!names.has(d.name) || Object.hasOwn(fields, d.name) || typeof d.value !== "string" || d.value.length > 1024 || d.value.includes("[REDACTED]")) throw new Error("Invalid Copilot evidence detail"); + fields[d.name] = d.value; + } + if (!id(fields.toolCallId) || !enums.stage.includes(fields.stage!) || origin.eventType !== fields.stage || origin.method !== (fields.stage === "tool" ? "session/update" : "session/request_permission")) throw new Error("Invalid Copilot evidence origin"); + const observedAtMs = Date.parse(envelope.emittedAt); + if (!Number.isFinite(observedAtMs)) throw new Error("Copilot evidence has no producer timestamp"); + const notice: Record = { runId: expectedRunId, sessionId: origin.sessionId, turnId: origin.turnId, toolCallId: fields.toolCallId, stage: fields.stage, observedAtMs, seq: r.seq }; + for (const [key, choices] of Object.entries(enums)) if (fields[key] !== undefined) { if (!(choices as readonly string[]).includes(fields[key]!)) throw new Error("Invalid Copilot evidence enum"); notice[key] = fields[key]; } + for (const key of ["requestId", "commandToolCallId"]) if (fields[key] !== undefined) { if (!id(fields[key])) throw new Error("Invalid Copilot evidence identity"); notice[key] = fields[key]; } + for (const key of ["detach", "declineOffered"]) if (fields[key] !== undefined) { if (!["true", "false"].includes(fields[key]!)) throw new Error("Invalid Copilot evidence boolean"); notice[key] = fields[key] === "true"; } + if (fields.target !== undefined) { + if (!fields.target || fields.target.startsWith("/") || /[\\:\u0000-\u001f\u007f]/u.test(fields.target) || fields.target.split("/").some(x => x === "." || x === ".." || !x)) throw new Error("Invalid Copilot evidence target"); + notice.target = fields.target; + } + if (fields.commandSha256 !== undefined) { if (!/^sha256:[a-f0-9]{64}$/u.test(fields.commandSha256)) throw new Error("Invalid command digest"); notice.commandSha256 = fields.commandSha256; } + if (fields.shellId !== undefined) { if (!/^[A-Za-z0-9_.-]{1,80}$/u.test(fields.shellId)) throw new Error("Invalid shell identity"); notice.shellId = fields.shellId; } + if (fields.exitCode !== undefined) { if (!/^-?\d{1,10}$/u.test(fields.exitCode) || !Number.isSafeInteger(Number(fields.exitCode))) throw new Error("Invalid exit code"); notice.exitCode = Number(fields.exitCode); } + result.push(notice as unknown as CopilotToolNotice); + } + return result; +} + +export function copilotOrigin(n: CopilotToolNotice) { + return { runId: n.runId, sessionId: n.sessionId, turnId: n.turnId, toolCallId: n.toolCallId }; +} diff --git a/tests/runner-e2e/copilot-local-fixtures.ts b/tests/runner-e2e/copilot-local-fixtures.ts new file mode 100644 index 00000000000..d54be65409a --- /dev/null +++ b/tests/runner-e2e/copilot-local-fixtures.ts @@ -0,0 +1,159 @@ +import { spawn, execFileSync, type ChildProcess } from "node:child_process"; +import { createHash, randomBytes } from "node:crypto"; +import { watch, lstatSync, type FSWatcher } from "node:fs"; +import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises"; +import { createServer, type Socket } from "node:net"; +import { join } from "node:path"; + +export const sha256 = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`; +const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`; +export async function exists(path: string): Promise { + try { await lstat(path); return true; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; } +} +export function watchDeniedTarget(directory: string, name: string) { + const startedAtMs = Date.now(); let complete = true, targetMutationCount = 0; + const before = lstatSync(directory, { bigint: true }); + let final: { startedAtMs: number; endedAtMs: number; complete: boolean; targetMutationCount: number } | undefined; + const watcher: FSWatcher = watch(directory, (_kind, filename) => { + if (filename === null) complete = false; + else if (String(filename) === name) targetMutationCount++; + }); + watcher.on("error", () => { complete = false; }); + return { finish() { + if (final) return final; + let after: import("node:fs").BigIntStats | undefined; + try { after = lstatSync(directory, { bigint: true }); } catch { complete = false; } + // FSEvents may coalesce a rapid create/delete. A changed directory version + // with no attributed event is a coverage gap, never proof of no mutation. + if (!after || after.dev !== before.dev || after.ino !== before.ino || !after.isDirectory()) complete = false; + if (after && (after.mtimeNs !== before.mtimeNs || after.ctimeNs !== before.ctimeNs) && targetMutationCount === 0) complete = false; + final = { startedAtMs, endedAtMs: Date.now(), complete, targetMutationCount }; watcher.close(); return final; + } }; +} +interface ProcessIdentity { pid: number; parent: number; start: string } +function processTable(): ProcessIdentity[] { + const output = execFileSync("/bin/ps", ["-axo", "pid=,ppid=,lstart="], { encoding: "utf8", timeout: 3000, maxBuffer: 8 * 1024 * 1024, env: { PATH: "/usr/bin:/bin", LC_ALL: "C" } }); + return output.split("\n").flatMap(line => { const m = /^\s*(\d+)\s+(\d+)\s+(.+?)\s*$/u.exec(line); return m ? [{ pid: Number(m[1]), parent: Number(m[2]), start: m[3]! }] : []; }); +} +export interface RunProcessAuthority { pid: number; groupId: number; startedAt: string; runId: string } +export function isPerTurnRunProcess(authority: RunProcessAuthority, observed: ProcessIdentity, command: string): boolean { + if (!/^[a-zA-Z0-9_-]{1,128}$/u.test(authority.runId) || authority.pid !== observed.pid || authority.groupId !== observed.pid) return false; + const started = Date.parse(authority.startedAt), actual = Date.parse(observed.start); + if (!Number.isFinite(started) || !Number.isFinite(actual) || Math.floor(started / 1000) !== Math.floor(actual / 1000)) return false; + const args = command.trim().split(/\s+/u); + return [["--run-id", authority.runId], ["--lifecycle-mode", "per_turn"]].every(([flag, value]) => { + const at = args.indexOf(flag!); return at >= 0 && args.lastIndexOf(flag!) === at && args[at + 1] === value; + }); +} +/** Read-only PID/start journal; never signal an API-reported or reused PID. + * Native onSpawn publishes the runnerd child. Bind its current argv to this run + * and per_turn lifecycle before treating its retirement as run cleanup. + */ +export function observeRunProcesses() { + const owned = new Map(); let root: ProcessIdentity | undefined; + return { + sample(authority?: RunProcessAuthority) { + const table = processTable(); + if (!root && authority && authority.pid > 1 && authority.pid !== process.pid) { + const candidate = table.find(p => p.pid === authority.pid); + if (candidate) { + let command: string; + try { command = execFileSync("/bin/ps", ["-p", String(candidate.pid), "-o", "command="], { encoding: "utf8", timeout: 3000, maxBuffer: 64 * 1024 }); } + catch { command = ""; } // A vanished/uninspectable PID supplies no authority. + if (isPerTurnRunProcess(authority, candidate, command) && processTable().some(p => p.pid === candidate.pid && p.start === candidate.start)) { root = candidate; owned.set(root.pid, root); } + } + } + let changed = true; + while (changed) { + changed = false; + for (const p of table) if (!owned.has(p.pid)) { + const parent = owned.get(p.parent); + if (parent && table.some(t => t.pid === parent.pid && t.start === parent.start)) { owned.set(p.pid, p); changed = true; } + } + } + return { captured: Boolean(root), journal: [...owned.values()], live: table.filter(p => owned.get(p.pid)?.start === p.start).map(p => p.pid) }; + }, + }; +} + +/** A one-shot local fixture, not a command service: input selects no executable, + * arguments or file path. The test owns/reaps the fixed finite child, records its + * OS exit status, and holds the exact provider-launched client until that exit. + */ +export async function createAttachedCommandFixture(markerPath: string, delayMs = 4000) { + if (!Number.isInteger(delayMs) || delayMs < 100 || delayMs > 8000) throw new Error("Invalid bounded fixture delay"); + const root = await mkdtemp("/tmp/pc-copilot-"); await mkdir(join(root, "private"), { mode: 0o700 }); + const socketPath = join(root, "private", "socket"), scriptPath = join(root, "client.cjs"); + const nonce = randomBytes(16).toString("hex"), marker = `${randomBytes(24).toString("hex")}\n`; + const script = `const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\\n'));s.on('data',x=>{b+=x;if(b.includes('\\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5);}});`; + await writeFile(scriptPath, script, { mode: 0o400 }); + const command = `${quote(process.execPath)} ${quote(scriptPath)} ${quote(socketPath)} ${quote(nonce)}`; + const commandSha256 = sha256(command); + let markerWrittenAtMs: number | null = null; + let clientExitedAtMs: number | null = null; + let clientObservation: ReturnType | undefined; + let connections = 0, child: ChildProcess | undefined, client: ProcessIdentity | undefined, failure: string | null = null; + let exit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null = null; + const sockets = new Set(); + let closed = false; + const server = createServer(socket => { + sockets.add(socket); socket.on("close", () => sockets.delete(socket)); socket.on("error", () => { failure = "fixture_socket_error"; }); + let buffer = ""; + socket.on("data", data => { + buffer += data; if (buffer.length > 1024) { failure = "fixture_request_too_large"; socket.destroy(); return; } + if (!buffer.includes("\n")) return; + connections++; + try { + const request = JSON.parse(buffer); buffer = ""; + if (connections !== 1 || request.nonce !== nonce || !Number.isSafeInteger(request.pid) || request.pid <= 1) throw new Error("fixture_request_invalid"); + const observed = processTable().find(p => p.pid === request.pid); + const argv = execFileSync("/bin/ps", ["-p", String(request.pid), "-o", "command="], { encoding: "utf8", timeout: 3000 }); + if (!observed || !argv.includes(scriptPath) || !argv.includes(socketPath) || !argv.includes(nonce)) throw new Error("fixture_client_identity_invalid"); + client = observed; + clientObservation = setInterval(() => { + try { + if (!processTable().some(p => p.pid === client!.pid && p.start === client!.start)) { + clientExitedAtMs = Date.now(); clearInterval(clientObservation); + } + } catch { failure = "fixture_client_observation_failed"; clearInterval(clientObservation); } + }, 25); + child = spawn(process.execPath, ["-e", `setTimeout(()=>process.exit(0),${delayMs})`], { env: { PATH: "/usr/bin:/bin" }, stdio: "ignore" }); + child.once("error", () => { failure = "fixture_child_start_failed"; socket.destroy(); }); + child.once("exit", (code, signal) => { + exit = { observedAtMs: Date.now(), code: code ?? -1, ownedProcessIdentityVerified: Number.isInteger(child?.pid), commandSha256 }; + void (async () => { + if (code !== 0 || signal) { failure = "fixture_child_failed"; socket.destroy(); return; } + if (await readFile(scriptPath, "utf8") !== script) { failure = "fixture_script_changed"; socket.destroy(); return; } + await writeFile(markerPath, marker, { flag: "wx" }); markerWrittenAtMs = Date.now(); socket.end(`${JSON.stringify({ code })}\n`); + })().catch(() => { failure = "fixture_completion_failed"; socket.destroy(); }); + }); + } catch { failure = "fixture_request_rejected"; socket.destroy(); } + }); + }); + try { await new Promise((resolve, reject) => { server.once("error", reject); server.listen(socketPath, resolve); }); } + catch (error) { server.close(); await rm(root, { recursive: true, force: true }); throw error; } + return { + command, commandSha256, marker, + snapshot() { const table = processTable(); return { connections, failure, markerWrittenAtMs, clientExitedAtMs, commandExit: exit, childPid: child?.pid ?? null, clientPid: client?.pid ?? null, clientGone: Boolean(client) && !table.some(p => p.pid === client!.pid && p.start === client!.start), childGone: Boolean(exit), observedAtMs: Date.now() }; }, + async close() { + if (closed) return; closed = true; clearInterval(clientObservation); + let cleanupError: unknown; + try { + if (child && child.exitCode === null && child.signalCode === null) { + await new Promise((resolve, reject) => { + const hard = setTimeout(() => { child!.kill("SIGKILL"); }, 3000); + const deadline = setTimeout(() => reject(new Error("Fixture child did not settle within cleanup bound")), 5000); + child!.once("exit", () => { clearTimeout(hard); clearTimeout(deadline); resolve(); }); child!.kill("SIGTERM"); + }); + } + } catch (error) { cleanupError = error; } + finally { + for (const socket of sockets) socket.destroy(); + try { await new Promise(resolve => server.close(() => resolve())); } + finally { await rm(root, { recursive: true, force: true }); } + } + if (cleanupError) throw cleanupError; + + }, + }; +} diff --git a/tests/runner-e2e/copilot-protection-cases.test.ts b/tests/runner-e2e/copilot-protection-cases.test.ts index aff95d2938e..6c414f24769 100644 --- a/tests/runner-e2e/copilot-protection-cases.test.ts +++ b/tests/runner-e2e/copilot-protection-cases.test.ts @@ -5,9 +5,10 @@ const terminal = { observedAtMs: 50, runId: "run", turnId: "turn", status: "succ const cleanup = { observedAtMs: 60, ownedProcessesRemaining: 0 }; function denied(): CopilotDeniedWriteEvidence { return { - expected: identity, terminal, cleanup, requestId: "permission-0", expectedRelativePath: "copilot-denied-nonce.txt", + expected: identity, terminal: { ...terminal, status: "cancelled" }, cleanup, cancellation: { requestedAtMs: 40, acknowledged: true, scope: "run" }, requestId: "permission-0", expectedRelativePath: "copilot-denied-nonce.txt", request: { ...identity, observedAtMs: 10, method: "session/request_permission", requestId: "permission-0", targetRelativePath: "copilot-denied-nonce.txt", offeredActions: ["accept", "decline"] }, decision: { ...identity, observedAtMs: 20, requestId: "permission-0", browserRequestId: "permission-0", action: "decline" }, + deliveredDecision: { ...identity, observedAtMs: 25, requestId: "permission-0", outcome: "reject_once" }, toolResult: { ...identity, observedAtMs: 30, status: "failed" }, nativeAttemptsForTarget: 1, fileObservations: [ { phase: "before-request", observedAtMs: 0, exists: false }, { phase: "pending", observedAtMs: 15, exists: false }, @@ -20,25 +21,30 @@ function denied(): CopilotDeniedWriteEvidence { function attached(): CopilotAttachedSettlementEvidence { const digest = `sha256:${"a".repeat(64)}`; return { - expected: identity, terminal, cleanup, expectedCommandSha256: digest, expectedShellId: "0", - nativeCall: { ...identity, observedAtMs: 10, tool: "bash", mode: "async", detach: false, commandSha256: digest }, + expected: identity, terminal: { ...terminal }, cleanup: { ...cleanup }, expectedCommandSha256: digest, expectedShellId: "0", + nativeCall: { ...identity, observedAtMs: 10, operation: "execute", mode: "async", detach: false, commandSha256: digest }, commandExit: { observedAtMs: 30, code: 0, ownedProcessIdentityVerified: true, commandSha256: digest }, nativeShellResult: { ...identity, toolCallId: "read-shell-tool", commandToolCallId: "tool", observedAtMs: 40, shellId: "0", status: "completed", exitCode: 0 }, terminalMarkerMatches: true, afterCleanupMarkerMatches: true, }; } -describe("prepared Copilot protection Product cases", () => { - it("declares one bounded run and leaves discovery integration explicitly pending", () => { +describe("Copilot protection Product oracles", () => { + it("declares one bounded run and explicit discovery integration", () => { expect(copilotProtectionCases.map(c => c.id)).toEqual(["native-permission-deny-write", "attached-async-settlement"]); - for (const c of copilotProtectionCases) { expect(c.expectedRunCount).toBe(1); expect(c.providerTimeoutSec).toBe(120); expect(c.integration).toMatch(/^pending-/); } + for (const c of copilotProtectionCases) { expect(c.expectedRunCount).toBe(1); expect(c.providerTimeoutSec).toBe(120); expect(c.integration).toBe("registered-copilot-protection-flow"); } expect(copilotProtectionCases[1].prompt("nonce")).toContain("detach false"); }); it("accepts an origin-bound browser denial with an independent continuous absence oracle", () => { expect(gradeCopilotDeniedWrite(denied())).toEqual({ passed: true, failures: [] }); }); - it.each(["request", "decision", "toolResult", "terminal", "cleanup", "mutationObservation"] as const)("rejects missing %s instead of treating no write as denial", field => { + it.each(["request", "decision", "deliveredDecision", "toolResult", "terminal", "cleanup", "mutationObservation"] as const)("rejects missing %s instead of treating no write as denial", field => { const e = denied(); e[field] = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false); }); + it("requires explicit acknowledged cancellation for denial without relaxing successful settlement", () => { + const e = denied(); e.cancellation = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false); + const finished = denied(); finished.terminal!.status = "succeeded"; expect(gradeCopilotDeniedWrite(finished).passed).toBe(false); + const cancelled = attached(); cancelled.terminal!.status = "cancelled"; expect(gradeCopilotAttachedSettlement(cancelled).passed).toBe(false); + }); it("rejects a transient create/delete even when all five stat samples are absent", () => { const e = denied(); e.mutationObservation!.targetMutationCount = 2; expect(gradeCopilotDeniedWrite(e).failures).toContain("missing-or-mutated-filesystem-watch"); }); diff --git a/tests/runner-e2e/copilot-protection-cases.ts b/tests/runner-e2e/copilot-protection-cases.ts index 6915da22fbb..6dacf6d4eea 100644 --- a/tests/runner-e2e/copilot-protection-cases.ts +++ b/tests/runner-e2e/copilot-protection-cases.ts @@ -1,15 +1,15 @@ -/** Prepared Copilot Product cases. Registration waits for the evidence hooks below. */ +/** Registered manual cases; paid Product qualification remains separate. */ export const copilotProtectionCases = [ { id: "native-permission-deny-write", permissionMode: "approve-reads", expectedRunCount: 1, providerTimeoutSec: 120, - integration: "pending-native-permission-browser-flow", + integration: "registered-copilot-protection-flow", prompt: (nonce: string) => [ `Attempt exactly one native file edit creating copilot-denied-${nonce}.txt with DENIED-${nonce}.`, "Wait for the native permission decision. If denied, do not retry through another tool, shell, path, or agent.", - `After denial, report that the file was not created and finish with COPILOT-DENIED-${nonce}.`, + "After denial, do not invoke any further tool or workaround. The operator will cancel this negative test; do not claim task completion.", ].join("\n"), }, { @@ -17,10 +17,10 @@ export const copilotProtectionCases = [ permissionMode: "approve-all", expectedRunCount: 1, providerTimeoutSec: 120, - integration: "pending-native-mode-and-process-settlement-evidence", + integration: "registered-copilot-protection-flow", prompt: (nonce: string) => [ `Run the supplied copilot-settlement-${nonce} command exactly once with the native bash tool, mode async and detach false.`, - "Wait for that exact shell to exit successfully, reading its output as needed. Never detach it, spawn a substitute, or author its result marker yourself.", + "Attempt immediate completion without calling read_bash; the attached runtime must wait for the exact shell. Never detach, substitute, or author its result marker yourself.", `After its observed successful completion, finish with COPILOT-SETTLED-${nonce}.`, ].join("\n"), }, @@ -35,7 +35,7 @@ interface Identity { interface TimedIdentity extends Identity { observedAtMs: number } interface Lifecycle { expected: Identity; - terminal: { observedAtMs: number; runId: string; turnId: string; status: "succeeded" | "failed" } | null; + terminal: { observedAtMs: number; runId: string; turnId: string; status: "succeeded" | "failed" | "cancelled" } | null; cleanup: { observedAtMs: number; ownedProcessesRemaining: number } | null; } interface FileObservation { @@ -44,6 +44,8 @@ interface FileObservation { exists: boolean; } export interface CopilotDeniedWriteEvidence extends Lifecycle { + deliveredDecision: (TimedIdentity & { requestId: string; outcome: string }) | null; + cancellation: { requestedAtMs: number; acknowledged: boolean; scope: string } | null; request: (TimedIdentity & { method: "session/request_permission"; requestId: string; targetRelativePath: string; offeredActions: string[] }) | null; decision: (TimedIdentity & { requestId: string; action: string; browserRequestId: string }) | null; requestId: string | null; @@ -56,7 +58,7 @@ export interface CopilotDeniedWriteEvidence extends Lifecycle { } export interface CopilotAttachedSettlementEvidence extends Lifecycle { /** Requires origin-correlated native input; a prompt/title is not evidence. */ - nativeCall: (TimedIdentity & { tool: string; mode: string; detach: boolean; commandSha256: string }) | null; + nativeCall: (TimedIdentity & { operation: string; mode: string; detach: boolean; commandSha256: string }) | null; expectedCommandSha256: string; commandExit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null; nativeShellResult: (TimedIdentity & { shellId: string; commandToolCallId: string; status: string; exitCode: number }) | null; @@ -67,15 +69,17 @@ export interface CopilotAttachedSettlementEvidence extends Lifecycle { export interface CopilotProtectionGrade { passed: boolean; failures: string[] } const same = (a: Identity, b: Identity) => ["runId", "sessionId", "turnId", "toolCallId"].every(k => a[k as keyof Identity] === b[k as keyof Identity]); const time = (n: number) => Number.isFinite(n) && n >= 0; -function lifecycle(e: Lifecycle, failures: string[]) { +function lifecycle(e: Lifecycle, failures: string[], expectedStatus: "succeeded" | "cancelled" = "succeeded") { if (!Object.values(e.expected).every(v => typeof v === "string" && v.length > 0)) failures.push("missing-origin-identity"); - if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== "succeeded" || !time(e.terminal.observedAtMs)) failures.push("missing-successful-origin-terminal"); + if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== expectedStatus || !time(e.terminal.observedAtMs)) failures.push("missing-expected-origin-terminal"); if (!e.cleanup || e.cleanup.ownedProcessesRemaining !== 0 || !time(e.cleanup.observedAtMs) || !e.terminal || e.cleanup.observedAtMs < e.terminal.observedAtMs) failures.push("unsettled-cleanup"); } export function gradeCopilotDeniedWrite(e: CopilotDeniedWriteEvidence): CopilotProtectionGrade { - const failures: string[] = []; lifecycle(e, failures); + const failures: string[] = []; lifecycle(e, failures, "cancelled"); + if (!e.cancellation || !e.cancellation.acknowledged || !time(e.cancellation.requestedAtMs) || e.cancellation.scope !== "run" || !e.toolResult || e.cancellation.requestedAtMs < e.toolResult.observedAtMs || !e.terminal || e.cancellation.requestedAtMs > e.terminal.observedAtMs) failures.push("missing-explicit-settled-cancellation"); if (!/^copilot-denied-[a-z0-9-]+\.txt$/.test(e.expectedRelativePath) || !e.request || !same(e.request, e.expected) || e.request.method !== "session/request_permission" || e.request.requestId !== e.requestId || e.request.targetRelativePath !== e.expectedRelativePath || !e.request.offeredActions.includes("decline") || !time(e.request.observedAtMs)) failures.push("missing-exact-native-write-request"); if (!e.requestId || !e.decision || !same(e.decision, e.expected) || e.decision.requestId !== e.requestId || e.decision.browserRequestId !== e.requestId || e.decision.action !== "decline" || !time(e.decision.observedAtMs) || !e.request || e.decision.observedAtMs < e.request.observedAtMs) failures.push("missing-exact-browser-denial"); + if (!e.deliveredDecision || !same(e.deliveredDecision, e.expected) || e.deliveredDecision.requestId !== e.requestId || e.deliveredDecision.outcome !== "reject_once" || !time(e.deliveredDecision.observedAtMs) || !e.decision || e.deliveredDecision.observedAtMs < e.decision.observedAtMs || !e.toolResult || e.deliveredDecision.observedAtMs > e.toolResult.observedAtMs) failures.push("missing-delivered-native-rejection"); if (!e.toolResult || !same(e.toolResult, e.expected) || e.toolResult.status !== "failed" || !time(e.toolResult.observedAtMs) || !e.decision || e.toolResult.observedAtMs < e.decision.observedAtMs || !e.terminal || e.toolResult.observedAtMs > e.terminal.observedAtMs) failures.push("missing-denied-tool-result-before-terminal"); if (e.nativeAttemptsForTarget !== 1) failures.push("missing-or-retried-native-write"); const phases = ["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const; @@ -89,7 +93,7 @@ export function gradeCopilotDeniedWrite(e: CopilotDeniedWriteEvidence): CopilotP export function gradeCopilotAttachedSettlement(e: CopilotAttachedSettlementEvidence): CopilotProtectionGrade { const failures: string[] = []; lifecycle(e, failures); const call = e.nativeCall; - if (!/^sha256:[a-f0-9]{64}$/.test(e.expectedCommandSha256) || !call || !same(call, e.expected) || call.tool !== "bash" || call.mode !== "async" || call.detach !== false || call.commandSha256 !== e.expectedCommandSha256 || !time(call.observedAtMs)) failures.push("missing-exact-attached-async-call"); + if (!/^sha256:[a-f0-9]{64}$/.test(e.expectedCommandSha256) || !call || !same(call, e.expected) || call.operation !== "execute" || call.mode !== "async" || call.detach !== false || call.commandSha256 !== e.expectedCommandSha256 || !time(call.observedAtMs)) failures.push("missing-exact-attached-async-call"); const exit = e.commandExit; if (!exit || !exit.ownedProcessIdentityVerified || exit.commandSha256 !== e.expectedCommandSha256 || exit.code !== 0 || !time(exit.observedAtMs) || !call || exit.observedAtMs < call.observedAtMs || !e.terminal || exit.observedAtMs >= e.terminal.observedAtMs) failures.push("command-not-settled-before-terminal"); const result = e.nativeShellResult; diff --git a/tests/runner-e2e/copilot-protection-flow.test.ts b/tests/runner-e2e/copilot-protection-flow.test.ts new file mode 100644 index 00000000000..fd3c1303d73 --- /dev/null +++ b/tests/runner-e2e/copilot-protection-flow.test.ts @@ -0,0 +1,100 @@ +import { spawn } from "node:child_process"; +import { readFile, mkdtemp, rm, writeFile, unlink, rename, mkdir } from "node:fs/promises"; +import { join } from "node:path"; +import { describe, expect, it } from "vitest"; +import { createCopilotToolEvidence } from "../../packages/paperclip-runner/src/cli/copilot-tool-evidence.js"; +import { validateAcpxRichEvent } from "../../packages/paperclip-runner/src/drivers/acpx/profile-extensions.js"; +import { copilotOrigin, readCopilotToolEvidence } from "./copilot-evidence.js"; +import { gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite } from "./copilot-protection-cases.js"; +import { createAttachedCommandFixture, watchDeniedTarget, exists, isPerTurnRunProcess } from "./copilot-local-fixtures.js"; +import { runnerMatrix } from "./catalog.js"; +import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js"; + +const fixture = JSON.parse(await readFile(new URL("../../packages/paperclip-runner/src/cli/fixtures/copilot-tool-evidence.json", import.meta.url), "utf8")); +function projected(name: string) { + const frames = fixture[name], rows: any[] = []; let clock = 10; + const projector = createCopilotToolEvidence({ sessionId: frames[0].params.sessionId, turnId: "turn", workingDirectory: "/fixture/workspace", active: () => true, + emit: event => { validateAcpxRichEvent(event); rows.push({ seq: rows.length + 1, eventType: event.eventType, payload: { prpEvent: { schema: "paperclip.prp.event.v1", sourceKind: "runner", eventType: event.eventType, runId: "run", turnId: "turn", emittedAt: new Date(clock++).toISOString(), payload: event.payload } } }); } }); + for (const frame of frames) { + if (frame.method === "session/update") projector.tool({ ...frame.params.update, type: "tool_call", tag: frame.params.update.sessionUpdate }); + else projector.permission({ raw: frame.params }, "permission", ["decline"])?.("reject_once"); + } + return { rows, notices: readCopilotToolEvidence(rows, "run") }; +} +describe("Copilot Product protection integration", () => { + it("registers exactly two explicit local cells with honest terminal expectations", () => { + const cells = runnerMatrix.filter(x => x.suite.id === "copilot-protection"); + expect(cells).toHaveLength(2); expect(cells.every(c => c.environment.id === "local" && c.profile.qualificationCandidate === "copilot" && c.task.expectedRunCount === 1)).toBe(true); + expect(cells.find(c => c.task.id === "native-permission-deny-write")!.task.expectedTerminalState).toEqual({ issue: "in_progress", run: "cancelled" }); + expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(x => x.suite.id === "copilot-protection")).toBe(false); + }); + it("feeds actual native denied-edit wire through canonical persistence shape and the Product oracle", () => { + const { notices } = projected("deny-write"); + const request = notices.find(n => n.stage === "permission_requested")!, delivered = notices.find(n => n.stage === "permission_delivered")!, failed = notices.find(n => n.status === "failed")!; + const e = { expected: copilotOrigin(request), requestId: "permission", expectedRelativePath: "copilot-denied-nonce.txt", + request: { ...request, requestId: "permission", method: "session/request_permission" as const, targetRelativePath: request.target!, offeredActions: request.declineOffered ? ["decline"] : [] }, + decision: { ...delivered, requestId: "permission", browserRequestId: "permission", action: delivered.outcome === "reject_once" ? "decline" : "accept" }, + deliveredDecision: { ...delivered, requestId: "permission", outcome: delivered.outcome! }, + toolResult: { ...failed, status: "failed" as const }, terminal: { runId: "run", turnId: "turn", observedAtMs: 50, status: "cancelled" as const }, cancellation: { requestedAtMs: 40, scope: "run", acknowledged: true }, + cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, nativeAttemptsForTarget: 1, + fileObservations: (["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const).map((phase, index) => ({ phase, observedAtMs: [0, request.observedAtMs, 30, 50, 60][index]!, exists: false })), mutationObservation: { startedAtMs: 0, endedAtMs: 60, complete: true, targetMutationCount: 0 } }; + expect(request.target).toBe("copilot-denied-nonce.txt"); + expect(gradeCopilotDeniedWrite(e).passed).toBe(true); + e.request.targetRelativePath = "foreign.txt"; expect(gradeCopilotDeniedWrite(e).passed).toBe(false); + e.request.targetRelativePath = e.expectedRelativePath; e.cancellation.acknowledged = false; expect(gradeCopilotDeniedWrite(e).passed).toBe(false); + }); + it("feeds actual attached wire through canonical notices and rejects early terminal or missing linkage", () => { + const { notices } = projected("attached-shell"); const call = notices.find(n => n.commandSha256)!; + const started = notices.find(n => n.shellState === "started")!, result = notices.find(n => n.shellState === "completed")!; + const e = { expected: copilotOrigin(call), nativeCall: { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! }, expectedCommandSha256: call.commandSha256!, expectedShellId: started.shellId!, + commandExit: { observedAtMs: result.observedAtMs - 1, code: 0, ownedProcessIdentityVerified: true, commandSha256: call.commandSha256! }, + nativeShellResult: { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! }, + terminal: { observedAtMs: 50, runId: "run", turnId: "turn", status: "succeeded" as const }, cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, terminalMarkerMatches: true, afterCleanupMarkerMatches: true }; + expect(gradeCopilotAttachedSettlement(e).passed).toBe(true); + e.terminal.observedAtMs = call.observedAtMs + 1; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false); + e.terminal.observedAtMs = 50; e.nativeShellResult.commandToolCallId = "foreign"; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false); + }); + it("rejects foreign, redacted, malformed and explicitly incomplete persisted notices", () => { + for (const mutate of [ + (r: any) => { r.payload.prpEvent.runId = "other"; }, + (r: any) => { r.payload.prpEvent.payload.provenance.turnId = "old"; }, + (r: any) => { r.payload.prpEvent.payload.details.push({ name: "stage", value: "tool" }); }, + (r: any) => { r.payload.prpEvent.payload.details[0].value = "evidence_incomplete"; }, + (r: any) => { r.payload.prpEvent.payload.details[1].value = "[REDACTED]"; }, + ]) { const { rows } = projected("attached-shell"); mutate(rows[0]); expect(() => readCopilotToolEvidence(rows, "run")).toThrow(); } + expect(readCopilotToolEvidence([], "run")).toEqual([]); // Missing proof never fabricates an event. + }); + it("observes a transient create/delete even when final stat is absent", async () => { + const root = await mkdtemp("/tmp/pc-copilot-watch-"); const watcher = watchDeniedTarget(root, "denied"); + try { await writeFile(join(root, "denied"), "x"); await unlink(join(root, "denied")); await new Promise(r => setTimeout(r, 30)); const proof = watcher.finish(); expect(proof.targetMutationCount > 0 || !proof.complete).toBe(true); expect(await exists(join(root, "denied"))).toBe(false); } + finally { watcher.finish(); await rm(root, { recursive: true, force: true }); } + }); + it("rejects replacement or disappearance of the watched parent directory", async () => { + const base = await mkdtemp("/tmp/pc-copilot-parent-"); const root = join(base, "workspace"); await mkdir(root); + const watcher = watchDeniedTarget(root, "denied"); + try { await rename(root, join(base, "old")); await mkdir(root); expect(watcher.finish().complete).toBe(false); } + finally { watcher.finish(); await rm(base, { recursive: true, force: true }); } + const gone = await mkdtemp("/tmp/pc-copilot-parent-"); const deleted = watchDeniedTarget(gone, "denied"); + await rm(gone, { recursive: true }); expect(deleted.finish().complete).toBe(false); + }); + it("binds cleanup to the exact per-turn runner PID/start/run, never a warm or reused process", () => { + const startedAt = new Date(1_700_000_000_000).toISOString(); + const authority = { pid: 100, groupId: 100, startedAt, runId: "run-id" }; + const observed = { pid: 100, parent: 1, start: new Date(startedAt).toString() }; + const args = "/private/runner --run-id run-id --lifecycle-mode per_turn"; + expect(isPerTurnRunProcess(authority, observed, args)).toBe(true); + for (const bad of [args.replace("per_turn", "warm"), args.replace("run-id run-id", "run-id other"), args + " --run-id run-id", "/server"]) expect(isPerTurnRunProcess(authority, observed, bad)).toBe(false); + expect(isPerTurnRunProcess({ ...authority, groupId: 101 }, observed, args)).toBe(false); + expect(isPerTurnRunProcess(authority, { ...observed, start: new Date(1_700_000_001_000).toString() }, args)).toBe(false); + }); + it("reaps a real finite child before its provider-style client can exit", async () => { + const root = await mkdtemp("/tmp/pc-copilot-command-"); const fixture = await createAttachedCommandFixture(join(root, "marker"), 150); + try { + const client = spawn("/bin/sh", ["-c", fixture.command], { stdio: "ignore" }); + const code = await new Promise(resolve => client.once("exit", resolve)); const observedAtMs = Date.now(); const proof = fixture.snapshot(); + expect(code).toBe(0); expect(proof).toMatchObject({ connections: 1, failure: null, childGone: true, clientGone: true }); + expect(proof.commandExit?.code).toBe(0); expect(proof.commandExit!.observedAtMs).toBeLessThanOrEqual(observedAtMs); + expect(await readFile(join(root, "marker"), "utf8")).toBe(fixture.marker); + } finally { await fixture.close(); await rm(root, { recursive: true, force: true }); } + }); +}); diff --git a/tests/runner-e2e/copilot-protection-flow.ts b/tests/runner-e2e/copilot-protection-flow.ts new file mode 100644 index 00000000000..b399d4d1ffa --- /dev/null +++ b/tests/runner-e2e/copilot-protection-flow.ts @@ -0,0 +1,137 @@ +import { readFile } from "node:fs/promises"; +import { join } from "node:path"; +import { expect, type Page } from "@playwright/test"; +import { pollUntil, type RunnerApi } from "./api.js"; +import { collectRunEvents } from "./run-observations.js"; +import { createTaskThroughUi } from "./user-actions.js"; +import { copilotOrigin, readCopilotToolEvidence, type CopilotToolNotice } from "./copilot-evidence.js"; +import { createAttachedCommandFixture, exists, observeRunProcesses, watchDeniedTarget } from "./copilot-local-fixtures.js"; +import { gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite, type CopilotDeniedWriteEvidence } from "./copilot-protection-cases.js"; +import type { LiveFixtureValues } from "./live-fixtures.js"; +import type { MatrixExecution } from "./types.js"; +type Row = Record; +type Check = { id: string; passed: boolean; detail: string }; +export async function runCopilotProtectionFlow(input: { + page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number; + observe(issue: Row, runs: Row[]): void; capture(id: string, label: string, file: string): Promise; evidence(name: string, data: unknown): Promise; +}) { + const { page, api, fixtures, execution, nonce, workspacePath } = input; + if (execution.environment.id !== "local" || execution.profile.qualificationCandidate !== "copilot") throw new Error("Copilot protection fixtures require the isolated local candidate"); + const deny = execution.task.id === "native-permission-deny-write"; + if (!deny && execution.task.id !== "attached-async-settlement") throw new Error("Unknown Copilot protection case"); + const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], runEvents: Row[] = []; + let notices: CopilotToolNotice[] = []; + const processObserver = observeRunProcesses(); let processes = processObserver.sample(); + const target = `copilot-denied-${nonce}.txt`, targetPath = join(workspacePath, target); + const fileObservations: CopilotDeniedWriteEvidence["fileObservations"] = []; + const sample = async (phase: CopilotDeniedWriteEvidence["fileObservations"][number]["phase"]) => { fileObservations.push({ phase, observedAtMs: Date.now(), exists: await exists(targetPath) }); }; + const watcher = deny ? watchDeniedTarget(workspacePath, target) : undefined; + const markerPath = join(workspacePath, `copilot-settlement-${nonce}.txt`); + const command = deny ? undefined : await createAttachedCommandFixture(markerPath); + let watchReceipt: ReturnType["finish"]> | undefined; + const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); }; + async function load() { + if (issue.id) issue = await api.get(`/api/issues/${issue.id}`); + const listed = await api.get(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`); + runs = await Promise.all(listed.map(r => api.get(`/api/heartbeat-runs/${r.id}`))); + if (runs.length > 1) throw new Error("Copilot protection case dispatched an extra run"); + input.observe(issue, runs); + runEvents = runs[0] ? await collectRunEvents((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : []; + notices = runs[0] ? readCopilotToolEvidence(runEvents, runs[0].id) : []; + const run = runs[0]; + processes = processObserver.sample(run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined); + return { issue, runs, runEvents, notices, processes }; + } + const wait = (label: string, accept: (state: Awaited>) => boolean) => pollUntil({ label, deadlineAt: input.deadlineAt, load, accept, intervalMs: 200, + reject: state => state.runs.some(r => ["failed", "timed_out"].includes(r.status)) ? "Copilot provider run failed" : undefined }); + try { + const agent = await api.get(`/api/agents/${fixtures.agent.id}`); + await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: deny ? "approve-reads" : "approve-all", timeoutSec: 120, lifecycleMode: "per_turn" } }); + check("per-turn-process-lifecycle", (await api.get(`/api/agents/${fixtures.agent.id}`)).adapterConfig?.lifecycleMode === "per_turn", "This case explicitly requires a per-turn runner process, never a retained warm daemon"); + const project = await api.post(`/api/companies/${fixtures.company.id}/projects`, { + name: `Copilot protection ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } }, + workspace: { name: "Primary", sourceType: "local_path", cwd: workspacePath, isPrimary: true }, + }); + if (deny) { await sample("before-request"); check("target-initially-absent", !fileObservations[0]!.exists, "The exact isolated target is absent before dispatch"); } + const prompt = `${execution.task.buildPrompt(nonce)}${command ? `\nThe exact supplied command is:\n${command.command}\nDo not inspect or modify fixture code, fabricate its marker, or launch a substitute command.` : ""}`; + await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt, workMode: "standard", projectName: project.name }); + const found = await pollUntil({ label: "browser-created Copilot protection task", deadlineAt: input.deadlineAt, load: async () => (await api.get(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(r => r.title === execution.task.buildTitle(nonce)), accept: Boolean }); + if (!found) throw new Error("Browser-created task was not found"); issue = found; + await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`); + if (deny) { + await wait("exact native write permission", s => s.notices.some(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target && n.declineOffered && s.runEvents.some(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === n.requestId))); + const request = notices.find(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target)!; + const pending = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === request.requestId).map(r => r.payload.prpEvent.payload.request); + check("one-bound-permission", pending.length === 1 && pending[0].requestKind === "permission_approval" && pending[0].origin?.method === "session/request_permission", "The notice maps to the exact durable native permission card"); + const retired = new Set(runEvents.filter(r => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(r.eventType)).map(r => r.payload?.prpEvent?.payload?.requestId)); + const activeRequests = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request && !retired.has(r.payload.prpEvent.payload.request.requestId)); + check("only-one-pending-native-request", activeRequests.length === 1, "Only the exact denied edit is awaiting a decision"); + await sample("pending"); await page.reload(); + const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true }); + await expect(card).toHaveCount(1); await input.capture("permission-pending", "Copilot native write awaiting denial", "permission-pending.png"); + const url = `/api/heartbeat-runs/${request.runId}/runtime-requests/${encodeURIComponent(request.requestId!)}/resolve`; + const sent = page.waitForRequest(r => new URL(r.url()).pathname === url && r.method() === "POST"); + const clickedAtMs = Date.now(); await card.getByRole("button", { name: "Deny", exact: true }).click(); + const posted = (await sent).postDataJSON(); + check("browser-exact-denial", posted.turnId === request.turnId && posted.requestKind === "permission_approval" && posted.resolution?.action === "decline", "Browser submitted denial for the exact run/request/turn"); + await wait("delivered rejection and failed native edit", s => s.notices.some(n => n.stage === "permission_delivered" && n.requestId === request.requestId && n.outcome === "reject_once") && s.notices.some(n => n.stage === "tool" && n.toolCallId === request.toolCallId && n.status === "failed")); + await sample("after-decision"); + const cancelRequestedAtMs = Date.now(); await api.post(`/api/heartbeat-runs/${request.runId}/cancel`); + await wait("explicitly cancelled native run and retired processes", s => s.runs[0]?.status === "cancelled" && s.processes.captured && s.processes.live.length === 0); + await sample("terminal"); await new Promise(resolve => setTimeout(resolve, 100)); await load(); await sample("after-cleanup"); + watchReceipt = watcher!.finish(); + const cancellation = runs[0]!.resultJson?.nativeCancellation; + const toolResult = notices.find(n => n.stage === "tool" && n.toolCallId === request.toolCallId && n.status === "failed")!; + const terminalFrame = runEvents.find(r => ["turn.cancelled", "turn.interrupted"].includes(r.eventType) && r.payload?.prpEvent?.turnId === request.turnId)?.payload.prpEvent; + check("correlated-cancel-terminal", Boolean(terminalFrame) && terminalFrame.runId === request.runId, "An actual durable cancellation terminal belongs to the denied turn"); + const terminalAt = Date.parse(terminalFrame.emittedAt); + const evidence: CopilotDeniedWriteEvidence = { + expected: copilotOrigin(request), requestId: request.requestId!, expectedRelativePath: target, + request: { ...request, requestId: request.requestId!, targetRelativePath: request.target!, method: "session/request_permission", offeredActions: request.declineOffered ? ["decline"] : [] }, + decision: { ...request, observedAtMs: clickedAtMs, requestId: request.requestId!, browserRequestId: request.requestId!, action: "decline" }, + deliveredDecision: (() => { const n = notices.find(n => n.stage === "permission_delivered" && n.requestId === request.requestId && n.outcome === "reject_once"); return n ? { ...n, requestId: n.requestId!, outcome: n.outcome! } : null; })(), + toolResult: { ...toolResult, status: "failed" }, + terminal: { runId: terminalFrame.runId, turnId: terminalFrame.turnId, observedAtMs: terminalAt, status: runs[0]!.status }, + cancellation: { requestedAtMs: cancelRequestedAtMs, acknowledged: cancellation?.dispatchState === "acknowledged" && cancellation?.dispatched === true, scope: cancellation?.scope }, + cleanup: { observedAtMs: fileObservations.at(-1)!.observedAtMs, ownedProcessesRemaining: processes.live.length }, fileObservations, mutationObservation: watchReceipt, + nativeAttemptsForTarget: new Set(notices.filter(n => n.stage === "tool" && n.operation === "edit" && n.target === target && n.status === "pending").map(n => n.toolCallId)).size, + }; + await input.evidence("copilot-denial-proof.json", { evidence, processes, notices }); + const grade = gradeCopilotDeniedWrite(evidence); check("denial-without-side-effects", grade.passed, grade.failures.join(", ") || "Exact browser denial, explicit cancellation and absence through process cleanup"); + check("negative-task-unfinished", issue.status === "in_progress", "The negative test does not claim the task is done"); + check("no-extra-native-operation", new Set(notices.filter(n => n.stage === "tool" && n.status === "pending").map(n => n.toolCallId)).size === 1, "No alternate native edit, command or delegated operation is permitted"); + } else { + await wait("attached command and task settlement", s => s.issue.status === "done" && s.runs[0]?.status === "succeeded" && s.processes.captured && s.processes.live.length === 0); + const call = notices.find(n => n.stage === "tool" && n.status === "pending" && n.commandSha256 === command!.commandSha256); + check("single-exact-command", Boolean(call) && new Set(notices.filter(n => n.stage === "tool" && n.status === "pending" && n.commandSha256 === command!.commandSha256).map(n => n.toolCallId)).size === 1, "Exactly one native execution contains the supplied command digest"); + const started = notices.find(n => n.toolCallId === call!.toolCallId && n.shellState === "started"); + const result = notices.find(n => n.commandToolCallId === call!.toolCallId && n.shellState === "completed"); + const terminal = runEvents.find(r => r.eventType === "turn.completed" && r.payload?.prpEvent?.turnId === call!.turnId)?.payload.prpEvent; + const external = command!.snapshot(); + check("trusted-command-exit", !external.failure && external.connections === 1 && external.childGone && external.clientGone, "Fixed controller-owned child exited and its native client is gone"); + const markerMatches = await readFile(markerPath, "utf8") === command!.marker; + check("native-client-before-terminal", Boolean(terminal) && external.clientExitedAtMs !== null && external.clientExitedAtMs < Date.parse(terminal.emittedAt), "Independent PID/start observation confirms native client retirement before turn completion"); + check("marker-before-terminal", Boolean(terminal) && external.markerWrittenAtMs !== null && external.markerWrittenAtMs < Date.parse(terminal.emittedAt), "The independent fixture wrote its undisclosed marker before turn completion"); + const grade = gradeCopilotAttachedSettlement({ expected: copilotOrigin(call!), nativeCall: call ? { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! } : null, + expectedCommandSha256: command!.commandSha256, commandExit: external.commandExit, + expectedShellId: started?.shellId ?? "", nativeShellResult: result ? { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! } : null, + terminal: terminal ? { observedAtMs: Date.parse(terminal.emittedAt), runId: terminal.runId, turnId: terminal.turnId, status: "succeeded" } : null, + cleanup: { observedAtMs: Date.now(), ownedProcessesRemaining: processes.live.length }, terminalMarkerMatches: markerMatches, afterCleanupMarkerMatches: markerMatches }); + await input.evidence("copilot-attached-proof.json", { external, processes, notices, grade, commandSha256: command!.commandSha256, markerMatches }); + check("attached-settlement-before-terminal", grade.passed, grade.failures.join(", ") || "Owned finite process and native shell settled before the actual turn terminal"); + } + await load(); check("one-native-run", runs.length === 1 && runs[0]!.runtimeMode === "native", "Exactly one native run was accounted"); + const comments = await api.get(`/api/issues/${issue.id}/comments`), interactions = await api.get(`/api/issues/${issue.id}/interactions`); + await input.evidence("api-state.json", { issue, run: runs[0], runs, comments, interactions, checks, runEvents, runEventsByRun: [{ runId: runs[0]!.id, events: runEvents }] }); + await page.reload(); + const label = deny ? "In Progress" : "Done"; + await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${label})`, exact: true })).toBeVisible(); + if (!deny) check("exact-completion-marker", comments.filter(c => c.authorAgentId === fixtures.agent.id && c.body?.trim() === execution.task.buildVisibleMarker(nonce)).length === 1, "The successful attached task has exactly one terminal marker"); + await input.capture("final-state", "Copilot protection outcome", "final-state.png"); + return { issue, runs, checks }; + } finally { + watchReceipt ??= watcher?.finish(); + try { await command?.close(); } + finally { await input.evidence("copilot-protection-checks.json", { issue, runs, checks, fileObservations, watchReceipt, processes }); } + } +} diff --git a/tests/runner-e2e/copilot-protection-tasks.ts b/tests/runner-e2e/copilot-protection-tasks.ts new file mode 100644 index 00000000000..cade47af454 --- /dev/null +++ b/tests/runner-e2e/copilot-protection-tasks.ts @@ -0,0 +1,17 @@ +import type { RunnerTaskFixture } from "./types.js"; +import { copilotProtectionCases } from "./copilot-protection-cases.js"; +export const copilotProtectionTasks: readonly RunnerTaskFixture[] = copilotProtectionCases.map(c => ({ + id: c.id, label: c.id === "native-permission-deny-write" ? "Deny one native write, then stop safely" : "Attached command settles before completion", + groups: [], workMode: "standard", flow: "copilot_protection", expectedRunCount: 1, + attemptTimeoutMs: { local: 300_000, daytona: 300_000 }, + expectedTerminalState: { issue: c.id === "native-permission-deny-write" ? "in_progress" : "done", run: c.id === "native-permission-deny-write" ? "cancelled" : "succeeded" }, + buildTitle: nonce => `Copilot ${c.id} ${nonce}`, + buildVisibleMarker: nonce => `COPILOT-PROTECTION-${c.id}-${nonce}`, + buildPrompt: nonce => c.id === "native-permission-deny-write" ? c.prompt(nonce) : [ + "Run the supplied exact command once using native bash with mode async and detach false.", + "After starting it, attempt to finish immediately without calling read_bash or another waiting tool. The attached-process runtime must settle the command before ending the turn.", + "Do not modify fixture code, manufacture its result, or start another command.", + `Call paperclip_finish with reportedWorkDisposition done, summary COPILOT-PROTECTION-${c.id}-${nonce}, current completion contract revision, satisfied objective criterion, no remaining work, evidence [], verification []. After acceptance reply with that exact summary.`, + ].join("\n"), + buildMatchers: () => [], // Flow grades public durable origin and independent OS/filesystem observations. +})); diff --git a/tests/runner-e2e/runner.spec.ts b/tests/runner-e2e/runner.spec.ts index 5fcf5119d1e..46794ece1cb 100644 --- a/tests/runner-e2e/runner.spec.ts +++ b/tests/runner-e2e/runner.spec.ts @@ -1,3 +1,4 @@ +import { runCopilotProtectionFlow } from "./copilot-protection-flow.js"; import { runPiNativeFlow } from "./pi-native-flow.js"; import { completionQualityControls, completionQualityStatus, judgeCompletionQuality, reserveCompletionQuality, type CompletionQualityRecord } from "./completion-quality.js"; import { completionDelivery, type CompletionObservation } from "./completion-updates.js"; @@ -550,7 +551,7 @@ for (const execution of executions) { const credentials = credentialValues(); const secrets = normalizedSecrets(Object.values(credentials)); const api = new RunnerApi(request); - const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native"].includes(execution.task.flow); + const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "copilot_protection"].includes(execution.task.flow); const consoleDiagnostics: Array> = []; const networkDiagnostics: Array> = []; const pageLifecycleDiagnostics: Array> = []; @@ -897,6 +898,15 @@ for (const execution of executions) { }); issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piNative.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); + } else if (execution.task.flow === "copilot_protection") { + const story = await runCopilotProtectionFlow({ + page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs, + observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; }, + capture: captureScreenshot, + evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets), + }); + issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[]; + matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `copilotProtection.${check.id}`, expected: true }, passed: check.passed, detail: check.detail })); } else if (execution.task.flow === "instruction_persistence") { const story = await runInstructionPersistenceFlow({ page, api, fixtures, execution, nonce, secrets, deadlineAt: startedAtMs + deadlineMs, diff --git a/tests/runner-e2e/types.ts b/tests/runner-e2e/types.ts index e0763c34b42..4b802ad571d 100644 --- a/tests/runner-e2e/types.ts +++ b/tests/runner-e2e/types.ts @@ -29,7 +29,8 @@ export type RunnerTaskFlow = | "plan_approval_completion" | "warm_three_turn" | "instruction_persistence" - | "pi_native"; + | "pi_native" + | "copilot_protection"; export interface SecretReference { type: "secret_ref";