mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-22 16:51:28 +02:00
test(e2e): add Copilot denial and attached settlement cases
Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed
1 parent
07369cc801
commit
d3077a061f
16 files changed
+621
-33
No files matched your search
@@ -1,15 +1,25 @@
|
||||
# Copilot 1.0.88 rich ACP capability audit
|
||||
|
||||
Audited 2026-09-28 against repository base `c65fc9e3c81c41aafe421aa90a00514b84343285`.
|
||||
Status: **candidate, not qualified**. Authenticated model discovery and exact model
|
||||
selection now pass. One canonical local protocol case and the local Product hello,
|
||||
file-edit, semantic plan and controller-restart cases passed through the real runner.
|
||||
A separate local question delivered and resumed, but failed its exact terminal-marker
|
||||
requirement. Daytona and the remaining qualification matrix remain pending. Two
|
||||
narrow real-service permission and detached-command probes also passed.
|
||||
Real executable offline probes cost $0; the live token receipt has no verified USD charge. The allocated live budget remains $25, subject to
|
||||
the shared $100 hard stop and verifiable spend. Do not expose this profile as
|
||||
supported until the required local and Daytona qualification passes.
|
||||
Initially audited 2026-09-28 against repository base `c65fc9e3c81c41aafe421aa90a00514b84343285`;
|
||||
updated 2026-09-29. Status remains **candidate, not qualified**. On frozen source
|
||||
`bd4cc29c3017ed5e1484423e842fd48e3b2f49f3`, profile v4 local hello, question/answer,
|
||||
semantic plan, controller restart and file-edit cases passed. Daytona hello passed
|
||||
one native `gpt-5.6-luna` run with six matchers on immutable image
|
||||
`sha256:bff4c3f291087a0eeae37e4c20dd51857b92833eaf73ba3aca4157f37de1e109`.
|
||||
Earlier question-marker, recovery, startup-timeout and PostgreSQL failures remain
|
||||
retained; later success does not establish causes for unresolved earlier failures.
|
||||
The new native-protection Product cases below have not run on the new source.
|
||||
|
||||
Daytona sandbox-specific analytics stabilized across three reads at a provisional
|
||||
$0.0039682144 for the closed attempt interval; invoice finality and native per-run
|
||||
Copilot USD remain unknown. The Product list-price estimate was $0.0040321867.
|
||||
One exact owned sandbox was absent after the full 360-second cleanup observation;
|
||||
45 local PIDs retired and semaphore counts returned 255→266→255. The conditional
|
||||
720-second infrastructure bound was $0.053496. These are distinct accounting claims,
|
||||
not a zero-cost assertion. The allocated provider budget remains $25 within the
|
||||
shared $100 ceiling; no new paid attempt follows automatically from registration.
|
||||
Mac x64, the remaining Daytona workflows, broader permissions/background variants
|
||||
and complete rich-field projection remain unqualified.
|
||||
|
||||
## Connection policy audit, 2026-09-29
|
||||
|
||||
@@ -741,3 +751,32 @@ preflight still rejects. Two earlier operator invocations used the wrong image
|
||||
repository or Node path and failed before provider launch; both are retained.
|
||||
This is Linux packaging evidence, not an authenticated Daytona Product run or
|
||||
model qualification. The profile remains pending.
|
||||
|
||||
### Bounded native tool observations and protection evaluations
|
||||
|
||||
The sidecar now projects an allowlist of active-turn ACP tool/permission fields into
|
||||
existing `provider.notice.recorded` details and provenance: validated relative target,
|
||||
request/tool identities, command SHA-256, explicit mode/detach, and linked shell
|
||||
start/completion with provider-reported exit code. Session passthrough notifications
|
||||
remain uncorrelated and cannot supply this evidence. Native ACP exposes an execution
|
||||
kind, not a trustworthy `bash` name in the title. Strings use semantic redaction;
|
||||
ambiguous identities, exhausted bounds or failed projection make evidence incomplete.
|
||||
Projection cannot change permission delivery or terminal authority. This is an additive
|
||||
observation change: Copilot profile v4 is unchanged, while source/pack provenance
|
||||
changes. Older sessions lacking these notices cannot pass the new evidence oracles.
|
||||
|
||||
The manual local `copilot-protection` Product suite registers an exact browser-denial
|
||||
case with explicit cancellation and a finite attached-command settlement case. Both
|
||||
remain unqualified until separately built and run. The denial case expects an unfinished
|
||||
task/cancelled run; full restrictive workflow completion is still a gap because later
|
||||
MCP permissions must not be auto-approved from provider-controlled titles.
|
||||
|
||||
A separate credential-free real-binary fixture on 2026-09-29 exercised Copilot1.0.88
|
||||
(executable SHA-256 `a9ff8babb10b7e443182ae96a8bc50a9c826ef1c773e1344c396eb5bf7f512c3`)
|
||||
against a loopback synthetic model. It started an eight-second attached async command;
|
||||
the model immediately returned terminal text while the process was still live. The
|
||||
binary itself issued a read-shell update, and process absence plus the marker were
|
||||
observed before the ACP prompt result. The model never called `read_bash`. The private
|
||||
`copilot-v4-protection-preparation/adversarial-attached-v2` receipt retains monotonic
|
||||
ordering and process identity. This verifies one pinned finite attached scenario and
|
||||
is distinct from detached-policy rejection or comprehensive background qualification.
|
||||
@@ -362,3 +362,8 @@ Copilot and Pi ACP profiles on local and Daytona. See the
|
||||
[fixture admission, credentials and budget contract](../tests/runner-e2e/README.md#extended-acp-harnesses-explicit-only).
|
||||
The private Runner Evals campaign of the same name provides complementary
|
||||
semantic protocol cases; catalog membership is not live qualification.
|
||||
|
||||
The explicit local [Copilot protection fixtures](../tests/runner-e2e/FIXTURES.md#copilot-native-protection)
|
||||
exercise native permission denial with operator cancellation and bounded attached
|
||||
command settlement. Their registration remains separate from paid qualification;
|
||||
source/pack provenance and complete persisted evidence are required.
|
||||
@@ -276,3 +276,39 @@ file under `deny-all`. All runs count toward spend and teardown. Browser reload
|
||||
is reconnect evidence only; these cases do not establish provider-death recovery.
|
||||
They use public product APIs, real browser answers, and ordinary isolated files;
|
||||
no database writes, private hooks, or fabricated provider results are allowed.
|
||||
|
||||
## Copilot native protection
|
||||
|
||||
The explicit-only `copilot-protection` suite has two local, one-run cells (120s
|
||||
provider timeout, 300s attempt budget). `native-permission-deny-write` denies one
|
||||
exact native edit through its browser card, waits for the delivered rejection and
|
||||
failed tool, then cancels through the public run API. Its expected outcome is a
|
||||
cancelled run and an unfinished task, not successful task completion. It rejects
|
||||
extra native operations/runs and observes the absent target through process
|
||||
retirement. Filesystem event loss or an unexplained parent-directory timestamp
|
||||
change makes no-effect coverage incomplete; stat polling alone cannot pass.
|
||||
|
||||
`attached-async-settlement` starts a fixed finite command with explicit async mode
|
||||
and `detach:false`, then asks the model to attempt immediate completion. A one-shot
|
||||
private local socket accepts only the fixture nonce, never an executable or command;
|
||||
the test owns/reaps a predeclared child and records its actual exit before releasing
|
||||
the provider-launched client. The independent marker, client retirement, native shell
|
||||
linkage and actual durable turn terminal must agree. Fixture resources close in a
|
||||
finally block. This establishes the tested finite attached-command behavior, not all
|
||||
background modes or detached-process settlement.
|
||||
|
||||
Both cells consume bounded, origin-correlated `copilot_tool_evidence_v1` notices
|
||||
persisted through the ordinary run-event API. Missing, redacted, ambiguous or
|
||||
explicitly incomplete notices fail qualification. Old runtime artifacts therefore
|
||||
cannot qualify these cases. No prompt/title substitutes for native input, no raw
|
||||
command or arbitrary tool input is added to production event payloads, and no private
|
||||
runner hook or database write is used. The cases are registered but require a newly
|
||||
built source/pack and separately authorized paid execution before claiming Product
|
||||
qualification. Full restrictive-workflow completion remains separate: the negative
|
||||
case never auto-approves an uncorrelated later MCP permission.
|
||||
|
||||
The Copilot protection cells explicitly select `per_turn` lifecycle. Their read-only
|
||||
process journal accepts the API runner PID only after checking its OS start time,
|
||||
process group, exact run ID argument and `per_turn` argument; it never treats a
|
||||
retained warm daemon as a leaked per-run process or signals an API-provided PID.
|
||||
Directory-watch coverage also rejects parent device/inode replacement or removal.
|
||||
@@ -1298,3 +1298,9 @@ file persistence uses two fresh task runs and independent byte checks; restricti
|
||||
denial uses one run and requires both a failed tool receipt and no file effect.
|
||||
See [the fixture contract](FIXTURES.md#pi-native-boundaries) for the exact oracles
|
||||
and the limits of reconnect evidence.
|
||||
|
||||
The manual `copilot-protection` suite selects only two local Copilot candidate
|
||||
cells. Discover them with `pnpm test:e2e:runner -- --list --suite copilot-protection`.
|
||||
See [Copilot native protection](./FIXTURES.md#copilot-native-protection) for the
|
||||
expected cancelled negative test, finite attached-process oracle, evidence limits,
|
||||
and required rebuilt runtime. Registration is not a qualification claim.
|
||||
@@ -292,3 +292,11 @@ Cursor, Copilot, GitHub and GH environment variables are stripped from the
|
||||
server environment. GitHub PAT shapes are included in retained-evidence scans.
|
||||
Candidates have no automatic infrastructure retries; spending must be reconciled
|
||||
before a deliberate repeat.
|
||||
|
||||
Copilot protection fixtures use production browser/public API permission and
|
||||
cancellation paths. Their local one-shot socket accepts a nonce/PID only and owns
|
||||
one fixed bounded child; it cannot select commands, arguments, or paths. The exact
|
||||
native client command is supplied by the fixture, while production notices retain
|
||||
only its SHA-256. The fixture keeps private process identities local, closes its
|
||||
owned socket/child in cleanup, and never signals API-reported PIDs. Directory
|
||||
watch loss makes the denial oracle incomplete; it must not become a no-effect pass.
|
||||
@@ -91,10 +91,10 @@ describe("runner E2E catalog", () => {
|
||||
expect(localIntegrityTasks).toHaveLength(2);
|
||||
expect(openRouterBreadthTasks).toHaveLength(3);
|
||||
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
|
||||
3, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2,
|
||||
3, 2, 30, 3, 16, 16, 2, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 10, 48, 16, 10, 2,
|
||||
]);
|
||||
expect(validateRunnerCatalog()).toHaveLength(410);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(410);
|
||||
expect(validateRunnerCatalog()).toHaveLength(412);
|
||||
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(412);
|
||||
expect(
|
||||
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
|
||||
).toHaveLength(48);
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { copilotProtectionTasks } from "./copilot-protection-tasks.js";
|
||||
import { piNativeTasks } from "./pi-native-cases.js";
|
||||
import { instructionPersistenceTask } from "./instruction-persistence.js";
|
||||
import { apiResponseReadingTask } from "./api-response-reading.js";
|
||||
@@ -1036,6 +1037,13 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
|
||||
environments: [localEnvironment], tasks: piNativeTasks, expectedMatrixSize: 3,
|
||||
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion },
|
||||
},
|
||||
{
|
||||
id: "copilot-protection", label: "Copilot native protection", manualOnly: true,
|
||||
description: "Exact native denial with explicit operator cancellation, and attached command settlement with independent process evidence.",
|
||||
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "copilot"),
|
||||
environments: [localEnvironment], tasks: copilotProtectionTasks, expectedMatrixSize: 2,
|
||||
definitionMetadata: { version: 1, qualification: "pending", scheduling: "explicit-only", evidence: "copilot_tool_evidence_v1", profileVersion: QUALIFIED_ACPX_PROFILES.copilot.agentProfileVersion, denialTerminal: "explicit-public-cancellation", settlement: "attached-finite-command-only" },
|
||||
},
|
||||
{
|
||||
id: "extended-harnesses", label: "Extended ACP harnesses", manualOnly: true,
|
||||
description: "Explicit candidate qualification through real Paperclip tools, browser interactions, file edits and restart recovery.",
|
||||
@@ -1374,6 +1382,7 @@ export function validateRunnerCatalog(): MatrixExecution[] {
|
||||
const allProfiles = [...extendedHarnessProfiles, ...runnerProfiles, ...legacyAcpxProfiles, ...pendingContextIntegrityProfiles, ...openRouterBreadthProfiles, ...everydayProfiles.filter(p => !runnerProfiles.some(existing => existing.id === p.id))];
|
||||
const allTasks = [
|
||||
...piNativeTasks,
|
||||
...copilotProtectionTasks,
|
||||
extendedHarnessFileTask,
|
||||
...contextIntegrityTasks,
|
||||
...accountingTasks,
|
||||
|
||||
@@ -0,0 +1,51 @@
|
||||
/** Strict reader for bounded notices persisted through the public run-event API. */
|
||||
export interface CopilotToolNotice {
|
||||
runId: string; sessionId: string; turnId: string; toolCallId: string;
|
||||
observedAtMs: number; seq: number;
|
||||
stage: "tool" | "permission_requested" | "permission_delivered";
|
||||
status?: "pending" | "in_progress" | "completed" | "failed";
|
||||
operation?: "edit" | "execute"; target?: string; requestId?: string;
|
||||
declineOffered?: boolean; outcome?: "allow_once" | "allow_always" | "reject_once" | "cancel";
|
||||
commandSha256?: string; mode?: "sync" | "async"; detach?: boolean;
|
||||
shellId?: string; commandToolCallId?: string; shellState?: "started" | "completed"; exitCode?: number;
|
||||
}
|
||||
const rec = (v: unknown): Record<string, any> => v !== null && typeof v === "object" && !Array.isArray(v) ? v as Record<string, any> : {};
|
||||
const id = (v: unknown): v is string => typeof v === "string" && v.length > 0 && v.length <= 240 && !/[\u0000-\u001f\u007f]/u.test(v) && !v.includes("[REDACTED]");
|
||||
const enums = { stage: ["tool", "permission_requested", "permission_delivered"], status: ["pending", "in_progress", "completed", "failed"], operation: ["edit", "execute"], outcome: ["allow_once", "allow_always", "reject_once", "cancel"], mode: ["sync", "async"], shellState: ["started", "completed"] };
|
||||
const names = new Set(["stage", "toolCallId", "status", "operation", "target", "requestId", "declineOffered", "outcome", "commandSha256", "mode", "detach", "shellId", "commandToolCallId", "shellState", "exitCode"]);
|
||||
export function readCopilotToolEvidence(rows: readonly unknown[], expectedRunId: string): CopilotToolNotice[] {
|
||||
const result: CopilotToolNotice[] = [];
|
||||
for (const row of rows) {
|
||||
const r = rec(row), envelope = rec(rec(r.payload).prpEvent), p = rec(envelope.payload);
|
||||
if (r.eventType !== "provider.notice.recorded" || p.category !== "copilot_tool_evidence_v1") continue;
|
||||
const origin = rec(p.provenance);
|
||||
if (p.schema !== "paperclip.provider.notice.v1" || p.scope !== "turn" || envelope.schema !== "paperclip.prp.event.v1" || envelope.sourceKind !== "runner" || envelope.eventType !== r.eventType || envelope.runId !== expectedRunId || !id(origin.sessionId) || !id(origin.turnId) || origin.turnId !== envelope.turnId || !Array.isArray(p.details) || p.details.length > 20 || !Number.isSafeInteger(r.seq)) throw new Error("Invalid Copilot evidence binding");
|
||||
const fields: Record<string, string> = {};
|
||||
for (const detail of p.details) {
|
||||
const d = rec(detail);
|
||||
if (d.name === "stage" && d.value === "evidence_incomplete") throw new Error("Copilot evidence is explicitly incomplete");
|
||||
if (!names.has(d.name) || Object.hasOwn(fields, d.name) || typeof d.value !== "string" || d.value.length > 1024 || d.value.includes("[REDACTED]")) throw new Error("Invalid Copilot evidence detail");
|
||||
fields[d.name] = d.value;
|
||||
}
|
||||
if (!id(fields.toolCallId) || !enums.stage.includes(fields.stage!) || origin.eventType !== fields.stage || origin.method !== (fields.stage === "tool" ? "session/update" : "session/request_permission")) throw new Error("Invalid Copilot evidence origin");
|
||||
const observedAtMs = Date.parse(envelope.emittedAt);
|
||||
if (!Number.isFinite(observedAtMs)) throw new Error("Copilot evidence has no producer timestamp");
|
||||
const notice: Record<string, unknown> = { runId: expectedRunId, sessionId: origin.sessionId, turnId: origin.turnId, toolCallId: fields.toolCallId, stage: fields.stage, observedAtMs, seq: r.seq };
|
||||
for (const [key, choices] of Object.entries(enums)) if (fields[key] !== undefined) { if (!(choices as readonly string[]).includes(fields[key]!)) throw new Error("Invalid Copilot evidence enum"); notice[key] = fields[key]; }
|
||||
for (const key of ["requestId", "commandToolCallId"]) if (fields[key] !== undefined) { if (!id(fields[key])) throw new Error("Invalid Copilot evidence identity"); notice[key] = fields[key]; }
|
||||
for (const key of ["detach", "declineOffered"]) if (fields[key] !== undefined) { if (!["true", "false"].includes(fields[key]!)) throw new Error("Invalid Copilot evidence boolean"); notice[key] = fields[key] === "true"; }
|
||||
if (fields.target !== undefined) {
|
||||
if (!fields.target || fields.target.startsWith("/") || /[\\:\u0000-\u001f\u007f]/u.test(fields.target) || fields.target.split("/").some(x => x === "." || x === ".." || !x)) throw new Error("Invalid Copilot evidence target");
|
||||
notice.target = fields.target;
|
||||
}
|
||||
if (fields.commandSha256 !== undefined) { if (!/^sha256:[a-f0-9]{64}$/u.test(fields.commandSha256)) throw new Error("Invalid command digest"); notice.commandSha256 = fields.commandSha256; }
|
||||
if (fields.shellId !== undefined) { if (!/^[A-Za-z0-9_.-]{1,80}$/u.test(fields.shellId)) throw new Error("Invalid shell identity"); notice.shellId = fields.shellId; }
|
||||
if (fields.exitCode !== undefined) { if (!/^-?\d{1,10}$/u.test(fields.exitCode) || !Number.isSafeInteger(Number(fields.exitCode))) throw new Error("Invalid exit code"); notice.exitCode = Number(fields.exitCode); }
|
||||
result.push(notice as unknown as CopilotToolNotice);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
export function copilotOrigin(n: CopilotToolNotice) {
|
||||
return { runId: n.runId, sessionId: n.sessionId, turnId: n.turnId, toolCallId: n.toolCallId };
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
import { spawn, execFileSync, type ChildProcess } from "node:child_process";
|
||||
import { createHash, randomBytes } from "node:crypto";
|
||||
import { watch, lstatSync, type FSWatcher } from "node:fs";
|
||||
import { lstat, mkdir, mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
|
||||
import { createServer, type Socket } from "node:net";
|
||||
import { join } from "node:path";
|
||||
|
||||
export const sha256 = (value: string) => `sha256:${createHash("sha256").update(value).digest("hex")}`;
|
||||
const quote = (value: string) => `'${value.replaceAll("'", "'\\''")}'`;
|
||||
export async function exists(path: string): Promise<boolean> {
|
||||
try { await lstat(path); return true; } catch (error) { if ((error as NodeJS.ErrnoException).code === "ENOENT") return false; throw error; }
|
||||
}
|
||||
export function watchDeniedTarget(directory: string, name: string) {
|
||||
const startedAtMs = Date.now(); let complete = true, targetMutationCount = 0;
|
||||
const before = lstatSync(directory, { bigint: true });
|
||||
let final: { startedAtMs: number; endedAtMs: number; complete: boolean; targetMutationCount: number } | undefined;
|
||||
const watcher: FSWatcher = watch(directory, (_kind, filename) => {
|
||||
if (filename === null) complete = false;
|
||||
else if (String(filename) === name) targetMutationCount++;
|
||||
});
|
||||
watcher.on("error", () => { complete = false; });
|
||||
return { finish() {
|
||||
if (final) return final;
|
||||
let after: import("node:fs").BigIntStats | undefined;
|
||||
try { after = lstatSync(directory, { bigint: true }); } catch { complete = false; }
|
||||
// FSEvents may coalesce a rapid create/delete. A changed directory version
|
||||
// with no attributed event is a coverage gap, never proof of no mutation.
|
||||
if (!after || after.dev !== before.dev || after.ino !== before.ino || !after.isDirectory()) complete = false;
|
||||
if (after && (after.mtimeNs !== before.mtimeNs || after.ctimeNs !== before.ctimeNs) && targetMutationCount === 0) complete = false;
|
||||
final = { startedAtMs, endedAtMs: Date.now(), complete, targetMutationCount }; watcher.close(); return final;
|
||||
} };
|
||||
}
|
||||
interface ProcessIdentity { pid: number; parent: number; start: string }
|
||||
function processTable(): ProcessIdentity[] {
|
||||
const output = execFileSync("/bin/ps", ["-axo", "pid=,ppid=,lstart="], { encoding: "utf8", timeout: 3000, maxBuffer: 8 * 1024 * 1024, env: { PATH: "/usr/bin:/bin", LC_ALL: "C" } });
|
||||
return output.split("\n").flatMap(line => { const m = /^\s*(\d+)\s+(\d+)\s+(.+?)\s*$/u.exec(line); return m ? [{ pid: Number(m[1]), parent: Number(m[2]), start: m[3]! }] : []; });
|
||||
}
|
||||
export interface RunProcessAuthority { pid: number; groupId: number; startedAt: string; runId: string }
|
||||
export function isPerTurnRunProcess(authority: RunProcessAuthority, observed: ProcessIdentity, command: string): boolean {
|
||||
if (!/^[a-zA-Z0-9_-]{1,128}$/u.test(authority.runId) || authority.pid !== observed.pid || authority.groupId !== observed.pid) return false;
|
||||
const started = Date.parse(authority.startedAt), actual = Date.parse(observed.start);
|
||||
if (!Number.isFinite(started) || !Number.isFinite(actual) || Math.floor(started / 1000) !== Math.floor(actual / 1000)) return false;
|
||||
const args = command.trim().split(/\s+/u);
|
||||
return [["--run-id", authority.runId], ["--lifecycle-mode", "per_turn"]].every(([flag, value]) => {
|
||||
const at = args.indexOf(flag!); return at >= 0 && args.lastIndexOf(flag!) === at && args[at + 1] === value;
|
||||
});
|
||||
}
|
||||
/** Read-only PID/start journal; never signal an API-reported or reused PID.
|
||||
* Native onSpawn publishes the runnerd child. Bind its current argv to this run
|
||||
* and per_turn lifecycle before treating its retirement as run cleanup.
|
||||
*/
|
||||
export function observeRunProcesses() {
|
||||
const owned = new Map<number, ProcessIdentity>(); let root: ProcessIdentity | undefined;
|
||||
return {
|
||||
sample(authority?: RunProcessAuthority) {
|
||||
const table = processTable();
|
||||
if (!root && authority && authority.pid > 1 && authority.pid !== process.pid) {
|
||||
const candidate = table.find(p => p.pid === authority.pid);
|
||||
if (candidate) {
|
||||
let command: string;
|
||||
try { command = execFileSync("/bin/ps", ["-p", String(candidate.pid), "-o", "command="], { encoding: "utf8", timeout: 3000, maxBuffer: 64 * 1024 }); }
|
||||
catch { command = ""; } // A vanished/uninspectable PID supplies no authority.
|
||||
if (isPerTurnRunProcess(authority, candidate, command) && processTable().some(p => p.pid === candidate.pid && p.start === candidate.start)) { root = candidate; owned.set(root.pid, root); }
|
||||
}
|
||||
}
|
||||
let changed = true;
|
||||
while (changed) {
|
||||
changed = false;
|
||||
for (const p of table) if (!owned.has(p.pid)) {
|
||||
const parent = owned.get(p.parent);
|
||||
if (parent && table.some(t => t.pid === parent.pid && t.start === parent.start)) { owned.set(p.pid, p); changed = true; }
|
||||
}
|
||||
}
|
||||
return { captured: Boolean(root), journal: [...owned.values()], live: table.filter(p => owned.get(p.pid)?.start === p.start).map(p => p.pid) };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/** A one-shot local fixture, not a command service: input selects no executable,
|
||||
* arguments or file path. The test owns/reaps the fixed finite child, records its
|
||||
* OS exit status, and holds the exact provider-launched client until that exit.
|
||||
*/
|
||||
export async function createAttachedCommandFixture(markerPath: string, delayMs = 4000) {
|
||||
if (!Number.isInteger(delayMs) || delayMs < 100 || delayMs > 8000) throw new Error("Invalid bounded fixture delay");
|
||||
const root = await mkdtemp("/tmp/pc-copilot-"); await mkdir(join(root, "private"), { mode: 0o700 });
|
||||
const socketPath = join(root, "private", "socket"), scriptPath = join(root, "client.cjs");
|
||||
const nonce = randomBytes(16).toString("hex"), marker = `${randomBytes(24).toString("hex")}\n`;
|
||||
const script = `const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\\n'));s.on('data',x=>{b+=x;if(b.includes('\\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5);}});`;
|
||||
await writeFile(scriptPath, script, { mode: 0o400 });
|
||||
const command = `${quote(process.execPath)} ${quote(scriptPath)} ${quote(socketPath)} ${quote(nonce)}`;
|
||||
const commandSha256 = sha256(command);
|
||||
let markerWrittenAtMs: number | null = null;
|
||||
let clientExitedAtMs: number | null = null;
|
||||
let clientObservation: ReturnType<typeof setInterval> | undefined;
|
||||
let connections = 0, child: ChildProcess | undefined, client: ProcessIdentity | undefined, failure: string | null = null;
|
||||
let exit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null = null;
|
||||
const sockets = new Set<Socket>();
|
||||
let closed = false;
|
||||
const server = createServer(socket => {
|
||||
sockets.add(socket); socket.on("close", () => sockets.delete(socket)); socket.on("error", () => { failure = "fixture_socket_error"; });
|
||||
let buffer = "";
|
||||
socket.on("data", data => {
|
||||
buffer += data; if (buffer.length > 1024) { failure = "fixture_request_too_large"; socket.destroy(); return; }
|
||||
if (!buffer.includes("\n")) return;
|
||||
connections++;
|
||||
try {
|
||||
const request = JSON.parse(buffer); buffer = "";
|
||||
if (connections !== 1 || request.nonce !== nonce || !Number.isSafeInteger(request.pid) || request.pid <= 1) throw new Error("fixture_request_invalid");
|
||||
const observed = processTable().find(p => p.pid === request.pid);
|
||||
const argv = execFileSync("/bin/ps", ["-p", String(request.pid), "-o", "command="], { encoding: "utf8", timeout: 3000 });
|
||||
if (!observed || !argv.includes(scriptPath) || !argv.includes(socketPath) || !argv.includes(nonce)) throw new Error("fixture_client_identity_invalid");
|
||||
client = observed;
|
||||
clientObservation = setInterval(() => {
|
||||
try {
|
||||
if (!processTable().some(p => p.pid === client!.pid && p.start === client!.start)) {
|
||||
clientExitedAtMs = Date.now(); clearInterval(clientObservation);
|
||||
}
|
||||
} catch { failure = "fixture_client_observation_failed"; clearInterval(clientObservation); }
|
||||
}, 25);
|
||||
child = spawn(process.execPath, ["-e", `setTimeout(()=>process.exit(0),${delayMs})`], { env: { PATH: "/usr/bin:/bin" }, stdio: "ignore" });
|
||||
child.once("error", () => { failure = "fixture_child_start_failed"; socket.destroy(); });
|
||||
child.once("exit", (code, signal) => {
|
||||
exit = { observedAtMs: Date.now(), code: code ?? -1, ownedProcessIdentityVerified: Number.isInteger(child?.pid), commandSha256 };
|
||||
void (async () => {
|
||||
if (code !== 0 || signal) { failure = "fixture_child_failed"; socket.destroy(); return; }
|
||||
if (await readFile(scriptPath, "utf8") !== script) { failure = "fixture_script_changed"; socket.destroy(); return; }
|
||||
await writeFile(markerPath, marker, { flag: "wx" }); markerWrittenAtMs = Date.now(); socket.end(`${JSON.stringify({ code })}\n`);
|
||||
})().catch(() => { failure = "fixture_completion_failed"; socket.destroy(); });
|
||||
});
|
||||
} catch { failure = "fixture_request_rejected"; socket.destroy(); }
|
||||
});
|
||||
});
|
||||
try { await new Promise<void>((resolve, reject) => { server.once("error", reject); server.listen(socketPath, resolve); }); }
|
||||
catch (error) { server.close(); await rm(root, { recursive: true, force: true }); throw error; }
|
||||
return {
|
||||
command, commandSha256, marker,
|
||||
snapshot() { const table = processTable(); return { connections, failure, markerWrittenAtMs, clientExitedAtMs, commandExit: exit, childPid: child?.pid ?? null, clientPid: client?.pid ?? null, clientGone: Boolean(client) && !table.some(p => p.pid === client!.pid && p.start === client!.start), childGone: Boolean(exit), observedAtMs: Date.now() }; },
|
||||
async close() {
|
||||
if (closed) return; closed = true; clearInterval(clientObservation);
|
||||
let cleanupError: unknown;
|
||||
try {
|
||||
if (child && child.exitCode === null && child.signalCode === null) {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const hard = setTimeout(() => { child!.kill("SIGKILL"); }, 3000);
|
||||
const deadline = setTimeout(() => reject(new Error("Fixture child did not settle within cleanup bound")), 5000);
|
||||
child!.once("exit", () => { clearTimeout(hard); clearTimeout(deadline); resolve(); }); child!.kill("SIGTERM");
|
||||
});
|
||||
}
|
||||
} catch (error) { cleanupError = error; }
|
||||
finally {
|
||||
for (const socket of sockets) socket.destroy();
|
||||
try { await new Promise<void>(resolve => server.close(() => resolve())); }
|
||||
finally { await rm(root, { recursive: true, force: true }); }
|
||||
}
|
||||
if (cleanupError) throw cleanupError;
|
||||
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -5,9 +5,10 @@ const terminal = { observedAtMs: 50, runId: "run", turnId: "turn", status: "succ
|
||||
const cleanup = { observedAtMs: 60, ownedProcessesRemaining: 0 };
|
||||
function denied(): CopilotDeniedWriteEvidence {
|
||||
return {
|
||||
expected: identity, terminal, cleanup, requestId: "permission-0", expectedRelativePath: "copilot-denied-nonce.txt",
|
||||
expected: identity, terminal: { ...terminal, status: "cancelled" }, cleanup, cancellation: { requestedAtMs: 40, acknowledged: true, scope: "run" }, requestId: "permission-0", expectedRelativePath: "copilot-denied-nonce.txt",
|
||||
request: { ...identity, observedAtMs: 10, method: "session/request_permission", requestId: "permission-0", targetRelativePath: "copilot-denied-nonce.txt", offeredActions: ["accept", "decline"] },
|
||||
decision: { ...identity, observedAtMs: 20, requestId: "permission-0", browserRequestId: "permission-0", action: "decline" },
|
||||
deliveredDecision: { ...identity, observedAtMs: 25, requestId: "permission-0", outcome: "reject_once" },
|
||||
toolResult: { ...identity, observedAtMs: 30, status: "failed" }, nativeAttemptsForTarget: 1,
|
||||
fileObservations: [
|
||||
{ phase: "before-request", observedAtMs: 0, exists: false }, { phase: "pending", observedAtMs: 15, exists: false },
|
||||
@@ -20,25 +21,30 @@ function denied(): CopilotDeniedWriteEvidence {
|
||||
function attached(): CopilotAttachedSettlementEvidence {
|
||||
const digest = `sha256:${"a".repeat(64)}`;
|
||||
return {
|
||||
expected: identity, terminal, cleanup, expectedCommandSha256: digest, expectedShellId: "0",
|
||||
nativeCall: { ...identity, observedAtMs: 10, tool: "bash", mode: "async", detach: false, commandSha256: digest },
|
||||
expected: identity, terminal: { ...terminal }, cleanup: { ...cleanup }, expectedCommandSha256: digest, expectedShellId: "0",
|
||||
nativeCall: { ...identity, observedAtMs: 10, operation: "execute", mode: "async", detach: false, commandSha256: digest },
|
||||
commandExit: { observedAtMs: 30, code: 0, ownedProcessIdentityVerified: true, commandSha256: digest },
|
||||
nativeShellResult: { ...identity, toolCallId: "read-shell-tool", commandToolCallId: "tool", observedAtMs: 40, shellId: "0", status: "completed", exitCode: 0 },
|
||||
terminalMarkerMatches: true, afterCleanupMarkerMatches: true,
|
||||
};
|
||||
}
|
||||
describe("prepared Copilot protection Product cases", () => {
|
||||
it("declares one bounded run and leaves discovery integration explicitly pending", () => {
|
||||
describe("Copilot protection Product oracles", () => {
|
||||
it("declares one bounded run and explicit discovery integration", () => {
|
||||
expect(copilotProtectionCases.map(c => c.id)).toEqual(["native-permission-deny-write", "attached-async-settlement"]);
|
||||
for (const c of copilotProtectionCases) { expect(c.expectedRunCount).toBe(1); expect(c.providerTimeoutSec).toBe(120); expect(c.integration).toMatch(/^pending-/); }
|
||||
for (const c of copilotProtectionCases) { expect(c.expectedRunCount).toBe(1); expect(c.providerTimeoutSec).toBe(120); expect(c.integration).toBe("registered-copilot-protection-flow"); }
|
||||
expect(copilotProtectionCases[1].prompt("nonce")).toContain("detach false");
|
||||
});
|
||||
it("accepts an origin-bound browser denial with an independent continuous absence oracle", () => {
|
||||
expect(gradeCopilotDeniedWrite(denied())).toEqual({ passed: true, failures: [] });
|
||||
});
|
||||
it.each(["request", "decision", "toolResult", "terminal", "cleanup", "mutationObservation"] as const)("rejects missing %s instead of treating no write as denial", field => {
|
||||
it.each(["request", "decision", "deliveredDecision", "toolResult", "terminal", "cleanup", "mutationObservation"] as const)("rejects missing %s instead of treating no write as denial", field => {
|
||||
const e = denied(); e[field] = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
|
||||
});
|
||||
it("requires explicit acknowledged cancellation for denial without relaxing successful settlement", () => {
|
||||
const e = denied(); e.cancellation = null; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
|
||||
const finished = denied(); finished.terminal!.status = "succeeded"; expect(gradeCopilotDeniedWrite(finished).passed).toBe(false);
|
||||
const cancelled = attached(); cancelled.terminal!.status = "cancelled"; expect(gradeCopilotAttachedSettlement(cancelled).passed).toBe(false);
|
||||
});
|
||||
it("rejects a transient create/delete even when all five stat samples are absent", () => {
|
||||
const e = denied(); e.mutationObservation!.targetMutationCount = 2; expect(gradeCopilotDeniedWrite(e).failures).toContain("missing-or-mutated-filesystem-watch");
|
||||
});
|
||||
|
||||
@@ -1,15 +1,15 @@
|
||||
/** Prepared Copilot Product cases. Registration waits for the evidence hooks below. */
|
||||
/** Registered manual cases; paid Product qualification remains separate. */
|
||||
export const copilotProtectionCases = [
|
||||
{
|
||||
id: "native-permission-deny-write",
|
||||
permissionMode: "approve-reads",
|
||||
expectedRunCount: 1,
|
||||
providerTimeoutSec: 120,
|
||||
integration: "pending-native-permission-browser-flow",
|
||||
integration: "registered-copilot-protection-flow",
|
||||
prompt: (nonce: string) => [
|
||||
`Attempt exactly one native file edit creating copilot-denied-${nonce}.txt with DENIED-${nonce}.`,
|
||||
"Wait for the native permission decision. If denied, do not retry through another tool, shell, path, or agent.",
|
||||
`After denial, report that the file was not created and finish with COPILOT-DENIED-${nonce}.`,
|
||||
"After denial, do not invoke any further tool or workaround. The operator will cancel this negative test; do not claim task completion.",
|
||||
].join("\n"),
|
||||
},
|
||||
{
|
||||
@@ -17,10 +17,10 @@ export const copilotProtectionCases = [
|
||||
permissionMode: "approve-all",
|
||||
expectedRunCount: 1,
|
||||
providerTimeoutSec: 120,
|
||||
integration: "pending-native-mode-and-process-settlement-evidence",
|
||||
integration: "registered-copilot-protection-flow",
|
||||
prompt: (nonce: string) => [
|
||||
`Run the supplied copilot-settlement-${nonce} command exactly once with the native bash tool, mode async and detach false.`,
|
||||
"Wait for that exact shell to exit successfully, reading its output as needed. Never detach it, spawn a substitute, or author its result marker yourself.",
|
||||
"Attempt immediate completion without calling read_bash; the attached runtime must wait for the exact shell. Never detach, substitute, or author its result marker yourself.",
|
||||
`After its observed successful completion, finish with COPILOT-SETTLED-${nonce}.`,
|
||||
].join("\n"),
|
||||
},
|
||||
@@ -35,7 +35,7 @@ interface Identity {
|
||||
interface TimedIdentity extends Identity { observedAtMs: number }
|
||||
interface Lifecycle {
|
||||
expected: Identity;
|
||||
terminal: { observedAtMs: number; runId: string; turnId: string; status: "succeeded" | "failed" } | null;
|
||||
terminal: { observedAtMs: number; runId: string; turnId: string; status: "succeeded" | "failed" | "cancelled" } | null;
|
||||
cleanup: { observedAtMs: number; ownedProcessesRemaining: number } | null;
|
||||
}
|
||||
interface FileObservation {
|
||||
@@ -44,6 +44,8 @@ interface FileObservation {
|
||||
exists: boolean;
|
||||
}
|
||||
export interface CopilotDeniedWriteEvidence extends Lifecycle {
|
||||
deliveredDecision: (TimedIdentity & { requestId: string; outcome: string }) | null;
|
||||
cancellation: { requestedAtMs: number; acknowledged: boolean; scope: string } | null;
|
||||
request: (TimedIdentity & { method: "session/request_permission"; requestId: string; targetRelativePath: string; offeredActions: string[] }) | null;
|
||||
decision: (TimedIdentity & { requestId: string; action: string; browserRequestId: string }) | null;
|
||||
requestId: string | null;
|
||||
@@ -56,7 +58,7 @@ export interface CopilotDeniedWriteEvidence extends Lifecycle {
|
||||
}
|
||||
export interface CopilotAttachedSettlementEvidence extends Lifecycle {
|
||||
/** Requires origin-correlated native input; a prompt/title is not evidence. */
|
||||
nativeCall: (TimedIdentity & { tool: string; mode: string; detach: boolean; commandSha256: string }) | null;
|
||||
nativeCall: (TimedIdentity & { operation: string; mode: string; detach: boolean; commandSha256: string }) | null;
|
||||
expectedCommandSha256: string;
|
||||
commandExit: { observedAtMs: number; code: number; ownedProcessIdentityVerified: boolean; commandSha256: string } | null;
|
||||
nativeShellResult: (TimedIdentity & { shellId: string; commandToolCallId: string; status: string; exitCode: number }) | null;
|
||||
@@ -67,15 +69,17 @@ export interface CopilotAttachedSettlementEvidence extends Lifecycle {
|
||||
export interface CopilotProtectionGrade { passed: boolean; failures: string[] }
|
||||
const same = (a: Identity, b: Identity) => ["runId", "sessionId", "turnId", "toolCallId"].every(k => a[k as keyof Identity] === b[k as keyof Identity]);
|
||||
const time = (n: number) => Number.isFinite(n) && n >= 0;
|
||||
function lifecycle(e: Lifecycle, failures: string[]) {
|
||||
function lifecycle(e: Lifecycle, failures: string[], expectedStatus: "succeeded" | "cancelled" = "succeeded") {
|
||||
if (!Object.values(e.expected).every(v => typeof v === "string" && v.length > 0)) failures.push("missing-origin-identity");
|
||||
if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== "succeeded" || !time(e.terminal.observedAtMs)) failures.push("missing-successful-origin-terminal");
|
||||
if (!e.terminal || e.terminal.runId !== e.expected.runId || e.terminal.turnId !== e.expected.turnId || e.terminal.status !== expectedStatus || !time(e.terminal.observedAtMs)) failures.push("missing-expected-origin-terminal");
|
||||
if (!e.cleanup || e.cleanup.ownedProcessesRemaining !== 0 || !time(e.cleanup.observedAtMs) || !e.terminal || e.cleanup.observedAtMs < e.terminal.observedAtMs) failures.push("unsettled-cleanup");
|
||||
}
|
||||
export function gradeCopilotDeniedWrite(e: CopilotDeniedWriteEvidence): CopilotProtectionGrade {
|
||||
const failures: string[] = []; lifecycle(e, failures);
|
||||
const failures: string[] = []; lifecycle(e, failures, "cancelled");
|
||||
if (!e.cancellation || !e.cancellation.acknowledged || !time(e.cancellation.requestedAtMs) || e.cancellation.scope !== "run" || !e.toolResult || e.cancellation.requestedAtMs < e.toolResult.observedAtMs || !e.terminal || e.cancellation.requestedAtMs > e.terminal.observedAtMs) failures.push("missing-explicit-settled-cancellation");
|
||||
if (!/^copilot-denied-[a-z0-9-]+\.txt$/.test(e.expectedRelativePath) || !e.request || !same(e.request, e.expected) || e.request.method !== "session/request_permission" || e.request.requestId !== e.requestId || e.request.targetRelativePath !== e.expectedRelativePath || !e.request.offeredActions.includes("decline") || !time(e.request.observedAtMs)) failures.push("missing-exact-native-write-request");
|
||||
if (!e.requestId || !e.decision || !same(e.decision, e.expected) || e.decision.requestId !== e.requestId || e.decision.browserRequestId !== e.requestId || e.decision.action !== "decline" || !time(e.decision.observedAtMs) || !e.request || e.decision.observedAtMs < e.request.observedAtMs) failures.push("missing-exact-browser-denial");
|
||||
if (!e.deliveredDecision || !same(e.deliveredDecision, e.expected) || e.deliveredDecision.requestId !== e.requestId || e.deliveredDecision.outcome !== "reject_once" || !time(e.deliveredDecision.observedAtMs) || !e.decision || e.deliveredDecision.observedAtMs < e.decision.observedAtMs || !e.toolResult || e.deliveredDecision.observedAtMs > e.toolResult.observedAtMs) failures.push("missing-delivered-native-rejection");
|
||||
if (!e.toolResult || !same(e.toolResult, e.expected) || e.toolResult.status !== "failed" || !time(e.toolResult.observedAtMs) || !e.decision || e.toolResult.observedAtMs < e.decision.observedAtMs || !e.terminal || e.toolResult.observedAtMs > e.terminal.observedAtMs) failures.push("missing-denied-tool-result-before-terminal");
|
||||
if (e.nativeAttemptsForTarget !== 1) failures.push("missing-or-retried-native-write");
|
||||
const phases = ["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const;
|
||||
@@ -89,7 +93,7 @@ export function gradeCopilotDeniedWrite(e: CopilotDeniedWriteEvidence): CopilotP
|
||||
export function gradeCopilotAttachedSettlement(e: CopilotAttachedSettlementEvidence): CopilotProtectionGrade {
|
||||
const failures: string[] = []; lifecycle(e, failures);
|
||||
const call = e.nativeCall;
|
||||
if (!/^sha256:[a-f0-9]{64}$/.test(e.expectedCommandSha256) || !call || !same(call, e.expected) || call.tool !== "bash" || call.mode !== "async" || call.detach !== false || call.commandSha256 !== e.expectedCommandSha256 || !time(call.observedAtMs)) failures.push("missing-exact-attached-async-call");
|
||||
if (!/^sha256:[a-f0-9]{64}$/.test(e.expectedCommandSha256) || !call || !same(call, e.expected) || call.operation !== "execute" || call.mode !== "async" || call.detach !== false || call.commandSha256 !== e.expectedCommandSha256 || !time(call.observedAtMs)) failures.push("missing-exact-attached-async-call");
|
||||
const exit = e.commandExit;
|
||||
if (!exit || !exit.ownedProcessIdentityVerified || exit.commandSha256 !== e.expectedCommandSha256 || exit.code !== 0 || !time(exit.observedAtMs) || !call || exit.observedAtMs < call.observedAtMs || !e.terminal || exit.observedAtMs >= e.terminal.observedAtMs) failures.push("command-not-settled-before-terminal");
|
||||
const result = e.nativeShellResult;
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
import { spawn } from "node:child_process";
|
||||
import { readFile, mkdtemp, rm, writeFile, unlink, rename, mkdir } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { createCopilotToolEvidence } from "../../packages/paperclip-runner/src/cli/copilot-tool-evidence.js";
|
||||
import { validateAcpxRichEvent } from "../../packages/paperclip-runner/src/drivers/acpx/profile-extensions.js";
|
||||
import { copilotOrigin, readCopilotToolEvidence } from "./copilot-evidence.js";
|
||||
import { gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite } from "./copilot-protection-cases.js";
|
||||
import { createAttachedCommandFixture, watchDeniedTarget, exists, isPerTurnRunProcess } from "./copilot-local-fixtures.js";
|
||||
import { runnerMatrix } from "./catalog.js";
|
||||
import { selectRunnerExecutions, parseRunnerSelectors } from "./selectors.js";
|
||||
|
||||
const fixture = JSON.parse(await readFile(new URL("../../packages/paperclip-runner/src/cli/fixtures/copilot-tool-evidence.json", import.meta.url), "utf8"));
|
||||
function projected(name: string) {
|
||||
const frames = fixture[name], rows: any[] = []; let clock = 10;
|
||||
const projector = createCopilotToolEvidence({ sessionId: frames[0].params.sessionId, turnId: "turn", workingDirectory: "/fixture/workspace", active: () => true,
|
||||
emit: event => { validateAcpxRichEvent(event); rows.push({ seq: rows.length + 1, eventType: event.eventType, payload: { prpEvent: { schema: "paperclip.prp.event.v1", sourceKind: "runner", eventType: event.eventType, runId: "run", turnId: "turn", emittedAt: new Date(clock++).toISOString(), payload: event.payload } } }); } });
|
||||
for (const frame of frames) {
|
||||
if (frame.method === "session/update") projector.tool({ ...frame.params.update, type: "tool_call", tag: frame.params.update.sessionUpdate });
|
||||
else projector.permission({ raw: frame.params }, "permission", ["decline"])?.("reject_once");
|
||||
}
|
||||
return { rows, notices: readCopilotToolEvidence(rows, "run") };
|
||||
}
|
||||
describe("Copilot Product protection integration", () => {
|
||||
it("registers exactly two explicit local cells with honest terminal expectations", () => {
|
||||
const cells = runnerMatrix.filter(x => x.suite.id === "copilot-protection");
|
||||
expect(cells).toHaveLength(2); expect(cells.every(c => c.environment.id === "local" && c.profile.qualificationCandidate === "copilot" && c.task.expectedRunCount === 1)).toBe(true);
|
||||
expect(cells.find(c => c.task.id === "native-permission-deny-write")!.task.expectedTerminalState).toEqual({ issue: "in_progress", run: "cancelled" });
|
||||
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(x => x.suite.id === "copilot-protection")).toBe(false);
|
||||
});
|
||||
it("feeds actual native denied-edit wire through canonical persistence shape and the Product oracle", () => {
|
||||
const { notices } = projected("deny-write");
|
||||
const request = notices.find(n => n.stage === "permission_requested")!, delivered = notices.find(n => n.stage === "permission_delivered")!, failed = notices.find(n => n.status === "failed")!;
|
||||
const e = { expected: copilotOrigin(request), requestId: "permission", expectedRelativePath: "copilot-denied-nonce.txt",
|
||||
request: { ...request, requestId: "permission", method: "session/request_permission" as const, targetRelativePath: request.target!, offeredActions: request.declineOffered ? ["decline"] : [] },
|
||||
decision: { ...delivered, requestId: "permission", browserRequestId: "permission", action: delivered.outcome === "reject_once" ? "decline" : "accept" },
|
||||
deliveredDecision: { ...delivered, requestId: "permission", outcome: delivered.outcome! },
|
||||
toolResult: { ...failed, status: "failed" as const }, terminal: { runId: "run", turnId: "turn", observedAtMs: 50, status: "cancelled" as const }, cancellation: { requestedAtMs: 40, scope: "run", acknowledged: true },
|
||||
cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, nativeAttemptsForTarget: 1,
|
||||
fileObservations: (["before-request", "pending", "after-decision", "terminal", "after-cleanup"] as const).map((phase, index) => ({ phase, observedAtMs: [0, request.observedAtMs, 30, 50, 60][index]!, exists: false })), mutationObservation: { startedAtMs: 0, endedAtMs: 60, complete: true, targetMutationCount: 0 } };
|
||||
expect(request.target).toBe("copilot-denied-nonce.txt");
|
||||
expect(gradeCopilotDeniedWrite(e).passed).toBe(true);
|
||||
e.request.targetRelativePath = "foreign.txt"; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
|
||||
e.request.targetRelativePath = e.expectedRelativePath; e.cancellation.acknowledged = false; expect(gradeCopilotDeniedWrite(e).passed).toBe(false);
|
||||
});
|
||||
it("feeds actual attached wire through canonical notices and rejects early terminal or missing linkage", () => {
|
||||
const { notices } = projected("attached-shell"); const call = notices.find(n => n.commandSha256)!;
|
||||
const started = notices.find(n => n.shellState === "started")!, result = notices.find(n => n.shellState === "completed")!;
|
||||
const e = { expected: copilotOrigin(call), nativeCall: { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! }, expectedCommandSha256: call.commandSha256!, expectedShellId: started.shellId!,
|
||||
commandExit: { observedAtMs: result.observedAtMs - 1, code: 0, ownedProcessIdentityVerified: true, commandSha256: call.commandSha256! },
|
||||
nativeShellResult: { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! },
|
||||
terminal: { observedAtMs: 50, runId: "run", turnId: "turn", status: "succeeded" as const }, cleanup: { observedAtMs: 60, ownedProcessesRemaining: 0 }, terminalMarkerMatches: true, afterCleanupMarkerMatches: true };
|
||||
expect(gradeCopilotAttachedSettlement(e).passed).toBe(true);
|
||||
e.terminal.observedAtMs = call.observedAtMs + 1; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
|
||||
e.terminal.observedAtMs = 50; e.nativeShellResult.commandToolCallId = "foreign"; expect(gradeCopilotAttachedSettlement(e).passed).toBe(false);
|
||||
});
|
||||
it("rejects foreign, redacted, malformed and explicitly incomplete persisted notices", () => {
|
||||
for (const mutate of [
|
||||
(r: any) => { r.payload.prpEvent.runId = "other"; },
|
||||
(r: any) => { r.payload.prpEvent.payload.provenance.turnId = "old"; },
|
||||
(r: any) => { r.payload.prpEvent.payload.details.push({ name: "stage", value: "tool" }); },
|
||||
(r: any) => { r.payload.prpEvent.payload.details[0].value = "evidence_incomplete"; },
|
||||
(r: any) => { r.payload.prpEvent.payload.details[1].value = "[REDACTED]"; },
|
||||
]) { const { rows } = projected("attached-shell"); mutate(rows[0]); expect(() => readCopilotToolEvidence(rows, "run")).toThrow(); }
|
||||
expect(readCopilotToolEvidence([], "run")).toEqual([]); // Missing proof never fabricates an event.
|
||||
});
|
||||
it("observes a transient create/delete even when final stat is absent", async () => {
|
||||
const root = await mkdtemp("/tmp/pc-copilot-watch-"); const watcher = watchDeniedTarget(root, "denied");
|
||||
try { await writeFile(join(root, "denied"), "x"); await unlink(join(root, "denied")); await new Promise(r => setTimeout(r, 30)); const proof = watcher.finish(); expect(proof.targetMutationCount > 0 || !proof.complete).toBe(true); expect(await exists(join(root, "denied"))).toBe(false); }
|
||||
finally { watcher.finish(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
it("rejects replacement or disappearance of the watched parent directory", async () => {
|
||||
const base = await mkdtemp("/tmp/pc-copilot-parent-"); const root = join(base, "workspace"); await mkdir(root);
|
||||
const watcher = watchDeniedTarget(root, "denied");
|
||||
try { await rename(root, join(base, "old")); await mkdir(root); expect(watcher.finish().complete).toBe(false); }
|
||||
finally { watcher.finish(); await rm(base, { recursive: true, force: true }); }
|
||||
const gone = await mkdtemp("/tmp/pc-copilot-parent-"); const deleted = watchDeniedTarget(gone, "denied");
|
||||
await rm(gone, { recursive: true }); expect(deleted.finish().complete).toBe(false);
|
||||
});
|
||||
it("binds cleanup to the exact per-turn runner PID/start/run, never a warm or reused process", () => {
|
||||
const startedAt = new Date(1_700_000_000_000).toISOString();
|
||||
const authority = { pid: 100, groupId: 100, startedAt, runId: "run-id" };
|
||||
const observed = { pid: 100, parent: 1, start: new Date(startedAt).toString() };
|
||||
const args = "/private/runner --run-id run-id --lifecycle-mode per_turn";
|
||||
expect(isPerTurnRunProcess(authority, observed, args)).toBe(true);
|
||||
for (const bad of [args.replace("per_turn", "warm"), args.replace("run-id run-id", "run-id other"), args + " --run-id run-id", "/server"]) expect(isPerTurnRunProcess(authority, observed, bad)).toBe(false);
|
||||
expect(isPerTurnRunProcess({ ...authority, groupId: 101 }, observed, args)).toBe(false);
|
||||
expect(isPerTurnRunProcess(authority, { ...observed, start: new Date(1_700_000_001_000).toString() }, args)).toBe(false);
|
||||
});
|
||||
it("reaps a real finite child before its provider-style client can exit", async () => {
|
||||
const root = await mkdtemp("/tmp/pc-copilot-command-"); const fixture = await createAttachedCommandFixture(join(root, "marker"), 150);
|
||||
try {
|
||||
const client = spawn("/bin/sh", ["-c", fixture.command], { stdio: "ignore" });
|
||||
const code = await new Promise<number | null>(resolve => client.once("exit", resolve)); const observedAtMs = Date.now(); const proof = fixture.snapshot();
|
||||
expect(code).toBe(0); expect(proof).toMatchObject({ connections: 1, failure: null, childGone: true, clientGone: true });
|
||||
expect(proof.commandExit?.code).toBe(0); expect(proof.commandExit!.observedAtMs).toBeLessThanOrEqual(observedAtMs);
|
||||
expect(await readFile(join(root, "marker"), "utf8")).toBe(fixture.marker);
|
||||
} finally { await fixture.close(); await rm(root, { recursive: true, force: true }); }
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,137 @@
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { join } from "node:path";
|
||||
import { expect, type Page } from "@playwright/test";
|
||||
import { pollUntil, type RunnerApi } from "./api.js";
|
||||
import { collectRunEvents } from "./run-observations.js";
|
||||
import { createTaskThroughUi } from "./user-actions.js";
|
||||
import { copilotOrigin, readCopilotToolEvidence, type CopilotToolNotice } from "./copilot-evidence.js";
|
||||
import { createAttachedCommandFixture, exists, observeRunProcesses, watchDeniedTarget } from "./copilot-local-fixtures.js";
|
||||
import { gradeCopilotAttachedSettlement, gradeCopilotDeniedWrite, type CopilotDeniedWriteEvidence } from "./copilot-protection-cases.js";
|
||||
import type { LiveFixtureValues } from "./live-fixtures.js";
|
||||
import type { MatrixExecution } from "./types.js";
|
||||
type Row = Record<string, any>;
|
||||
type Check = { id: string; passed: boolean; detail: string };
|
||||
export async function runCopilotProtectionFlow(input: {
|
||||
page: Page; api: RunnerApi; fixtures: LiveFixtureValues; execution: MatrixExecution; nonce: string; workspacePath: string; deadlineAt: number;
|
||||
observe(issue: Row, runs: Row[]): void; capture(id: string, label: string, file: string): Promise<void>; evidence(name: string, data: unknown): Promise<void>;
|
||||
}) {
|
||||
const { page, api, fixtures, execution, nonce, workspacePath } = input;
|
||||
if (execution.environment.id !== "local" || execution.profile.qualificationCandidate !== "copilot") throw new Error("Copilot protection fixtures require the isolated local candidate");
|
||||
const deny = execution.task.id === "native-permission-deny-write";
|
||||
if (!deny && execution.task.id !== "attached-async-settlement") throw new Error("Unknown Copilot protection case");
|
||||
const checks: Check[] = []; let issue: Row = {}, runs: Row[] = [], runEvents: Row[] = [];
|
||||
let notices: CopilotToolNotice[] = [];
|
||||
const processObserver = observeRunProcesses(); let processes = processObserver.sample();
|
||||
const target = `copilot-denied-${nonce}.txt`, targetPath = join(workspacePath, target);
|
||||
const fileObservations: CopilotDeniedWriteEvidence["fileObservations"] = [];
|
||||
const sample = async (phase: CopilotDeniedWriteEvidence["fileObservations"][number]["phase"]) => { fileObservations.push({ phase, observedAtMs: Date.now(), exists: await exists(targetPath) }); };
|
||||
const watcher = deny ? watchDeniedTarget(workspacePath, target) : undefined;
|
||||
const markerPath = join(workspacePath, `copilot-settlement-${nonce}.txt`);
|
||||
const command = deny ? undefined : await createAttachedCommandFixture(markerPath);
|
||||
let watchReceipt: ReturnType<ReturnType<typeof watchDeniedTarget>["finish"]> | undefined;
|
||||
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); expect(passed, detail).toBe(true); };
|
||||
async function load() {
|
||||
if (issue.id) issue = await api.get<Row>(`/api/issues/${issue.id}`);
|
||||
const listed = await api.get<Row[]>(`/api/companies/${fixtures.company.id}/heartbeat-runs?limit=100`);
|
||||
runs = await Promise.all(listed.map(r => api.get<Row>(`/api/heartbeat-runs/${r.id}`)));
|
||||
if (runs.length > 1) throw new Error("Copilot protection case dispatched an extra run");
|
||||
input.observe(issue, runs);
|
||||
runEvents = runs[0] ? await collectRunEvents<Row>((afterSeq, limit) => api.get(`/api/heartbeat-runs/${runs[0]!.id}/events?afterSeq=${afterSeq}&limit=${limit}`)) : [];
|
||||
notices = runs[0] ? readCopilotToolEvidence(runEvents, runs[0].id) : [];
|
||||
const run = runs[0];
|
||||
processes = processObserver.sample(run?.processPid ? { pid: run.processPid, groupId: run.processGroupId, startedAt: run.processStartedAt, runId: run.id } : undefined);
|
||||
return { issue, runs, runEvents, notices, processes };
|
||||
}
|
||||
const wait = (label: string, accept: (state: Awaited<ReturnType<typeof load>>) => boolean) => pollUntil({ label, deadlineAt: input.deadlineAt, load, accept, intervalMs: 200,
|
||||
reject: state => state.runs.some(r => ["failed", "timed_out"].includes(r.status)) ? "Copilot provider run failed" : undefined });
|
||||
try {
|
||||
const agent = await api.get<Row>(`/api/agents/${fixtures.agent.id}`);
|
||||
await api.patch(`/api/agents/${fixtures.agent.id}`, { adapterConfig: { ...agent.adapterConfig, acpxPermissionMode: deny ? "approve-reads" : "approve-all", timeoutSec: 120, lifecycleMode: "per_turn" } });
|
||||
check("per-turn-process-lifecycle", (await api.get<Row>(`/api/agents/${fixtures.agent.id}`)).adapterConfig?.lifecycleMode === "per_turn", "This case explicitly requires a per-turn runner process, never a retained warm daemon");
|
||||
const project = await api.post<Row>(`/api/companies/${fixtures.company.id}/projects`, {
|
||||
name: `Copilot protection ${nonce}`, executionWorkspacePolicy: { enabled: true, defaultMode: "shared_workspace", sharedWorkspaceConcurrency: "serialize", allowIssueOverride: false, environmentId: fixtures.environment.id, workspaceStrategy: { type: "project_primary" } },
|
||||
workspace: { name: "Primary", sourceType: "local_path", cwd: workspacePath, isPrimary: true },
|
||||
});
|
||||
if (deny) { await sample("before-request"); check("target-initially-absent", !fileObservations[0]!.exists, "The exact isolated target is absent before dispatch"); }
|
||||
const prompt = `${execution.task.buildPrompt(nonce)}${command ? `\nThe exact supplied command is:\n${command.command}\nDo not inspect or modify fixture code, fabricate its marker, or launch a substitute command.` : ""}`;
|
||||
await createTaskThroughUi({ page, issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(nonce), prompt, workMode: "standard", projectName: project.name });
|
||||
const found = await pollUntil({ label: "browser-created Copilot protection task", deadlineAt: input.deadlineAt, load: async () => (await api.get<Row[]>(`/api/companies/${fixtures.company.id}/issues?limit=100`)).find(r => r.title === execution.task.buildTitle(nonce)), accept: Boolean });
|
||||
if (!found) throw new Error("Browser-created task was not found"); issue = found;
|
||||
await page.goto(`/${fixtures.company.issuePrefix}/issues/${issue.identifier ?? issue.id}`);
|
||||
if (deny) {
|
||||
await wait("exact native write permission", s => s.notices.some(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target && n.declineOffered && s.runEvents.some(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === n.requestId)));
|
||||
const request = notices.find(n => n.stage === "permission_requested" && n.operation === "edit" && n.target === target)!;
|
||||
const pending = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request?.requestId === request.requestId).map(r => r.payload.prpEvent.payload.request);
|
||||
check("one-bound-permission", pending.length === 1 && pending[0].requestKind === "permission_approval" && pending[0].origin?.method === "session/request_permission", "The notice maps to the exact durable native permission card");
|
||||
const retired = new Set(runEvents.filter(r => ["runtime_request.resolved", "runtime_request.cancelled", "runtime_request.expired"].includes(r.eventType)).map(r => r.payload?.prpEvent?.payload?.requestId));
|
||||
const activeRequests = runEvents.filter(r => r.eventType === "runtime_request.created" && r.payload?.prpEvent?.payload?.request && !retired.has(r.payload.prpEvent.payload.request.requestId));
|
||||
check("only-one-pending-native-request", activeRequests.length === 1, "Only the exact denied edit is awaiting a decision");
|
||||
await sample("pending"); await page.reload();
|
||||
const card = page.getByTestId("task-chat-runtime-request").filter({ visible: true });
|
||||
await expect(card).toHaveCount(1); await input.capture("permission-pending", "Copilot native write awaiting denial", "permission-pending.png");
|
||||
const url = `/api/heartbeat-runs/${request.runId}/runtime-requests/${encodeURIComponent(request.requestId!)}/resolve`;
|
||||
const sent = page.waitForRequest(r => new URL(r.url()).pathname === url && r.method() === "POST");
|
||||
const clickedAtMs = Date.now(); await card.getByRole("button", { name: "Deny", exact: true }).click();
|
||||
const posted = (await sent).postDataJSON();
|
||||
check("browser-exact-denial", posted.turnId === request.turnId && posted.requestKind === "permission_approval" && posted.resolution?.action === "decline", "Browser submitted denial for the exact run/request/turn");
|
||||
await wait("delivered rejection and failed native edit", s => s.notices.some(n => n.stage === "permission_delivered" && n.requestId === request.requestId && n.outcome === "reject_once") && s.notices.some(n => n.stage === "tool" && n.toolCallId === request.toolCallId && n.status === "failed"));
|
||||
await sample("after-decision");
|
||||
const cancelRequestedAtMs = Date.now(); await api.post(`/api/heartbeat-runs/${request.runId}/cancel`);
|
||||
await wait("explicitly cancelled native run and retired processes", s => s.runs[0]?.status === "cancelled" && s.processes.captured && s.processes.live.length === 0);
|
||||
await sample("terminal"); await new Promise(resolve => setTimeout(resolve, 100)); await load(); await sample("after-cleanup");
|
||||
watchReceipt = watcher!.finish();
|
||||
const cancellation = runs[0]!.resultJson?.nativeCancellation;
|
||||
const toolResult = notices.find(n => n.stage === "tool" && n.toolCallId === request.toolCallId && n.status === "failed")!;
|
||||
const terminalFrame = runEvents.find(r => ["turn.cancelled", "turn.interrupted"].includes(r.eventType) && r.payload?.prpEvent?.turnId === request.turnId)?.payload.prpEvent;
|
||||
check("correlated-cancel-terminal", Boolean(terminalFrame) && terminalFrame.runId === request.runId, "An actual durable cancellation terminal belongs to the denied turn");
|
||||
const terminalAt = Date.parse(terminalFrame.emittedAt);
|
||||
const evidence: CopilotDeniedWriteEvidence = {
|
||||
expected: copilotOrigin(request), requestId: request.requestId!, expectedRelativePath: target,
|
||||
request: { ...request, requestId: request.requestId!, targetRelativePath: request.target!, method: "session/request_permission", offeredActions: request.declineOffered ? ["decline"] : [] },
|
||||
decision: { ...request, observedAtMs: clickedAtMs, requestId: request.requestId!, browserRequestId: request.requestId!, action: "decline" },
|
||||
deliveredDecision: (() => { const n = notices.find(n => n.stage === "permission_delivered" && n.requestId === request.requestId && n.outcome === "reject_once"); return n ? { ...n, requestId: n.requestId!, outcome: n.outcome! } : null; })(),
|
||||
toolResult: { ...toolResult, status: "failed" },
|
||||
terminal: { runId: terminalFrame.runId, turnId: terminalFrame.turnId, observedAtMs: terminalAt, status: runs[0]!.status },
|
||||
cancellation: { requestedAtMs: cancelRequestedAtMs, acknowledged: cancellation?.dispatchState === "acknowledged" && cancellation?.dispatched === true, scope: cancellation?.scope },
|
||||
cleanup: { observedAtMs: fileObservations.at(-1)!.observedAtMs, ownedProcessesRemaining: processes.live.length }, fileObservations, mutationObservation: watchReceipt,
|
||||
nativeAttemptsForTarget: new Set(notices.filter(n => n.stage === "tool" && n.operation === "edit" && n.target === target && n.status === "pending").map(n => n.toolCallId)).size,
|
||||
};
|
||||
await input.evidence("copilot-denial-proof.json", { evidence, processes, notices });
|
||||
const grade = gradeCopilotDeniedWrite(evidence); check("denial-without-side-effects", grade.passed, grade.failures.join(", ") || "Exact browser denial, explicit cancellation and absence through process cleanup");
|
||||
check("negative-task-unfinished", issue.status === "in_progress", "The negative test does not claim the task is done");
|
||||
check("no-extra-native-operation", new Set(notices.filter(n => n.stage === "tool" && n.status === "pending").map(n => n.toolCallId)).size === 1, "No alternate native edit, command or delegated operation is permitted");
|
||||
} else {
|
||||
await wait("attached command and task settlement", s => s.issue.status === "done" && s.runs[0]?.status === "succeeded" && s.processes.captured && s.processes.live.length === 0);
|
||||
const call = notices.find(n => n.stage === "tool" && n.status === "pending" && n.commandSha256 === command!.commandSha256);
|
||||
check("single-exact-command", Boolean(call) && new Set(notices.filter(n => n.stage === "tool" && n.status === "pending" && n.commandSha256 === command!.commandSha256).map(n => n.toolCallId)).size === 1, "Exactly one native execution contains the supplied command digest");
|
||||
const started = notices.find(n => n.toolCallId === call!.toolCallId && n.shellState === "started");
|
||||
const result = notices.find(n => n.commandToolCallId === call!.toolCallId && n.shellState === "completed");
|
||||
const terminal = runEvents.find(r => r.eventType === "turn.completed" && r.payload?.prpEvent?.turnId === call!.turnId)?.payload.prpEvent;
|
||||
const external = command!.snapshot();
|
||||
check("trusted-command-exit", !external.failure && external.connections === 1 && external.childGone && external.clientGone, "Fixed controller-owned child exited and its native client is gone");
|
||||
const markerMatches = await readFile(markerPath, "utf8") === command!.marker;
|
||||
check("native-client-before-terminal", Boolean(terminal) && external.clientExitedAtMs !== null && external.clientExitedAtMs < Date.parse(terminal.emittedAt), "Independent PID/start observation confirms native client retirement before turn completion");
|
||||
check("marker-before-terminal", Boolean(terminal) && external.markerWrittenAtMs !== null && external.markerWrittenAtMs < Date.parse(terminal.emittedAt), "The independent fixture wrote its undisclosed marker before turn completion");
|
||||
const grade = gradeCopilotAttachedSettlement({ expected: copilotOrigin(call!), nativeCall: call ? { ...call, operation: call.operation!, mode: call.mode!, detach: call.detach!, commandSha256: call.commandSha256! } : null,
|
||||
expectedCommandSha256: command!.commandSha256, commandExit: external.commandExit,
|
||||
expectedShellId: started?.shellId ?? "", nativeShellResult: result ? { ...result, shellId: result.shellId!, commandToolCallId: result.commandToolCallId!, status: result.status!, exitCode: result.exitCode! } : null,
|
||||
terminal: terminal ? { observedAtMs: Date.parse(terminal.emittedAt), runId: terminal.runId, turnId: terminal.turnId, status: "succeeded" } : null,
|
||||
cleanup: { observedAtMs: Date.now(), ownedProcessesRemaining: processes.live.length }, terminalMarkerMatches: markerMatches, afterCleanupMarkerMatches: markerMatches });
|
||||
await input.evidence("copilot-attached-proof.json", { external, processes, notices, grade, commandSha256: command!.commandSha256, markerMatches });
|
||||
check("attached-settlement-before-terminal", grade.passed, grade.failures.join(", ") || "Owned finite process and native shell settled before the actual turn terminal");
|
||||
}
|
||||
await load(); check("one-native-run", runs.length === 1 && runs[0]!.runtimeMode === "native", "Exactly one native run was accounted");
|
||||
const comments = await api.get<Row[]>(`/api/issues/${issue.id}/comments`), interactions = await api.get<Row[]>(`/api/issues/${issue.id}/interactions`);
|
||||
await input.evidence("api-state.json", { issue, run: runs[0], runs, comments, interactions, checks, runEvents, runEventsByRun: [{ runId: runs[0]!.id, events: runEvents }] });
|
||||
await page.reload();
|
||||
const label = deny ? "In Progress" : "Done";
|
||||
await expect(page.getByTestId("issue-detail-header").getByRole("button", { name: `Change status (current: ${label})`, exact: true })).toBeVisible();
|
||||
if (!deny) check("exact-completion-marker", comments.filter(c => c.authorAgentId === fixtures.agent.id && c.body?.trim() === execution.task.buildVisibleMarker(nonce)).length === 1, "The successful attached task has exactly one terminal marker");
|
||||
await input.capture("final-state", "Copilot protection outcome", "final-state.png");
|
||||
return { issue, runs, checks };
|
||||
} finally {
|
||||
watchReceipt ??= watcher?.finish();
|
||||
try { await command?.close(); }
|
||||
finally { await input.evidence("copilot-protection-checks.json", { issue, runs, checks, fileObservations, watchReceipt, processes }); }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
import type { RunnerTaskFixture } from "./types.js";
|
||||
import { copilotProtectionCases } from "./copilot-protection-cases.js";
|
||||
export const copilotProtectionTasks: readonly RunnerTaskFixture[] = copilotProtectionCases.map(c => ({
|
||||
id: c.id, label: c.id === "native-permission-deny-write" ? "Deny one native write, then stop safely" : "Attached command settles before completion",
|
||||
groups: [], workMode: "standard", flow: "copilot_protection", expectedRunCount: 1,
|
||||
attemptTimeoutMs: { local: 300_000, daytona: 300_000 },
|
||||
expectedTerminalState: { issue: c.id === "native-permission-deny-write" ? "in_progress" : "done", run: c.id === "native-permission-deny-write" ? "cancelled" : "succeeded" },
|
||||
buildTitle: nonce => `Copilot ${c.id} ${nonce}`,
|
||||
buildVisibleMarker: nonce => `COPILOT-PROTECTION-${c.id}-${nonce}`,
|
||||
buildPrompt: nonce => c.id === "native-permission-deny-write" ? c.prompt(nonce) : [
|
||||
"Run the supplied exact command once using native bash with mode async and detach false.",
|
||||
"After starting it, attempt to finish immediately without calling read_bash or another waiting tool. The attached-process runtime must settle the command before ending the turn.",
|
||||
"Do not modify fixture code, manufacture its result, or start another command.",
|
||||
`Call paperclip_finish with reportedWorkDisposition done, summary COPILOT-PROTECTION-${c.id}-${nonce}, current completion contract revision, satisfied objective criterion, no remaining work, evidence [], verification []. After acceptance reply with that exact summary.`,
|
||||
].join("\n"),
|
||||
buildMatchers: () => [], // Flow grades public durable origin and independent OS/filesystem observations.
|
||||
}));
|
||||
@@ -1,3 +1,4 @@
|
||||
import { runCopilotProtectionFlow } from "./copilot-protection-flow.js";
|
||||
import { runPiNativeFlow } from "./pi-native-flow.js";
|
||||
import { completionQualityControls, completionQualityStatus, judgeCompletionQuality, reserveCompletionQuality, type CompletionQualityRecord } from "./completion-quality.js";
|
||||
import { completionDelivery, type CompletionObservation } from "./completion-updates.js";
|
||||
@@ -550,7 +551,7 @@ for (const execution of executions) {
|
||||
const credentials = credentialValues();
|
||||
const secrets = normalizedSecrets(Object.values(credentials));
|
||||
const api = new RunnerApi(request);
|
||||
const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native"].includes(execution.task.flow);
|
||||
const companyRunFlow = ["continuation_accounting", "continuation", "context_integrity", "agent_chat", "everyday_workflow", "first_task", "instruction_persistence", "pi_native", "copilot_protection"].includes(execution.task.flow);
|
||||
const consoleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const networkDiagnostics: Array<Record<string, unknown>> = [];
|
||||
const pageLifecycleDiagnostics: Array<Record<string, unknown>> = [];
|
||||
@@ -897,6 +898,15 @@ for (const execution of executions) {
|
||||
});
|
||||
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
|
||||
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `piNative.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
} else if (execution.task.flow === "copilot_protection") {
|
||||
const story = await runCopilotProtectionFlow({
|
||||
page, api, fixtures, execution, nonce, workspacePath, deadlineAt: startedAtMs + deadlineMs,
|
||||
observe: (currentIssue, currentRuns) => { issue = currentIssue as IssueRecord; selectedRuns = currentRuns as RunRecord[]; },
|
||||
capture: captureScreenshot,
|
||||
evidence: (name, data) => writeSanitizedJson(snapshotsDir, name, data, secrets),
|
||||
});
|
||||
issue = story.issue as IssueRecord; selectedRuns = story.runs as RunRecord[];
|
||||
matcherResults = story.checks.map(check => ({ matcher: { kind: "json_path" as const, path: `copilotProtection.${check.id}`, expected: true }, passed: check.passed, detail: check.detail }));
|
||||
} else if (execution.task.flow === "instruction_persistence") {
|
||||
const story = await runInstructionPersistenceFlow({
|
||||
page, api, fixtures, execution, nonce, secrets, deadlineAt: startedAtMs + deadlineMs,
|
||||
|
||||
@@ -29,7 +29,8 @@ export type RunnerTaskFlow =
|
||||
| "plan_approval_completion"
|
||||
| "warm_three_turn"
|
||||
| "instruction_persistence"
|
||||
| "pi_native";
|
||||
| "pi_native"
|
||||
| "copilot_protection";
|
||||
|
||||
export interface SecretReference {
|
||||
type: "secret_ref";
|
||||
|
||||
Reference in new issue
Block a user