test(runner-e2e): verify pending Pi input after provider loss

Add one explicit Daytona Product cell that admits the exact Pi child before faulting it, then requires production expiry of the original native question, failed-run Blocked disposition, stale-answer rejection, distinct unanswered fallback, and no replay through owned retirement. Retain the existing local scope and pending qualification.

Calibrate public lifecycle evidence, generated observer one-shot dispatch, and candidate failure classification. Runtime, provider profile, dependency closure, deadlines, and lockfiles are unchanged.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip committed 2026-10-01 19:26:25 -05:00
1 parent 30f5bc57b5
commit 4c6adcadcb
15 files changed
+462 -45

No files matched your search

+5 -2
View File
@@ -35,8 +35,11 @@ Pi's explicit-only `pi-controls` Product suite tests Stop while native permissio
is unanswered and browser-originated same-turn steering, locally and on Daytona.
Its [fixture contract](../tests/runner-e2e/FIXTURES.md#pi-active-controls) separates
control acknowledgment, actual message consumption and owned process retirement.
The current Pi matrix has 25 explicit cells (13 local and 12 Daytona), including
these four controls and pending-native-question controller restart. File editing
The current Pi matrix has 26 explicit cells (13 local and 13 Daytona), including
these four controls, pending-native-question controller restart, and a Daytona-only
exact-Pi-child provider-death journey. Provider death must expire the original
unanswered card and reject stale replies without a replacement run; any durable
fallback is distinct from restoration. File editing
also requires native edit/validation events and a registered artifact download.
All remain unqualified until measured on the exact candidate runtime and harness.
+28 -9
View File
@@ -350,21 +350,40 @@ a candidate failure; transport/infrastructure failures retain the existing
harness classification. Positive and plausible-wrong/missing-evidence unit
calibrations do not count as paid qualification. All automatic retries stay zero.
Provider-death coverage remains bounded: the existing patched-ACP package test
`provider exit fails promptly and missing owned extension fails admission` runs
an actual wrapper child against a deterministic fake Pi child. The bridge test
`expires pending requests and discards a late answer exactly once` and driver
abort/event-stream tests separately cover pending cleanup with mocked transport.
They do not establish a single actual native-provider-death to durable browser
request-expiry chain. No local or Daytona paid death case is claimed, and no
arbitrary process-kill API is introduced by these fixtures.
`native-pending-provider-death` adds one real Daytona-only Product journey.
After the native input is publicly durable and still unanswered, the existing
owned remote observer admits the exact Pi child through its verified wrapper
parent, source-pinned closure files, executable inode, workspace, run/lease/session
ancestry and fresh PID/start-time checks. Pi overwrites Linux argv via
`process.title`, so the private receipt explicitly uses pinned-parent entrypoint
attribution and never claims original child argv. A pidfd targets only that child;
worker death, broad process-name matching and controller Stop cannot substitute.
The runtime itself must emit `runtime_request.expired` for the original callback
with `provider_process_lost` and `replayAllowed:false`, followed by native turn
failure. The permanent failed-terminal recovery projection must put the owned
issue in Blocked, proved through API and browser and retained through cleanup.
The original durable card must expire, retain zero answers and lose its browser
answer controls; any supersession must name the separate fallback card.
Both public stale-response APIs must reject the old answer. Any production-created
`wake_assignee` fallback is separately identified and remains unanswered; it is not
a restored native callback. The sealed observer proves no continuation marker was
created and all owned processes retired before public lease deletion. The company
must retain exactly the original run through cleanup.
Local provider-death remains excluded. The Python admission tests include a
real pidfd calibration against a synthetic title-changing child on native Linux;
the normal E2E unit wrapper invokes it with no provider credentials or network.
macOS runs metadata negatives and explicitly skips this Linux-only calibration.
That calibration and the earlier fake-Pi wrapper/bridge tests do not count as the
real paid Product lifecycle proof. This new candidate cell remains unqualified.
## Pi active controls
The explicit-only `pi-controls` suite adds `pending-permission-stop` and
`same-turn-steering` on local and Daytona, each with one provider run, a
120-second active-turn timeout and a 300-second attempt budget. These four
control cases retain their behavior; the current matrix totals 25 Pi cells.
control cases retain their behavior; the current matrix totals 26 Pi cells.
Pi 1/profile 11 and coverage revisions intentionally change the affected suite
fingerprints, so older qualification receipts cannot be reused. Catalog presence and
deterministic calibration do not constitute paid qualification.
+19 -2
View File
@@ -1437,15 +1437,32 @@ mock control plane. Neither suite substitutes for the other.
### Pi native Product fixtures
The separate `pi-native` suite defines nine explicit Pi cells: native questions,
The separate `pi-native` suite defines ten explicit Pi cells: native questions,
agent-file persistence, browser permission denial and controller restart with an
unanswered native question on local and Daytona, plus
a local-only automatic `restrictive-denial` case. The latter cannot use the remote
a local-only automatic `restrictive-denial` case and one Daytona-only
`native-pending-provider-death` case. The latter cannot use the remote
setup read under `deny-all`, so it is explicitly excluded there. It uses the
pinned candidate profile and exact OpenRouter model. It is excluded from `--all`,
never automatically retries, and retains qualification as pending until live proof.
Reserve and reconcile each paid cell just as for `extended-harnesses`.
Provider death is a real, one-run paid Product journey, distinct from controller
restart and Stop. The owned Linux observer pins the verified wrapper ancestry,
closure and executable inode before signalling exactly its unique Pi child through
a pidfd. Pi overwrites its argv with `process.title`; entrypoint attribution is
explicitly parent-attested, not an original-child-argv claim. The fixture requires
production-generated non-replayable expiry, an expired original browser card,
rejection of stale answers by both public APIs, no continuation file effect, and
no second run. Any durable `wake_assignee` fallback stays separately identified
and unanswered. Full process/run/lease evidence remains private. Local provider
death is excluded because this exact ownership mechanism requires Linux pidfd.
The Python helper calibration uses a synthetic transport and a real title-changing
Node child; it proves fault ownership only, never paid Pi lifecycle behavior. Run
it on native Linux with pinned Node on PATH: `python3 tests/runner-e2e/pi-provider-fault.test.py`.
This adds one cell to the pending Pi Product matrix (26: 13 local, 13 Daytona);
no qualification or live success is implied by discovery or oracle tests.
```sh
pnpm test:e2e:runner -- --list --suite pi-native
pnpm test:e2e:runner -- --id pi-native.runner-acpx-pi.local.native-questions
+3 -3
View File
@@ -136,10 +136,10 @@ describe("runner E2E catalog", () => {
expect(localIntegrityTasks).toHaveLength(2);
expect(openRouterBreadthTasks).toHaveLength(3);
expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([
4, 8, 9, 4, 4, 6, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
4, 8, 10, 4, 4, 6, 6, 30, 3, 16, 16, 2, 6, 8, 46, 23, 47, 20, 52, 28, 18, 6, 6, 12, 10, 48, 16, 10, 2, 1, 1,
]);
expect(validateRunnerCatalog()).toHaveLength(468);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(468);
expect(validateRunnerCatalog()).toHaveLength(469);
expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(469);
expect(
runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"),
).toHaveLength(48);
+3 -3
View File
@@ -1103,9 +1103,9 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [
id: "pi-native", label: "Pi native boundaries", manualOnly: true,
description: "Pi native forms, registered agent files and human permission denial on local and Daytona execution; automatic deny-all remains local-only.",
groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"),
environments: runnerEnvironments, tasks: piNativeTasks, expectedMatrixSize: 9,
excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial"],
definitionMetadata: { version: 3, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request" },
environments: runnerEnvironments, tasks: piNativeTasks, expectedMatrixSize: 10,
excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial", "pi-native.runner-acpx-pi.local.native-pending-provider-death"],
definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, providerDeath: "daytona-exact-pi-child-pidfd-production-expiry", remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request" },
},
{
id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true,
+3
View File
@@ -34,6 +34,9 @@ export function classifyFailure(error: unknown): FailureClass {
: "cleanup_failure";
}
if (PERMANENT.test(message)) return "permanent_infrastructure";
// A deliberately killed Pi child is the fixture stimulus. Missing lifecycle
// proof must not become a retryable transport failure because of its wording.
if (/pi_provider_death_proof:/.test(message)) return "candidate_failure";
if (
/chat_idle_state_invariant/.test(message) ||
NON_RETRYABLE_SESSION_CLOSE.test(message) ||
@@ -124,24 +124,24 @@ describe("Pi controls catalog admission", () => {
expect(cells).toHaveLength(4); expect(cells.every(c => c.profile.qualificationCandidate === "pi" && c.task.expectedRunCount === 1)).toBe(true);
expect(cells.map(c => `${c.environment.id}/${c.task.id}`).sort()).toEqual(["daytona/pending-permission-stop", "daytona/same-turn-steering", "local/pending-permission-stop", "local/same-turn-steering"]);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(c => c.suite.id === "pi-controls")).toBe(false);
expect(validateRunnerCatalog()).toHaveLength(468);
expect(validateRunnerCatalog()).toHaveLength(469);
for (const cell of cells) expect(JSON.parse(buildRunnerE2EProcessEnvironment({}, [cell]).PAPERCLIP_RUNNER_ACPX_QUALIFICATION!)).toEqual([{ agent: "pi", model: cell.profile.model }]);
expect(() => assertRemoteNativeEvidencePrerequisites(cells, {})).toThrow();
});
it("pins the Pi 1 profile and versioned coverage while retaining active Stop identity", () => {
// Pi 1/profile 11 changes profile-bearing definitions. Coverage v3/v2 adds
// pending restart and the strict file oracle; no runtime admission is promoted.
// Pi 1/profile 11 changes profile-bearing definitions. Coverage v4/v2 adds
// provider death, pending restart and the strict file oracle; no runtime admission is promoted.
const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile;
expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter");
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 3, profileVersion: 11 });
expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 4, profileVersion: 11 });
expect(runnerSuites.find(s => s.id === "extended-harnesses")!.definitionMetadata).toMatchObject({ version: 2 });
const hashes = Object.fromEntries(runnerSuites.filter(s => ["pi-native", "native-active-stop", "extended-harnesses", "rich-acp-warm-continuity"].includes(s.id)).map(s => [s.id, suiteDefinitionHash(s)]));
expect(hashes).toEqual({
"pi-native": "00d4edfd0d61772977434389cb601bbdc08173db4a33fd1e153d2e18f488f597",
"pi-native": "8c4d4f1cba234223ee54b5a19d5fe3f815424e7335cf15f43bd3f77d342aec74",
"native-active-stop": "99682b2b106d816a011834fae5a944ed7729958893709d5b83a19b6f595e7e4d",
"rich-acp-warm-continuity": "25e69f031696aeb459e6722a056ae0802379f9137799648fdd1945881f32f41a",
"extended-harnesses": "e50f79cf604bffb2cdee3ad4e8bc1d106045a4f44ef07e877032ff5b8960c8ab",
});
expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi" && c.suite.id !== "pi-controls")).toHaveLength(21);
expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi" && c.suite.id !== "pi-controls")).toHaveLength(22);
});
});
+5 -5
View File
@@ -5,11 +5,11 @@ import { buildRunnerE2EProcessEnvironment } from "./harness-env.js";
import { parseRunnerSelectors, selectRunnerExecutions } from "./selectors.js";
describe("Pi native Product qualification", () => {
it("selects five local and four remote Pi cases without changing the basic extended matrix", () => {
it("selects five local and five remote Pi cases without changing the basic extended matrix", () => {
const suite = runnerSuites.find(row => row.id === "pi-native")!;
expect(suite.manualOnly).toBe(true); expect(suite.expectedMatrixSize).toBe(9);
expect(suite.manualOnly).toBe(true); expect(suite.expectedMatrixSize).toBe(10);
const cells = runnerMatrix.filter(row => row.suite.id === suite.id);
expect(cells).toHaveLength(9);
expect(cells).toHaveLength(10);
expect(cells.every(row => row.profile.qualificationCandidate === "pi")).toBe(true);
expect(cells.filter(row => row.environment.id === "local").map(row => [row.task.id, row.task.expectedRunCount])).toEqual([
["native-questions", 1], ["agent-files-fresh-run", 2], ["restrictive-denial", 1], ["human-permission-denial", 1],
@@ -17,9 +17,9 @@ describe("Pi native Product qualification", () => {
]);
expect(cells.filter(row => row.environment.id === "daytona").map(row => [row.task.id, row.task.expectedRunCount])).toEqual([
["native-questions", 1], ["agent-files-fresh-run", 2], ["human-permission-denial", 1],
["native-pending-controller-restart", 1],
["native-pending-controller-restart", 1], ["native-pending-provider-death", 1],
]);
expect(cells.reduce((sum, row) => sum + row.task.expectedRunCount, 0)).toBe(11);
expect(cells.reduce((sum, row) => sum + row.task.expectedRunCount, 0)).toBe(12);
expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(row => row.suite.id === suite.id)).toBe(false);
expect(runnerMatrix.filter(row => row.suite.id === "extended-harnesses")).toHaveLength(30);
expect(piNativeTasks[0]!.buildPrompt("fixture")).toContain("paperclip_native_question");
+7 -1
View File
@@ -7,10 +7,11 @@ export const piNativeTasks: readonly RunnerTaskFixture[] = [
["restrictive-denial", "Restrictive native write denial prevents a file effect", 1],
["human-permission-denial", "Browser denial prevents a native write through process retirement", 1],
["native-pending-controller-restart", "Pending native question survives controller restart in the same live run", 1],
["native-pending-provider-death", "Lost Pi provider expires its original unanswered native question", 1],
].map(([id, label, turns]) => ({
id: String(id), label: String(label), groups: [], workMode: "standard", flow: "pi_native",
expectedRunCount: Number(turns), attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 },
expectedTerminalState: { issue: "done", run: "succeeded" },
expectedTerminalState: id === "native-pending-provider-death" ? { issue: "blocked", run: "failed" } : { issue: "done", run: "succeeded" },
buildTitle: nonce => `Pi ${id} ${nonce}`,
buildVisibleMarker: nonce => `PI-NATIVE-${id}-${nonce}`,
buildPrompt: nonce => piNativePrompt(String(id), nonce),
@@ -22,6 +23,11 @@ export function piNativeFinish(marker: string): string {
}
export function piNativePrompt(id: string, nonce: string): string {
if (id === "native-pending-provider-death") return [
'Call paperclip_native_question exactly once with {method:"input",title:"Pi native provider death",placeholder:"Answer"}. Wait for the real answer; never guess it or substitute semantic request_human_input.',
"Only after receiving the actual answer, use native write to save its exact tool-result object as JSON to pi-provider-death-answer.json in the task workspace. Never write that file while the question is unanswered. This is an internal verification file, not a deliverable.",
piNativeFinish(`PI-NATIVE-${id}-${nonce}`),
].join("\n");
if (id === "native-pending-controller-restart") return [
'Call paperclip_native_question exactly once with {method:"input",title:"Pi native restart",placeholder:"Answer"}. Wait for the real answer; never guess it or substitute semantic request_human_input.',
"Use native write to save the exact returned tool-result object as JSON to pi-native-restart-answer.json in the task workspace. Do not change or infer its fields. This is an internal verification file, not a deliverable.",
+16 -1
View File
@@ -12,6 +12,7 @@ import type { LiveFixtureValues } from "./live-fixtures.js";
import type { MatrixExecution } from "./types.js";
import type { RemoteNativeFixture, RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { runPiPendingProviderDeath, PI_DEATH_MARKER } from "./pi-native-provider-death-flow.js";
import { runPiPendingControllerRestart } from "./pi-native-restart-flow.js";
export interface PiRemoteBootstrap {
@@ -93,7 +94,21 @@ export async function runPiNativeFlow(input: {
return result;
}
try {
if (execution.task.id === "native-pending-controller-restart") {
if (execution.task.id === "native-pending-provider-death") {
if (!remote) throw new Error("Exact Pi-child fault is available only on owned Daytona Linux");
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce), { targets: [PI_DEATH_MARKER] });
checks.push(...await runPiPendingProviderDeath({ page, api, companyId: fixtures.company.id, deadlineAt: input.deadlineAt,
fixture: currentRemote!, load, events, capture: input.capture, evidence: input.evidence }));
const originalRunId = runs[0]!.id;
input.registerCleanupAssertion!(async () => {
const state = await load();
const passed = state.issue.status === "blocked" && state.runs.length === 1 && state.runs[0]?.id === originalRunId && state.runs[0]?.status === "failed"
&& state.interactions.every(card => card.status === "expired" || (card.status === "pending" && card.continuationPolicy === "wake_assignee"));
await input.evidence("pi-provider-death-final-no-replay.json", { ...state, passed });
if (!passed) throw new Error("Provider loss replayed a run or consumed the unanswered fallback during cleanup");
return [{ id: "death-no-replay-through-cleanup", passed, detail: "Original failed run remains the only run through cleanup" }];
});
} else if (execution.task.id === "native-pending-controller-restart") {
await create(execution.task.buildTitle(nonce), execution.task.buildPrompt(nonce), { targets: ["pi-native-restart-answer.json"] });
checks.push(...await runPiPendingControllerRestart({
page, companyId: fixtures.company.id, deadlineAt: input.deadlineAt, load, events,
@@ -0,0 +1,116 @@
import { expect, it, vi } from "vitest";
import { gradePiProviderDeath, runPiPendingProviderDeath, PI_DEATH_MARKER } from "./pi-native-provider-death-flow.js";
import { classifyFailure } from "./failure-classifier.js";
import { observePiPendingNativeInput } from "./pi-native-restart-flow.js";
type Row = Record<string, any>;
vi.mock("@playwright/test", () => ({ expect: (locator: { count(): number }) => ({
toHaveCount: async (expected: number) => expect(locator.count()).toBe(expected),
toBeVisible: async () => expect(locator.count()).toBe(1),
}) }));
function fixture(fallback = true) {
const question = { schema: "paperclip.question_set.v1", questions: [{ id: "answer", header: "Pi native provider death", prompt: "Answer", answerMode: "text", required: true }] };
const request = { schema: "paperclip.runtime_request.v2", requestKind: "runtime", type: "input", status: "pending", requestId: "request", turnId: "turn", itemId: "request", input: question,
origin: { adapter: "acpx-runtime-sidecar", provider: "pi", method: "elicitation/create" } };
const wrap = (eventType: string, sourceSeq: number, payload: Row): Row => ({ companyId: "company", runId: "run", seq: sourceSeq, protocolSchemaVersion: 1, eventType,
payload: { prpEvent: { schema: "paperclip.prp.event.v1", schemaVersion: 1, sourceKind: "runner", eventType, runId: "run", turnId: "turn", itemId: "request",
normalizedSessionId: "session", sourceInstanceId: "runner", sourceSeq, sourceEventId: `runner:run:${sourceSeq}`, payload } } });
const state = { issue: { id: "issue", companyId: "company", status: "in_progress" }, runs: [{ id: "run", companyId: "company", nativeIssueId: "issue", runtimeMode: "native", status: "running", nativeSessionId: "session", runnerInstanceId: "runner" } as Row],
interactions: [{ id: "original", companyId: "company", issueId: "issue", kind: "ask_user_questions", sourceRunId: "run", status: "pending", result: null,
continuationPolicy: "none", resolverPolicy: "human_only", idempotencyKey: "paperclip-runner-question:run:request", payload: { runtimeRequestId: "request", questionSet: question } } as Row] };
const events = [wrap("turn.started", 1, {}), wrap("runtime_request.created", 2, { request })];
const pending = observePiPendingNativeInput(state, events, "company", "Pi native provider death");
const before = structuredClone({ state, events });
state.runs[0]!.status = "failed"; state.issue.status = "blocked"; state.interactions[0]!.status = "expired";
events.push(wrap("runtime_request.expired", 3, { requestId: "request", requestKind: "runtime", requestType: "input", turnId: "turn", itemId: "request", reason: "provider_process_lost", replayAllowed: false, request }),
wrap("turn.failed", 4, { code: "provider_transport_lost" }));
if (fallback) state.interactions.push({ ...structuredClone(state.interactions[0]!), id: "fallback", status: "pending", continuationPolicy: "wake_assignee", idempotencyKey: "runtime-input-durable:v1:run:request" });
state.interactions[0]!.result = { version: 1, answers: [], ...(fallback ? { expirationReason: "superseded_by_newer_interaction", supersededByInteractionId: "fallback" } : {}) };
const grade = () => gradePiProviderDeath(state, events, pending, question);
return { state, events, pending, question, wrap, grade, before };
}
it.each([false, true])("accepts production loss with distinct fallback=%s without claiming restoration", fallback => {
expect(fixture(fallback).grade()).toEqual({ originalCardExpired: true, replayAllowed: false, originalProviderRestored: false, fallbackInteractionId: fallback ? "fallback" : null });
});
it.each([
["wrong issue disposition", (f: ReturnType<typeof fixture>) => { f.state.issue.status = "in_progress"; }],
["completed issue", (f: ReturnType<typeof fixture>) => { f.state.issue.status = "done"; }],
["answered expired card", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result.answers = [{ questionId: "answer", otherText: "stale" }]; }],
["missing expired result", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result = null; }],
["foreign supersession", (f: ReturnType<typeof fixture>) => { f.state.interactions[0]!.result.supersededByInteractionId = "other"; }],
["missing expiry", (f: ReturnType<typeof fixture>) => { f.events.splice(2, 1); }],
["model-only claim", f => { f.events[2] = { eventType: "assistant.message", payload: { text: "The callback expired" } }; }],
["controller injected expiry", f => { f.events[2]!.payload.prpEvent.sourceKind = "control_plane"; }],
["cancelled instead of lost", f => { f.events[2]!.eventType = f.events[2]!.payload.prpEvent.eventType = "runtime_request.cancelled"; }],
["resolved instead of lost", f => { f.events[2]!.eventType = f.events[2]!.payload.prpEvent.eventType = "runtime_request.resolved"; }],
["normal durable handoff", f => { f.events[2]!.payload.prpEvent.payload.reason = "durable_handoff"; }],
["replay allowed", f => { f.events[2]!.payload.prpEvent.payload.replayAllowed = true; }],
["foreign request", f => { f.events[2]!.payload.prpEvent.payload.requestId = "other"; }],
["foreign item", f => { f.events[2]!.payload.prpEvent.payload.itemId = "other"; }],
["missing canonical request", f => { delete f.events[2]!.payload.prpEvent.payload.request; }],
["replaced canonical question", f => { f.events[2]!.payload.prpEvent.payload.request = { ...f.events[2]!.payload.prpEvent.payload.request, input: {} }; }],
["duplicate expiry", f => { f.events.push(f.wrap("runtime_request.expired", 5, f.events[2]!.payload.prpEvent.payload)); }],
["new native turn", f => { f.events.push(f.wrap("turn.started", 5, {})); }],
["successful provider", f => { f.state.runs[0]!.status = "succeeded"; }],
["Stop cancellation", f => { f.state.runs[0]!.resultJson = { nativeCancellation: {} }; }],
["worker failure without turn failure", f => { f.events.pop(); }],
["completed turn", f => { f.events[3]!.eventType = f.events[3]!.payload.prpEvent.eventType = "turn.completed"; }],
["terminal before expiry", f => { f.events[3]!.seq = 0; }],
["second run", f => { f.state.runs.push({ ...f.state.runs[0], id: "other" }); }],
["restored session", f => { f.state.runs[0]!.nativeSessionId = "restored"; }],
["foreign company", f => { f.events[2]!.companyId = "foreign"; }],
["foreign source", f => { f.events[2]!.payload.prpEvent.sourceInstanceId = "foreign"; }],
["original still pending", f => { f.state.interactions[0]!.status = "pending"; }],
["answered original", f => { f.state.interactions[0]!.status = "answered"; }],
["rebound original card", f => { f.state.interactions[0]!.sourceRunId = "other"; }],
["duplicate fallback", f => { f.state.interactions.push(structuredClone(f.state.interactions[1]!)); }],
["fallback masquerading as callback", f => { f.state.interactions[1]!.continuationPolicy = "none"; }],
["answered fallback", f => { f.state.interactions[1]!.status = "answered"; }],
["foreign fallback", f => { f.state.interactions[1]!.idempotencyKey = "foreign"; }],
] satisfies Array<[string, (f: ReturnType<typeof fixture>) => void]>)("rejects %s", (_name, mutate) => {
const f = fixture(); mutate(f); expect(f.grade).toThrow();
});
async function journey(staleStatus = 409, markerEffect = false) {
const f = fixture(), evidence = new Map<string, unknown>(), posted: string[] = [];
let lost = false, signals = 0, retired = false;
Object.assign(f.before.state.runs[0]!, { contextSnapshot: { executionWorkspaceId: "workspace", paperclipEnvironment: { leaseId: "lease" } },
runnerProfileJson: { nativeExecutionInput: { binding: { executionWorkspaceId: "workspace" } } } });
const root = { pid: 21, startTicks: "100", bootId: "boot" };
const composer: any = { count: () => 1, filter: () => composer };
const page: any = { getByTestId: (id: string) => id === "issue-detail-header" ? { getByRole: (_role: string, options: Row) => {
expect(options).toEqual({ name: "Change status (current: Blocked", exact: false }); return { count: () => 1 };
} } : composer, reload: async () => { expect(retired).toBe(true); },
locator: (selector: string) => { expect(selector).toBe("#interaction-original"); return { count: () => 1, locator: () => ({ count: () => 0 }) }; } };
const remoteFixture: any = { binding: { companyId: "company", runId: "run", leaseId: "lease" }, baseline: { scope: { observedPrpEnvironmentLeaseId: "workspace" } },
terminatePiProvider: async (lease: string) => { expect(lease).toBe("workspace"); expect(lost).toBe(false); signals++; lost = true; return { targetKind: "pi_native_child", signalled: true }; },
finish: async () => { expect(lost).toBe(true); retired = true; return { complete: true, observedAtMs: 1, watcher: { complete: true },
processes: { captured: true, root, journal: [root], live: [] }, targets: { [PI_DEATH_MARKER]: { absent: !markerEffect, complete: true, mutationCount: markerEffect ? 1 : 0 } } }; } };
const result = runPiPendingProviderDeath({ page, api: { request: { post: async (path: string) => { expect(retired).toBe(true); posted.push(path); return { status: () => staleStatus }; } } } as any,
companyId: "company", deadlineAt: Date.now() + 1000, fixture: remoteFixture,
load: async () => structuredClone(lost ? f.state : f.before.state), events: async () => structuredClone(lost ? f.events : f.before.events),
capture: async () => {}, evidence: async (name, value) => { evidence.set(name, value); } });
return { result, evidence, posted, signalCount: () => signals };
}
it("drives one admitted fault, retirement, original-card UI and both stale public APIs in order", async () => {
const j = await journey(); expect((await j.result).every(check => check.passed)).toBe(true);
expect(j.signalCount()).toBe(1); expect(j.posted).toEqual(["/api/issues/issue/interactions/original/respond", "/api/heartbeat-runs/run/runtime-requests/request/resolve"]);
expect(j.evidence.get("pi-provider-death-result.json")).toMatchObject({ originalProviderRestored: false, fallbackInteractionId: "fallback", staleCardStatus: 409, staleRuntimeStatus: 409 });
});
it("does not accept a successful stale answer despite otherwise valid expiry receipts", async () => {
const j = await journey(200); await expect(j.result).rejects.toThrow("Both original durable-card"); expect(j.signalCount()).toBe(1);
});
it("rejects a continuation file effect before sending either stale answer", async () => {
const j = await journey(409, true); await expect(j.result).rejects.toThrow("Exact owned process tree"); expect(j.posted).toEqual([]);
});
it("classifies deliberate-loss proof failures as candidate failures, preserving real API transport errors", () => {
const f = fixture(); f.state.runs[0]!.status = "succeeded";
let observed: unknown;
try { f.grade(); } catch (error) { observed = error; }
expect(String(observed)).toContain("pi_provider_death_proof:");
expect(classifyFailure(observed)).toBe("candidate_failure");
expect(classifyFailure(new Error("GET controller returned 503 service unavailable"))).toBe("transient_infrastructure");
expect(classifyFailure(new Error("pi_provider_death_proof: secret leak"))).toBe("secret_leak");
expect(classifyFailure(new Error("pi_provider_death_proof: cleanup incomplete"))).toBe("cleanup_failure");
});
@@ -0,0 +1,135 @@
import { createHash, randomBytes } from "node:crypto";
import { expect, type Page } from "@playwright/test";
import { canonicalJson } from "../../packages/shared/src/portability-hash.js";
import { pollUntil, type RunnerApi } from "./api.js";
import { isValidNativePrpEnvelope } from "./native-event-envelope.js";
import { observePiPendingNativeInput, type PiRestartPending } from "./pi-native-restart-flow.js";
import { hasPiRemoteRetirement } from "./pi-native-evidence.js";
import type { RemoteNativeFixture, PiProviderDeathReceipt } from "./remote-native-fixtures.js";
type Row = Record<string, any>;
type State = { issue: Row; runs: Row[]; interactions: Row[] };
type Check = { id: string; passed: boolean; detail: string };
const hash = (value: unknown) => createHash("sha256").update(canonicalJson(value)).digest("hex");
const demand = (value: unknown, reason: string): void => { if (!value) throw new Error(`pi_provider_death_proof: ${reason}`); };
export const PI_DEATH_MARKER = "pi-provider-death-answer.json";
/** All events are public durable production receipts, never a fault fixture's events. */
export function gradePiProviderDeath(state: State, events: readonly Row[], pending: PiRestartPending, originalQuestion: unknown) {
demand(state.issue.id === pending.issueId && state.issue.companyId === pending.companyId && state.issue.status === "blocked" && state.runs.length === 1, "foreign issue, wrong blocked disposition or second run");
const run = state.runs[0]!;
demand(run.id === pending.runId && run.companyId === pending.companyId && run.nativeIssueId === pending.issueId && run.runtimeMode === "native"
&& run.nativeSessionId === pending.nativeSessionId && run.runnerInstanceId === pending.sourceInstanceId
&& run.status === "failed" && run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null,
"original provider did not fail independently of Stop");
demand(Array.isArray(events) && events.length > 0 && events.length <= 20_000, "missing event history");
const rows = events.filter(row => row.payload?.prpEvent).sort((a, b) => a.seq - b.seq);
const ids = new Set<string>(), seqs = new Set<number>(), cursors = new Map<string, number>();
for (const row of rows) {
const e = row.payload.prpEvent;
demand(row.companyId === pending.companyId && row.runId === pending.runId && row.eventType === e.eventType
&& Number.isSafeInteger(row.seq) && row.seq > 0 && !seqs.has(row.seq) && isValidNativePrpEnvelope(e, row.protocolSchemaVersion)
&& e.runId === pending.runId && e.normalizedSessionId === pending.nativeSessionId
&& ((e.sourceKind === "runner" && e.sourceInstanceId === pending.sourceInstanceId)
|| (e.sourceKind === "control_plane" && e.sourceInstanceId === `${pending.sourceInstanceId}:control`))
&& e.sourceEventId === `${e.sourceInstanceId}:${pending.runId}:${e.sourceSeq}` && !ids.has(e.sourceEventId)
&& Number.isSafeInteger(e.sourceSeq) && e.sourceSeq > (cursors.get(e.sourceInstanceId) ?? 0), "foreign, duplicate or reordered event");
ids.add(e.sourceEventId); seqs.add(row.seq); cursors.set(e.sourceInstanceId, e.sourceSeq);
if (e.turnId != null) demand(e.turnId === pending.turnId, "replacement native turn");
}
const of = (kind: string) => rows.filter(row => row.eventType === kind);
const created = of("runtime_request.created");
demand(created.length === 1 && hash(created[0]) === pending.createdRowSha256, "original request changed or replayed");
const closures = rows.filter(row => ["runtime_request.expired", "runtime_request.resolved", "runtime_request.cancelled"].includes(row.eventType));
demand(closures.length === 1 && closures[0]!.eventType === "runtime_request.expired", "missing unique production expiry");
const expired = closures[0]!.payload.prpEvent, p = expired.payload;
demand(expired.sourceKind === "runner" && expired.sourceSeq > pending.createdSourceSeq && expired.turnId === pending.turnId
&& p.requestId === pending.requestId && p.requestKind === "runtime" && p.requestType === "input" && p.turnId === pending.turnId && p.itemId === pending.itemId
&& p.reason === "provider_process_lost" && p.replayAllowed === false && hash(p.request) === hash(created[0]!.payload.prpEvent.payload.request), "expiry is not the original lost Pi callback");
const failed = of("turn.failed");
demand(failed.length === 1 && failed[0]!.seq > closures[0]!.seq && failed[0]!.payload.prpEvent.sourceKind === "runner"
&& failed[0]!.payload.prpEvent.turnId === pending.turnId && !of("turn.completed").length && !of("turn.cancelled").length
&& of("turn.started").length === 1 && !of("run.result.proposed").length, "wrong terminal or replacement turn");
const card = state.interactions.find(card => card.id === pending.interactionId);
demand(card?.companyId === pending.companyId && card.issueId === pending.issueId && card.sourceRunId === pending.runId
&& card.idempotencyKey === `paperclip-runner-question:${pending.runId}:${pending.requestId}` && card.continuationPolicy === "none"
&& card.status === "expired" && card.result?.version === 1 && Array.isArray(card.result.answers) && card.result.answers.length === 0
&& card.kind === "ask_user_questions" && card.resolverPolicy === "human_only" && card.payload?.runtimeRequestId === pending.requestId && hash(card.payload.questionSet) === hash(originalQuestion), "original durable card did not expire");
const fallback = state.interactions.filter(card => card.id !== pending.interactionId);
demand(fallback.length <= 1, "duplicate fallback interaction");
if (fallback.length) {
const next = fallback[0]!;
demand(next.id !== pending.interactionId && next.companyId === pending.companyId && next.issueId === pending.issueId && next.sourceRunId === pending.runId
&& next.kind === "ask_user_questions" && next.status === "pending" && next.result == null && next.continuationPolicy === "wake_assignee"
&& next.idempotencyKey === `runtime-input-durable:v1:${pending.runId}:${pending.requestId}`
&& card!.result.expirationReason === "superseded_by_newer_interaction" && card!.result.supersededByInteractionId === next.id
&& next.payload?.runtimeRequestId === pending.requestId && hash(next.payload.questionSet) === hash(originalQuestion), "fallback was mistaken for restored input");
}
return { originalCardExpired: true, replayAllowed: false, originalProviderRestored: false, fallbackInteractionId: fallback[0]?.id ?? null };
}
export async function runPiPendingProviderDeath(input: {
page: Page; api: RunnerApi; companyId: string; deadlineAt: number; fixture: RemoteNativeFixture;
load(): Promise<State>; events(runId: string): Promise<Row[]>;
capture(id: string, label: string, file: string): Promise<void>; evidence(name: string, value: unknown): Promise<void>;
}): Promise<Check[]> {
const checks: Check[] = [];
const check = (id: string, passed: boolean, detail: string) => { checks.push({ id, passed, detail }); demand(passed, detail); };
let fault: PiProviderDeathReceipt | undefined;
const snapshot = async (name: string) => {
const state = await input.load(), events = state.runs.length === 1 ? await input.events(state.runs[0]!.id) : [];
await input.evidence(name, { ...state, events, fault }); return { state, events };
};
try {
await pollUntil({ label: "Pi native input before provider death", deadlineAt: input.deadlineAt, load: input.load,
accept: s => s.interactions.some(c => c.status === "pending"), reject: s => s.runs.length > 1 || s.runs.some(r => ["succeeded", "failed", "cancelled", "timed_out"].includes(r.status)) ? "Provider ended before fault admission" : undefined });
const before = await snapshot("pi-provider-death-before.json");
const pending = observePiPendingNativeInput(before.state, before.events, input.companyId, "Pi native provider death");
const run = before.state.runs[0]!, question = before.state.interactions[0]!.payload.questionSet;
demand(input.fixture.binding.runId === pending.runId && input.fixture.binding.companyId === pending.companyId && input.fixture.terminatePiProvider,
"missing exact owned Daytona fault capability");
// createRunnerdBackend's default PRP lease is the execution workspace ID;
// reject alternate/recovered authority instead of guessing that identity.
const workspaceId = run.contextSnapshot?.executionWorkspaceId;
demand(typeof workspaceId === "string" && workspaceId.length > 0
&& run.runnerProfileJson?.nativeExecutionInput?.binding?.executionWorkspaceId === workspaceId
&& run.contextSnapshot?.paperclipEnvironment?.leaseId === input.fixture.binding.leaseId
&& input.fixture.baseline.scope.observedPrpEnvironmentLeaseId === workspaceId, "native run/lease/workspace binding unavailable");
await expect(input.page.getByTestId("question-text-answer-composer").filter({ visible: true })).toHaveCount(1);
await input.capture("pi-provider-death-pending", "Unanswered native Pi input before provider loss", "pi-provider-death-pending.png");
// Refresh immediately before the one-shot irreversible fault.
const fresh = await snapshot("pi-provider-death-admission.json");
check("death-original-pending-input", hash(observePiPendingNativeInput(fresh.state, fresh.events, input.companyId, "Pi native provider death")) === hash(pending), "Fault admits only the unchanged original native input");
fault = await input.fixture.terminatePiProvider!(workspaceId);
await input.evidence("pi-provider-death-fault.json", fault);
await pollUntil({ label: "production expiry after Pi provider loss", deadlineAt: input.deadlineAt, load: input.load,
accept: s => s.issue.status === "blocked" && s.runs[0]?.status === "failed" && s.interactions.some(c => c.id === pending.interactionId && c.status === "expired"),
reject: s => s.runs.length !== 1 || s.runs.some(r => ["succeeded", "cancelled", "timed_out"].includes(r.status)) ? "Unexpected run after provider fault" : undefined });
const retired = await input.fixture.finish();
check("death-owned-retirement-no-effect", hasPiRemoteRetirement(retired) && retired.targets[PI_DEATH_MARKER]?.absent === true
&& retired.targets[PI_DEATH_MARKER]?.complete === true && retired.targets[PI_DEATH_MARKER]?.mutationCount === 0,
"Exact owned process tree retired while the unanswered continuation marker remained absent");
await input.evidence("pi-provider-death-retirement.json", retired);
await input.page.reload();
const final = await snapshot("pi-provider-death-expired.json"), result = gradePiProviderDeath(final.state, final.events, pending, question);
// Failed native terminals are permanent on the first attempt; the
// controller's nativeSessionRecoveryProjection projects Blocked while this
// run still owns the task (native-session-executor.ts).
await expect(input.page.getByTestId("issue-detail-header").getByRole("button", { name: "Change status (current: Blocked", exact: false })).toBeVisible();
const oldCard = input.page.locator(`#interaction-${pending.interactionId}`);
await expect(oldCard).toHaveCount(1);
await expect(oldCard.locator('textarea,[contenteditable="true"],button[type="submit"]')).toHaveCount(0);
await input.capture("pi-provider-death-expired", "Expired original input and any separate durable fallback", "pi-provider-death-expired.png");
const answer = `STALE-${randomBytes(16).toString("hex")}`;
const original = await input.api.request.post(`/api/issues/${pending.issueId}/interactions/${pending.interactionId}/respond`,
{ data: { answers: [{ questionId: pending.questionId, optionIds: [], otherText: answer }] } });
const runtime = await input.api.request.post(`/api/heartbeat-runs/${pending.runId}/runtime-requests/${encodeURIComponent(pending.requestId)}/resolve`,
{ data: { resolution: { action: "submit", response: { schema: "paperclip.question_response.v1", answers: { [pending.questionId]: { text: answer } } } } } });
check("death-stale-original-responses-rejected", original.status() === 409 && runtime.status() === 409, "Both original durable-card and native-runtime response APIs reject the stale answer");
const after = await snapshot("pi-provider-death-after-stale.json");
check("death-expiry-not-restoration", hash(gradePiProviderDeath(after.state, after.events, pending, question)) === hash(result)
&& hash(after.state.interactions) === hash(final.state.interactions), "Stale answers neither resolve/replay the original request nor start a new run; any fallback remains a separate unanswered interaction");
await input.evidence("pi-provider-death-result.json", { ...result, staleCardStatus: original.status(), staleRuntimeStatus: runtime.status(), paidProviderDeath: true });
return checks;
} finally { await input.evidence("pi-provider-death-checks.json", checks); }
}
+8 -4
View File
@@ -28,7 +28,7 @@ function interactionBinding(card: Row) {
continuationPolicy: card.continuationPolicy, resolverPolicy: card.resolverPolicy, payload: card.payload };
}
function inspect(state: State, events: readonly Row[], companyId: string) {
function inspect(state: State, events: readonly Row[], companyId: string, header = "Pi native restart") {
requireProof(id(companyId) && id(state.issue.id) && state.issue.companyId === companyId && state.runs.length === 1 && state.interactions.length === 1,
"one company-scoped task, run and question required");
const run = state.runs[0]!, card = state.interactions[0]!;
@@ -63,7 +63,7 @@ function inspect(state: State, events: readonly Row[], companyId: string) {
requireProof(request.schema === "paperclip.runtime_request.v2" && request.requestKind === "runtime" && request.type === "input" && request.status === "pending"
&& id(request.requestId) && id(request.itemId) && id(request.turnId) && request.turnId === opening.event.turnId && request.itemId === opening.event.itemId
&& hasAcpxNativeOrigin(request.origin, "pi", "elicitation/create") && questionSet.schema === "paperclip.question_set.v1"
&& id(question.id) && question.answerMode === "text" && question.header === "Pi native restart"
&& id(question.id) && question.answerMode === "text" && question.header === header
&& card.payload?.runtimeRequestId === request.requestId && digest(card.payload.questionSet) === digest(questionSet)
&& card.idempotencyKey === `paperclip-runner-question:${run.id}:${request.requestId}`, "canonical Pi input request identity missing");
requireProof(rows.filter(x => x.event.turnId != null || x.event.eventType.startsWith("turn.") || x.event.eventType.startsWith("runtime_request.")).every(({ event }) => event.turnId === request.turnId
@@ -77,14 +77,18 @@ function inspect(state: State, events: readonly Row[], companyId: string) {
}
/** Grade public state and PRP receipts; assistant prose is never restart proof. */
export function observePiRestartPending(state: State, events: readonly Row[], companyId: string): PiRestartPending {
const { run, card, rows, binding } = inspect(state, events, companyId);
export function observePiPendingNativeInput(state: State, events: readonly Row[], companyId: string, header: string): PiRestartPending {
const { run, card, rows, binding } = inspect(state, events, companyId, header);
requireProof(state.issue.status === "in_progress" && run.status === "running" && card.status === "pending" && card.result == null
&& run.resultJson?.nativeCancellation == null && run.resultJson?.startupCancellation == null
&& !rows.some(x => closures.has(x.event.eventType) || terminals.has(x.event.eventType) || x.event.sourceKind === "control_plane"), "question is no longer unanswered in a live run");
return binding;
}
export function observePiRestartPending(state: State, events: readonly Row[], companyId: string): PiRestartPending {
return observePiPendingNativeInput(state, events, companyId, "Pi native restart");
}
export function gradePiRestartCompletion(state: State, events: readonly Row[], pending: PiRestartPending, answer: string, proof: unknown): boolean {
const { run, card, rows, binding } = inspect(state, events, pending.companyId);
requireProof(digest(binding) === digest(pending), "original request/run/source identity changed");
@@ -7,8 +7,9 @@ import { tmpdir } from "node:os";
import { join } from "node:path";
import { Script } from "node:vm";
import { describe, expect, it, vi } from "vitest";
import { REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS, bindRemoteNativeFixture, createRemoteTargetWatch, isRemoteRunRoot, parseRemoteProcStat, remoteNativeFixtureDiagnostics, type RemoteNativeFixtureOptions, type RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { REMOTE_FIXTURE_MIN_SETUP_BUDGET_MS, bindRemoteNativeFixture, validatePiProviderDeathReceipt, createRemoteTargetWatch, isRemoteRunRoot, parseRemoteProcStat, remoteNativeFixtureDiagnostics, type RemoteNativeFixtureOptions, type RemoteNativeSnapshot } from "./remote-native-fixtures.js";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js";
import { createRemoteNativeBootstrap } from "./remote-native-bootstrap.js";
const hash = (s: string) => `sha256:${createHash("sha256").update(s).digest("hex")}`;
@@ -418,8 +419,9 @@ describe("actual generated observer state machine", () => {
close: vi.fn(() => listeners.clear()),
};
const net = { createServer(fn: (socket: any) => void) { handlers.push(fn); return server; } };
const faultSpawn = vi.fn(() => ({ status: 0, stdout: JSON.stringify(piFaultReceipt()) }));
const context = {
require(name: string) { if (name === "node:fs") return fs; if (name === "node:net") return net; if (name === "node:child_process") return { spawn: vi.fn(() => { const child = Object.assign(new EventEmitter(), { pid: 88, exitCode: null, signalCode: null, kill: vi.fn() }); children.push(child); return child; }) }; if (name === "node:path") return { join: (...paths: string[]) => paths.join("/"), dirname: (path: string) => path.slice(0, path.lastIndexOf("/")), basename: (path: string) => path.slice(path.lastIndexOf("/") + 1) }; if (name === "node:crypto") return { createHash }; throw new Error("unexpected module"); },
require(name: string) { if (name === "node:fs") return fs; if (name === "node:net") return net; if (name === "node:child_process") return { spawnSync: faultSpawn, spawn: vi.fn(() => { const child = Object.assign(new EventEmitter(), { pid: 88, exitCode: null, signalCode: null, kill: vi.fn() }); children.push(child); return child; }) }; if (name === "node:path") return { join: (...paths: string[]) => paths.join("/"), dirname: (path: string) => path.slice(0, path.lastIndexOf("/")), basename: (path: string) => path.slice(path.lastIndexOf("/") + 1) }; if (name === "node:crypto") return { createHash }; throw new Error("unexpected module"); },
process: { argv: ["node", `${config.root}/observer.cjs`, Buffer.from(JSON.stringify(config)).toString("base64")], execPath: "/node", hrtime: { bigint: () => 12345n }, exit: vi.fn() },
Buffer, __filename: `${config.root}/observer.cjs`,
setInterval(fn: () => void) { intervals.push(fn); return 1; }, clearInterval: vi.fn(),
@@ -431,8 +433,28 @@ describe("actual generated observer state machine", () => {
const replies: any[] = [], socket = Object.assign(new EventEmitter(), { end: (value: string) => replies.push(JSON.parse(value)), destroy: vi.fn() });
handlers[0]!(socket); socket.emit("data", Buffer.from(JSON.stringify({ op, nonce: config.nonce, ...args }) + "\n")); return replies;
}
return { request, proc, files, watches, fs, intervals, timers, config, handlers, children, symbolicLinks, context, server, directories, listeners, install: h.calls.find(c => c.request.op === "install")!, replaceRuntimeRoot() { runtimeInode = 999n; } };
return { request, proc, files, watches, fs, intervals, timers, config, handlers, children, faultSpawn, symbolicLinks, context, server, directories, listeners, install: h.calls.find(c => c.request.op === "install")!, replaceRuntimeRoot() { runtimeInode = 999n; } };
}
it("executes the actual observer fault phase once with a closed environment and exact identity", async () => {
for (const published of [false, true]) {
const rejected = await observerHarness();
if (published) rejected.request("publish", { path: "action.txt", text: "ask native input" });
expect(rejected.request("pi-provider-death", { runtimeEnvironmentLeaseId: published ? "foreign" : "workspace-id" })[0].ok).toBe(false);
expect(rejected.faultSpawn).not.toHaveBeenCalled();
}
const o = await observerHarness();
expect(o.request("snapshot")[0].ok).toBe(true);
expect(o.request("publish", { path: "action.txt", text: "ask native input" })[0].ok).toBe(true);
expect(o.request("pi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id" })[0]).toEqual({ ok: true, result: piFaultReceipt() });
expect(o.faultSpawn).toHaveBeenCalledTimes(1);
const [program, args, options] = o.faultSpawn.mock.calls[0] as unknown as [string, string[], Record<string, unknown>];
expect(program).toBe("/usr/bin/python3"); expect(args.slice(0, 2)).toEqual(["-I", "-c"]);
expect(args[2]).toContain("signal.pidfd_send_signal");
expect(JSON.parse(Buffer.from(args[3]!, "base64").toString())).toEqual({ root, binding, runtimeEnvironmentLeaseId: "workspace-id", runnerdSha256: hash("runnerd"), closureSha256: PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"] });
expect(options).toMatchObject({ env: { PATH: "/usr/bin:/bin" }, timeout: 8000, maxBuffer: 32768 });
expect(o.request("pi-provider-death", { runtimeEnvironmentLeaseId: "workspace-id" })[0].ok).toBe(false);
expect(o.faultSpawn).toHaveBeenCalledTimes(1);
});
async function generatedRpc(o: Awaited<ReturnType<typeof observerHarness>>, request: Record<string, unknown>, mutateSource = (source: string) => source) {
const quoted = o.install.command.match(/ -e (.+) '[A-Za-z0-9+/=]+'$/su)![1]!;
const source = quoted.slice(1, -1).replaceAll("'\\''", "'");
@@ -692,3 +714,41 @@ describe("actual generated observer state machine", () => {
expect(other.request("snapshot")[0].ok).toBe(false);
});
});
function piFaultReceipt() {
const parent = { pid: 22, ppid: 21, startTicks: "101", bootId }, target = { pid: 23, ppid: 22, startTicks: "102", bootId };
return { schema: "paperclip.e2e.pi-provider-death.v1", binding, runtimeEnvironmentLeaseId: "workspace-id", target, ancestry: [target, parent, root],
nodeSha256: hash("node"), entrypointSha256: hash("entry"), closureSha256: PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"],
entrypointAttribution: "pinned_wrapper_parent", originalChildArgvAvailable: false, observedChildTitle: "pi", signalled: true, signal: "SIGKILL", targetKind: "pi_native_child", workerSignalled: false };
}
describe("exact Pi-child fault capability", () => {
it("validates ancestry and honest parent-attested entrypoint identity", () => {
const receipt = piFaultReceipt();
expect(validatePiProviderDeathReceipt(receipt, binding, root, "workspace-id")).toEqual(receipt);
for (const patch of [{ workerSignalled: true }, { target: root }, { originalChildArgvAvailable: true }, { entrypointAttribution: "process_title" },
{ binding: { ...binding, runId: "foreign" } }, { runtimeEnvironmentLeaseId: "foreign" }, { closureSha256: "0".repeat(64) },
{ ancestry: [receipt.target, { ...receipt.ancestry[1], startTicks: "" }, root] }, { ancestry: [receipt.target, root] },
{ ancestry: [receipt.target, { ...receipt.ancestry[1], ppid: 999 }, root] }, { ancestry: [receipt.target, receipt.ancestry[1], { ...root, startTicks: "999" }] }]) {
expect(() => validatePiProviderDeathReceipt({ ...receipt, ...patch }, binding, root, "workspace-id")).toThrow();
}
});
it("requires published action and consumes the one-shot capability even on RPC failure", async () => {
const h = harness(), fixture = await bindRemoteNativeFixture(h.options);
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
await fixture.publishAction("action.txt", "Ask the native question");
h.override(request => request.op === "pi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: false, error: "pi_provider_identity_or_signal_failed" }) } : undefined);
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
expect(h.calls.filter(call => call.request.op === "pi-provider-death")).toHaveLength(1);
await fixture.close();
});
it("returns only the exact reviewed fault receipt and never signals the worker", async () => {
const h = harness(), fixture = await bindRemoteNativeFixture(h.options);
await fixture.publishAction("action.txt", "Ask the native question");
h.override(request => request.op === "pi-provider-death" ? { exitCode: 0, result: JSON.stringify({ ok: true, result: piFaultReceipt() }) } : undefined);
expect(await fixture.terminatePiProvider!("workspace-id")).toEqual(piFaultReceipt());
await expect(fixture.terminatePiProvider!("workspace-id")).rejects.toThrow();
expect(h.calls.filter(call => call.request.op === "pi-provider-death")).toHaveLength(1);
await fixture.close();
});
});
+45 -6
View File
@@ -1,7 +1,10 @@
import { createHash, randomBytes } from "node:crypto";
import { watch, lstatSync } from "node:fs";
import { watch, lstatSync, readFileSync } from "node:fs";
import { posix } from "node:path";
import { PI_DISTRIBUTION_CLOSURE_SHA256 } from "../../packages/paperclip-runner/src/drivers/acpx/pi-closure-pins.js";
const PI_FAULT_SOURCE = readFileSync(new URL("./pi-provider-fault.py", import.meta.url), "utf8");
export const REMOTE_FIXTURE_DAYTONA_SDK_VERSION = "0.203.0";
const NODE = "/opt/paperclip-runner/provider-pack/node_modules/node/bin/node";
const MAX_OUTPUT = 256 * 1024;
@@ -30,8 +33,8 @@ function relative(value: string): string {
}
// Only closed diagnostic enums cross the remote boundary; SDK errors and output
// are never retained. These diagnostics explain failed evidence, not qualification.
const RPC_PHASES = ["runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "read"] as const;
const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline"] as const;
const RPC_PHASES = ["runtime-ready", "install", "wait", "close", "arm", "publish", "snapshot", "attached", "read", "pi-provider-death"] as const;
const RPC_CODES = ["node_identity", "sentinel_type", "sentinel", "cwd", "runtime_root_identity", "runtime_binary_identity", "proc_bound", "ambiguous_run_root", "runtime_not_ready", "runtime_identity_changed", "invalid_proc_identity", "invalid_proc_fields", "socket_error", "socket_timeout", "output_bound", "rpc_deadline", "remote_unknown", "transport_failure", "invalid_response", "readiness_deadline", "pi_provider_identity_or_signal_failed"] as const;
type RpcPhase = typeof RPC_PHASES[number];
type RpcCode = typeof RPC_CODES[number];
interface RpcDiagnostic { phase: RpcPhase; code: RpcCode }
@@ -121,13 +124,13 @@ export function createRemoteTargetWatch(directory: string, name: string, io = {
const OBSERVER = String.raw`
const fs=require('node:fs'),path=require('node:path'),crypto=require('node:crypto'),net=require('node:net'),cp=require('node:child_process');
const config=JSON.parse(Buffer.from(process.argv[2],'base64').toString());
const parseStat=PARSE_STAT;const runRoot=RUN_ROOT;const watchTarget=WATCH_TARGET;
const parseStat=PARSE_STAT;const runRoot=RUN_ROOT;const watchTarget=WATCH_TARGET;const PI_FAULT_SOURCE=PI_FAULT_PROGRAM,PI_CLOSURE_PIN=PI_CLOSURE_DIGEST;
const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex');
const cwdStat=fs.lstatSync(config.binding.remoteCwd,{bigint:true}),rootStat=fs.lstatSync(config.root,{bigint:true}),scriptStat=fs.lstatSync(__filename,{bigint:true}),scriptHash=hash(fs.readFileSync(__filename));
const runtimeRoot=path.join(config.binding.remoteCwd,config.runtimeRelative),runtimeStat=fs.lstatSync(runtimeRoot,{bigint:true});if(!runtimeStat.isDirectory()||runtimeStat.isSymbolicLink()||fs.realpathSync(runtimeRoot)!==runtimeRoot)throw Error('runtime_root_identity');let observedPrpEnvironmentLeaseId=null;
const boot=fs.readFileSync('/proc/sys/kernel/random/boot_id','utf8').trim();
const targets=new Map();let complete=true,sealed=false,root=null,attached=null,child=null,client=null,childTimer=null;
const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false;
const journal=new Map(),sockets=new Set(),waiters=new Set(),armWaiters=new Set();let observedRootCount=0,publishedHash=null,finalReceipt=null,retiring=false,piFaultAttempted=false;
function identity(pid){return parseStat(pid,fs.readFileSync('/proc/'+pid+'/stat','utf8'),boot)}
function table(){const ids=fs.readdirSync('/proc').filter(x=>/^\d+$/.test(x)&&Number(x)>1);if(ids.length>4096)throw Error('process_bound');return ids.flatMap(x=>{try{return [identity(Number(x))]}catch(e){if(e.code==='ENOENT'||e.code==='ESRCH')return [];throw e}})}
function sample(){
@@ -166,6 +169,13 @@ const server=net.createServer(socket=>{sockets.add(socket);socket.on('close',()=
else if(r.op==='arm'){if(waiters.size)result={armed:true,sealed};else{armWaiters.add(socket);socket.on('close',()=>armWaiters.delete(socket));return}}
else if(r.op==='publish'){if(sealed||publishedHash||r.path!==config.actionFile||typeof r.text!=='string'||Buffer.byteLength(r.text)>16384)throw Error('publish_bound');const p=path.join(config.binding.remoteCwd,config.actionFile);if(fs.realpathSync(path.dirname(p))!==path.dirname(p))throw Error('publish_parent');publishedHash=hash(r.text);try{fs.writeFileSync(p,r.text,{flag:'wx',mode:0o600})}catch(e){publishedHash=null;throw e}result={path:r.path,sha256:publishedHash,published:true};}
else if(r.op==='read'){const p=r.path==='@cross-root'&&config.crossRoot?path.join(config.root,'cross-root-target'):path.join(config.binding.remoteCwd,r.path);if(r.path!=='@cross-root'&&!config.targets.includes(r.path))throw Error('unregistered_read');const status=file(p);if(status.absent)throw Error('file_missing');result={...status,base64:readSafe(p).toString('base64')};}
else if(r.op==='pi-provider-death'){
if(piFaultAttempted||sealed||!publishedHash||!root||!complete||typeof r.runtimeEnvironmentLeaseId!=='string'||r.runtimeEnvironmentLeaseId!==observedPrpEnvironmentLeaseId)throw Error('pi_provider_identity_or_signal_failed');
piFaultAttempted=true;const before=sample();if(!before.captured||!before.live.includes(root.pid))throw Error('pi_provider_identity_or_signal_failed');
const request={root:{pid:root.pid,ppid:root.ppid,startTicks:root.startTicks,bootId:root.bootId},binding:config.binding,runtimeEnvironmentLeaseId:r.runtimeEnvironmentLeaseId,runnerdSha256:config.runnerdSha256,closureSha256:PI_CLOSURE_PIN};
const childResult=cp.spawnSync('/usr/bin/python3',['-I','-c',PI_FAULT_SOURCE,Buffer.from(JSON.stringify(request)).toString('base64')],{env:{PATH:'/usr/bin:/bin'},timeout:8000,maxBuffer:32768,encoding:'utf8'});
if(childResult.status!==0||childResult.error)throw Error('pi_provider_identity_or_signal_failed');result=JSON.parse(childResult.stdout);
}
else if(r.op==='attached'){if(attached||sealed)throw Error('attached_already_configured');if(!config.targets.includes(r.marker)||!Number.isInteger(r.delayMs)||r.delayMs<100||r.delayMs>8000)throw Error('attached_bounds');
attached={connections:0,failure:null,commandExit:null,markerWrittenAtMs:null,markerWrittenMonotonicNs:null,clientExitedAtMs:null,clientExitedMonotonicNs:null};
const clientScript=path.join(config.root,'client.cjs'),clientSocket=path.join(config.root,'attached.sock');
@@ -186,7 +196,7 @@ server.listen(path.join(config.root,'control.sock'));setTimeout(()=>{complete=fa
`;
const ATTACHED_CLIENT = String.raw`const net=require('node:net');const s=net.connect(process.argv[2]);let b='';s.setTimeout(15000,()=>process.exit(3));s.on('error',()=>process.exit(4));s.on('connect',()=>s.write(JSON.stringify({nonce:process.argv[3],pid:process.pid})+'\n'));s.on('data',x=>{b+=x;if(b.length>1024)process.exit(6);if(b.includes('\n')){const r=JSON.parse(b);s.end();process.exit(r.code===0?0:5)}});`;
function observerSource() {
return OBSERVER.replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString())
return OBSERVER.replace("PI_FAULT_PROGRAM", () => JSON.stringify(PI_FAULT_SOURCE)).replace("PI_CLOSURE_DIGEST", () => JSON.stringify(PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"])).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString())
.replace("WATCH_TARGET", () => createRemoteTargetWatch.toString()).replace("ATTACHED_CLIENT", () => JSON.stringify(ATTACHED_CLIENT));
}
const RPC = String.raw`const fs=require('node:fs'),net=require('node:net'),cp=require('node:child_process'),crypto=require('node:crypto');const r=JSON.parse(Buffer.from(process.argv[1],'base64').toString());const hash=x=>'sha256:'+crypto.createHash('sha256').update(x).digest('hex');
@@ -204,6 +214,26 @@ function rpcSource() {
return RPC.replace("RPC_CODES", () => JSON.stringify(RPC_CODES)).replace("RPC_PHASES", () => JSON.stringify(RPC_PHASES)).replace("PARSE_STAT", () => parseRemoteProcStat.toString()).replace("RUN_ROOT", () => isRemoteRunRoot.toString());
}
export interface PiProviderDeathReceipt {
schema: "paperclip.e2e.pi-provider-death.v1"; binding: RemoteNativeBinding; runtimeEnvironmentLeaseId: string;
target: RemoteProcessIdentity; ancestry: RemoteProcessIdentity[]; nodeSha256: string; entrypointSha256: string; closureSha256: string;
entrypointAttribution: "pinned_wrapper_parent"; originalChildArgvAvailable: false; observedChildTitle: "pi";
signalled: true; signal: "SIGKILL"; targetKind: "pi_native_child"; workerSignalled: false;
}
export function validatePiProviderDeathReceipt(value: unknown, binding: RemoteNativeBinding, root: RemoteProcessIdentity, lease: string): PiProviderDeathReceipt {
const r = record(value), target = record(r.target), ancestry = Array.isArray(r.ancestry) ? r.ancestry.map(record) : [];
fail(r.schema === "paperclip.e2e.pi-provider-death.v1" && JSON.stringify(r.binding) === JSON.stringify(binding)
&& r.entrypointAttribution === "pinned_wrapper_parent" && r.originalChildArgvAvailable === false && r.observedChildTitle === "pi"
&& r.runtimeEnvironmentLeaseId === lease && r.signalled === true && r.signal === "SIGKILL" && r.targetKind === "pi_native_child" && r.workerSignalled === false
&& sha(r.nodeSha256) && sha(r.entrypointSha256) && r.closureSha256 === PI_DISTRIBUTION_CLOSURE_SHA256["linux-x64"], "pi_fault_receipt");
fail(ancestry.length >= 3 && ancestry.length <= 64 && new Set(ancestry.map(p => p.pid)).size === ancestry.length
&& ancestry.every(p => Number.isSafeInteger(p.pid) && Number(p.pid) > 1 && Number.isSafeInteger(p.ppid) && Number(p.ppid) > 0
&& typeof p.startTicks === "string" && /^\d+$/u.test(p.startTicks) && p.bootId === root.bootId)
&& ["pid", "ppid", "startTicks", "bootId"].every(k => ancestry[0]?.[k] === target[k] && ancestry.at(-1)?.[k] === record(root)[k])
&& ancestry.slice(0, -1).every((p, i) => p.ppid === ancestry[i + 1]?.pid) && target.pid !== root.pid, "pi_fault_ancestry");
return value as PiProviderDeathReceipt;
}
export interface RemoteNativeFixture {
readonly binding: RemoteNativeBinding;
readonly remoteCwd: string;
@@ -212,6 +242,8 @@ export interface RemoteNativeFixture {
/** Watchers and exact run-root identity are already armed at return from bind. */
readonly baseline: RemoteNativeSnapshot;
snapshot(label: string): Promise<RemoteNativeSnapshot>;
/** One-shot fault against the exact admitted Pi child; never a worker kill. */
terminatePiProvider?(runtimeEnvironmentLeaseId: string): Promise<PiProviderDeathReceipt>;
readFile(path: string): Promise<Buffer>;
publishAction(path: string, text: string): Promise<void>;
setupAttachedCommand(input: { marker: string; markerText: string; delayMs: number }): Promise<{ command: string; commandSha256: string }>;
@@ -424,6 +456,7 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
stopReceipt();
throw error;
}
let piFaultAttempted = false;
let closed = false, published = false, finished: RemoteNativeSnapshot | undefined;
const retainedFiles = new Map<string, Buffer>();
return {
@@ -433,6 +466,12 @@ export async function bindRemoteNativeFixture(options: RemoteNativeFixtureOption
fail(!closed && !finished, "fixture_closed");
return readSnapshot(await rpc({ op: "snapshot" }), binding!, names, actionFile, options.runnerdSha256);
},
async terminatePiProvider(runtimeEnvironmentLeaseId) {
fail(!closed && !finished && published && !piFaultAttempted && /^[A-Za-z0-9._:-]{1,240}$/u.test(runtimeEnvironmentLeaseId), "pi_fault_admission");
piFaultAttempted = true; // Uncertain delivery never retries the signal.
const receipt = await rpc({ op: "pi-provider-death", runtimeEnvironmentLeaseId });
return validatePiProviderDeathReceipt(receipt, binding!, baseline.processes.root!, runtimeEnvironmentLeaseId);
},
async readFile(path) {
fail(!closed && names.includes(path), "unregistered_read");
if (finished) { const bytes = retainedFiles.get(path); fail(bytes, "terminal_file_missing"); return Buffer.from(bytes!); }