From 120eb7f01eea958d939e2e2b3e39b205ea490b36 Mon Sep 17 00:00:00 2001 From: Dotta Date: Mon, 5 Oct 2026 11:31:59 -0500 Subject: [PATCH] fix(evals): make Pi file task shell limit explicit State the existing single-Bash oracle in the task prompt and use the supplied post-edit registration metadata. Preserve the grader, original failed attempts and free fault-calibration limits. Co-Authored-By: Paperclip --- .../2026-10-02-pi-production-readiness.md | 44 ++++++++++++++++++- tests/runner-e2e/FIXTURES.md | 14 +++--- tests/runner-e2e/catalog.ts | 2 +- tests/runner-e2e/pi-file-evidence.test.ts | 17 +++++++ tests/runner-e2e/pi-file-evidence.ts | 4 +- 5 files changed, 72 insertions(+), 9 deletions(-) diff --git a/doc/plans/2026-10-02-pi-production-readiness.md b/doc/plans/2026-10-02-pi-production-readiness.md index 770f5c4be26..01043a18635 100644 --- a/doc/plans/2026-10-02-pi-production-readiness.md +++ b/doc/plans/2026-10-02-pi-production-readiness.md @@ -27,7 +27,7 @@ qualification. The existing draft stack must be reviewed in dependency order. | Runner protocol | Pi roster passes through the native packaged runner and authenticated mock control plane; lifecycle/denial/control cases remain distinct from Product tests | All seven cases are attempted at `0bd040093`: five pass; context-before-action retains its 120-second terminal timeout, and finish-task retains use of the advisory report without the required task mutation. One context/document workflow retry passes only after verified protection against the recorded host sleep; its original failure stays unchanged. The seven `b148b73ea` passes remain historical. | | Installed distribution | Public CLI/server tars on ARM Mac, Intel Mac and Linux; normal Pi setup; exact Linux companion and immutable Daytona image imported without binary override | Exact-source public CLI/server installation and normal Pi setup pass on all three platforms. Public immutable image `ghcr.io/paperclipai/paperclip-daytona-runner@sha256:342f1fd5cb8cabfa2242f38f4f686608b28b6536aa879c54b0cb0da6fba9ef47` imports into native Linux Daytona and passes closed admission. Temporary sandbox cleanup passes. Image jobs 37102904889, 37106313185 and 37110683910 are terminal/cancelled. | | Governance and spend | Company isolation, human-only permission, duplicate/stale answers, Stop and budget hard stop; pricing estimates never become claimed bills | 58 focused governance/cost/session invariants pass at `f5f57e380`. Live pending-permission Stop passes at `03dd6ef93`. The qualification key cap is not proof of Paperclip budget enforcement. | -| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | Last verified PR head `4f2300d5b` has 53 successful CI checks, two skips, Greptile 5/5 and no unresolved root threads, including full Linux build and typecheck. This provider-fault fixture follow-up requires its own completed latest-head checks and review. All 116 targeted native Linux fixture tests and 13 browser-case collections pass at `04f639eeb`; the new descriptor correction separately passes all 17 free native Linux fault tests. Full typecheck, complete test and build evidence remains recorded below; original failed commands remain failed. Shipping inputs stay equivalent to `0bd040093`. Four prerequisite findings have downstream corrections; upstream review threads and standalone prerequisite CI remain open. Production stays held for nine failed gates and prerequisite disposition. No merge or release. | +| Integration and rollout | Review each prerequisite; final-head typecheck, tests, build and CI pass; no unresolved review; exact artifacts; rollback recorded | Last verified PR head `b224e4338` has 53 successful CI checks, two skips, Greptile 5/5 and no unresolved root threads, including full Linux build and typecheck. This file-prompt fixture follow-up requires its own completed latest-head checks and review. All 116 targeted native Linux fixture tests and 13 browser-case collections pass at `04f639eeb`; the new descriptor correction separately passes all 17 free native Linux fault tests. Full typecheck, complete test and build evidence remains recorded below; original failed commands remain failed. Shipping inputs stay equivalent to `0bd040093`. Four prerequisite findings have downstream corrections; upstream review threads and standalone prerequisite CI remain open. Production stays held for nine failed gates and prerequisite disposition. No merge or release. | ## Bounded execution @@ -167,6 +167,48 @@ The retained production matrix stays 24 passes and nine failures. All six earlier resumed attempts and their phases are terminal. Their controller and child sandboxes are deleted with canonical evidence retained. +## Free fault calibration and file-prompt correction — 2026-10-05 + +Head `b224e4338` completes 53 successful checks and two skips, including the +full Linux build and typecheck, with Greptile 5/5 and no unresolved root review +threads. The separately guarded descriptor-corrected provider-death attempt is +terminal and failed: its unchanged native question appears, but the observer +returns incomplete evidence without a fault-signal receipt. Cleanup separately +fails on a closed observer socket. The third failed attempt, both prior failures +and their source fingerprints are retained. Its owned child and controller are +deleted after collection. The matrix remains **24 passes and nine failures**; +none are running or unattempted. Production remains held. + +Four subsequent network-blocked Linux calibrations use no provider credentials +or model prompts. Both actual Pi-wrapper launches pass inspection: direct +held-FD launch and production lifetime fences. The real Rust Runner then admits +the frozen native model/low profile and passes exact child inspection. Finally, +the unchanged observer successfully performs its one-shot pidfd-bound Pi-child +fault and seals complete retirement. Every owned calibration sandbox is deleted. +These calibrations use a synthetic controller/lease caller and an idle provider; +they do not prove the paid pending-input journey, regrade that failure, establish +its missing rejection stage, or authorize an unchanged paid retry. + +Both original file-edit attempts complete the exact edit, validation and public +artifact, then fail the required single-Bash-execution oracle after an additional +metadata command. The old prompt specifies an exact validation command but does +not explicitly forbid other Bash calls. The correction states exactly one Bash +call for the whole task and directs registration to use the supplied byte size +and hash after exact-byte validation. The grader remains unchanged. Negative +calibration rejects extra metadata executions both before and after validation, +even when file and download bytes are correct. Extended definitions advance +from 2 to 3; original paid attempts retain their definitions and failed grades. +Free fixture verification, exact-source Linux preparation and fresh spend guards +must complete before separately bounded corrected file-edit attempts. No paid +attempt occurs during this correction, and its live outcome is unproven. +Both focused file/catalog files pass all 103 local tests. The broader name +selection also includes an unchanged report-catalog test that fails on a missing +generated result; its failed command remains retained. Local Product typecheck +still fails on six stale linked Runner/adapter declarations. Fresh latest-head +CI must verify typecheck, tests and build before a PR-ready handoff. The new +extended-suite fingerprint is +`121f4cf75267bcdb003e5ed59e165755b2ac0c3d31c82dc217a62c8abe3d1b4d`. + ## Qualification snapshot — 2026-10-03 The October 3 ledger has 21 passes, ten failed cells, two unattempted cells and diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index b7110865c17..6966595704e 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -337,11 +337,15 @@ identities still fail. ### Pi file editing and registered artifacts Pi's `extended-harnesses/file-edit-validate` seeds exact bytes before startup and -requires one native edit lifecycle followed by a successful native bash execution -with the exact nonce-bound byte-validation command as its projected title and a -validation marker. A marker-only echo cannot pass. Independent final bytes must -match the fixture. The -real `register_deliverable` receipt, attachment metadata, publication activity, +requires one native edit lifecycle followed by exactly one successful native +bash execution with the exact nonce-bound byte-validation command as its +projected title and a validation marker. A marker-only echo cannot pass. Final bytes must +match the fixture. The Pi task prompt explicitly forbids additional shell calls, +including metadata commands, and supplies the expected post-edit byte size and +hash for registration. Extra Bash calls fail the unchanged oracle even when the +edited file and downloadable artifact are correct. Extended definitions 3 retain +older attempts under their original definitions and grades. The real +`register_deliverable` receipt, attachment metadata, publication activity, visible task attachment and authenticated public download must all agree on the file's bytes, hash, company, issue, agent and originating run. A file on disk or a model completion claim cannot substitute for publication. Daytona additionally diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index e9f51913a26..077db787644 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1162,7 +1162,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ groups: ["native"], profiles: extendedHarnessProfiles, environments: runnerEnvironments, tasks: [...openRouterBreadthTasks, localIntegrityTasks[1]!, extendedHarnessFileTask], expectedMatrixSize: 30, - definitionMetadata: { version: 2, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28", piFileEvidence: "seed-edit-execute-public-download-v1" }, + definitionMetadata: { version: 3, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28", piFileEvidence: "seed-edit-single-execute-public-download-v2" }, }, { id: "instruction-persistence", label: "Instruction Persistence", diff --git a/tests/runner-e2e/pi-file-evidence.test.ts b/tests/runner-e2e/pi-file-evidence.test.ts index 64981c7ab67..3049eeda59f 100644 --- a/tests/runner-e2e/pi-file-evidence.test.ts +++ b/tests/runner-e2e/pi-file-evidence.test.ts @@ -85,6 +85,23 @@ describe("Pi edit, validation and public artifact oracle", () => { expect(fixture().events.slice(3).every(e => e.payload.prpEvent.payload.name === "bash")).toBe(true); expect(result.verification.nativeFileAttribution).toBe("workspace_relative_display_target"); }); + it.each(["before", "after"])("rejects an extra metadata bash call %s validation despite correct file and download bytes", placement => { + const f = fixture(); + const metadata = structuredClone(f.events.slice(3)); + for (const row of metadata) row.payload.prpEvent.payload.executionId = "metadata"; + metadata[1]!.payload.prpEvent.payload.progress = "wc -c extended-fixture.txt (in_progress): wc -c extended-fixture.txt"; + metadata[2]!.payload.prpEvent.payload.output = JSON.stringify({ + content: [{ type: "text", text: "17 extended-fixture.txt\n" }], + structuredContent: { output: "17 extended-fixture.txt\n", truncated: false, exit_code: 0 }, + }); + if (placement === "before") f.events.splice(3, 0, ...metadata); + else f.events.push(...metadata); + for (const [index, row] of f.events.entries()) { + row.seq = row.sourceSeq = row.payload.prpEvent.sourceSeq = index + 1; + row.sourceEventId = row.payload.prpEvent.sourceEventId = `runner:run:${index + 1}`; + } + expect(() => gradePiFileEvidence(f)).toThrow("one observed provider validation execution"); + }); const mutations: Array<[string, (f: ReturnType) => void]> = [ ["absent seed", f => { f.seed = {} as any; }], ["wrong final bytes", f => { f.workspaceBytes = Buffer.from("wrong\n"); }], diff --git a/tests/runner-e2e/pi-file-evidence.ts b/tests/runner-e2e/pi-file-evidence.ts index 7837666633f..08b4f1b3422 100644 --- a/tests/runner-e2e/pi-file-evidence.ts +++ b/tests/runner-e2e/pi-file-evidence.ts @@ -21,8 +21,8 @@ export function piFilePrompt(nonce: string) { const c = piFileContract(nonce); return [ `The fixture has already created ${c.filename} in your execution workspace with exactly ${JSON.stringify(c.before)}. Read it, then use your native edit tool once to replace those bytes with exactly ${JSON.stringify(c.after)}. Do not substitute write, bash, or an API for the edit.`, - `Run this exact native bash command, without an added shell prefix or suffix:\n${c.validationCommand}\nIt independently reads the file, fails on incorrect bytes, and prints the validation marker only after success.`, - `Deliver the edited text file as a downloadable attachment using the assigned register_deliverable semantic tool, with filename and workspace-relative contentRef ${c.filename}, contentType text/plain, byteSize ${c.byteSize}, sha256 ${c.sha256}, title Verified text, and a fresh idempotencyKey. Wait for the real registration receipt. Do not upload a different file or claim publication from a disk path.`, + `Run this exact native bash command exactly once, without an added shell prefix or suffix:\n${c.validationCommand}\nIt independently reads the file, fails on incorrect bytes, and prints the validation marker only after success. This is the only bash call allowed for this task; do not run any other shell commands.`, + `Deliver the edited text file as a downloadable attachment using the assigned register_deliverable semantic tool, with filename and workspace-relative contentRef ${c.filename}, contentType text/plain, byteSize ${c.byteSize}, sha256 ${c.sha256}, title Verified text, and a fresh idempotencyKey. Use the supplied byteSize and sha256 for the exact validated post-edit bytes; do not run extra commands to obtain metadata. Wait for the real registration receipt. Do not upload a different file or claim publication from a disk path.`, `After successful validation and registration, call paperclip_finish with reportedWorkDisposition done, summary EXTENDED-FILE-${nonce}, the current completion contract revision and satisfied objective criterion, no remaining work, the actual validation command/result, and the registered artifact reference.`, `Wait for paperclip_finish to succeed, then emit exactly EXTENDED-FILE-${nonce} as your final response. Do not create unrelated files or work.`, ].join("\n");