diff --git a/doc/plans/2026-10-02-pi-production-readiness.md b/doc/plans/2026-10-02-pi-production-readiness.md index 5c93f0e43a4..3a07cd692bf 100644 --- a/doc/plans/2026-10-02-pi-production-readiness.md +++ b/doc/plans/2026-10-02-pi-production-readiness.md @@ -7,31 +7,51 @@ widgets, images, Cursor or Copilot qualification. The existing draft stack must ## Production qualification state — 2026-10-07 -The final target is native Linux with Pi profile 17. All 26 accepted Product -cells and seven Runner cases require passes on the final source and installed -artifact set. Mac coverage is deferred. Historical passes do not transfer. +The shipping runtime is frozen at `15af2eefbc2dc9f270875836e9bb404f4ed3d135` +with Pi profile 17. The final target is native Linux. All 26 accepted Product +cells and seven Runner cases require passes on the final installed artifacts +and recorded harness definitions. Mac coverage is deferred. Historical passes +do not transfer. The caller chooses the production model; the accepted model +and low thinking remain fixture inputs. -Review found that Rust's initial broad `LD_` exclusion rejected valid custom -credentials; merely matching the controller then admitted real loader controls. -Both boundaries now reserve the same 40 loader, shell and process-control names -while permitting ordinary custom names such as `LD_API_KEY` and `DYLD_API_KEY`. -They share negative test inputs, and real child-process coverage verifies bound -credential forwarding and unrelated-secret exclusion. Profile 17 binds this -provider-configuration source correction. The Pi 1.0.0 native distributions, -wrapper, runtime extension and platform closure hashes remain unchanged. +Both credential boundaries reserve the same 40 loader, shell and process-control +names while permitting custom names such as `LD_API_KEY` and `DYLD_API_KEY`. +Shared negative inputs and actual child-process tests cover explicit bound-key +forwarding and unrelated-secret exclusion. Current generated release profiles +also decode automatically, with separate exact launch admission. Pi's native +1.0.0 distributions, wrapper, extension and closure hashes remain unchanged. -The normal native builds of `aa67de311` and `f817030f8` passed. Public Linux -package installation for `f817030f8` passed npm lifecycle checks but Pi setup -ran out of the disposable host's 10 GiB scratch disk. Preserve that failure and -superseded archives; reclaim only owned obsolete artifacts and completed caches -before rebuilding the corrected final source. No live attempt ran on `f817`. +| Release gate | Current evidence | +| --- | --- | +| Normal native Linux build | Workspace and standalone public builds pass at the frozen runtime. | +| Normal public installation | CLI/server and separate Daytona plugin npm installs and normal hooks pass; lockfiles stay unchanged. | +| Installed package integrity and Pi admission | Full archive graph and managed-asset verification, normal Pi runtime setup, credential-free public admission and actual server/UI startup pass. Installed daemon SHA-256 is `905949ca03c1859974ce9868aea6262a5338f98dace66bd029a57e11272996be`. | +| Free runtime checks | Full release Rust: 689 pass, two ignored. Runner TypeScript typecheck passes; provider/configuration/compatibility checks: 58 pass, one skip. Actual server native-input boundary: 68 pass. | +| Final Product acceptance | 0/26. The first installed file-edit attempt failed the shared artifact-title check; cleanup passed. Keep its original failure and token usage. | +| Final Runner acceptance | 0/7. Definitions `a29b289cb3a80731c6f8195159d263e5336b4d4a` validate seven cases and pass 139 free tests; final live runs remain pending. | +| Final Daytona image | Requires the prepared Linux workflow-host correction and explicit approval before applying it; 13 Daytona cells remain pending. | +| Current PR-head CI and review | Required after the fixture-only correction. Test-only follow-ups fix future-version and Cursor rejection fixtures. | -The full local Rust repeat at `f817` failed one Codex receipt-observation deadline; -the unchanged test passes in isolation. Linux CI passed its native Runner lane, -but browser shard 5 failed the signoff fixture's run-availability wait. Preserve -both observations and require new-head checks and review. The final source still -needs normal installed admission and all 33 live cases. No merge or release is -authorized. The fixture model and low thinking remain test inputs. +The first final-source file-edit run edited and independently validated the file, +registered its real download and completed the task. Its Pi prompt requested +artifact title `Verified text`, while the unchanged shared registration check +requires the exact filename as title. Extended definition 4 corrects that prompt +instruction and adds a cross-fixture regression check. Native edit, exact bytes, +command/result, registration, run attribution and cleanup grading stay unchanged. +The failed definition-3 attempt does not become a pass. Its recorded usage is +39,343 input, 77,312 cached input and 1,359 output tokens; provider cost is unpriced +and key snapshots may settle later. No unchanged paid retry is permitted. + +Earlier installs exhausted the owned host's 10 GiB scratch disk. The final +installation retires its completed npm cache only after strict base archive +verification, then checks every installed file plus normal managed Pi assets. +A nested/top-level npm bin classification error in the external verifier was +corrected on the same installed bytes; the original failed audit is retained. +No grading rule or production artifact was changed for that correction. +Local full database integration remains blocked by host System V shared-memory +capacity. Preserve those observations and require current Linux CI. No merge, +release or rollout is authorized. The dedicated key retains its $5 lifetime cap +inside the approved $100 total token ceiling, with no BYOK or automatic retries. ## Final fixture reconciliation — 2026-10-07 diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index a3c07a901a0..69e093af4ba 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -373,8 +373,10 @@ projected title and a validation marker. A marker-only echo cannot pass. Final b match the fixture. The Pi task prompt explicitly forbids additional shell calls, including metadata commands, and supplies the expected post-edit byte size and hash for registration. Extra Bash calls fail the unchanged oracle even when the -edited file and downloadable artifact are correct. Extended definitions 3 retain -older attempts under their original definitions and grades. The real +edited file and downloadable artifact are correct. Extended definition 4 makes +Pi's artifact title equal the exact filename required by the shared registered +artifact check. Definition 3 attempts retain their original definitions and +grades; its Pi prompt requested a different title. The real `register_deliverable` receipt, attachment metadata, publication activity, visible task attachment and authenticated public download must all agree on the file's bytes, hash, company, issue, agent and originating run. A file on disk or a diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 815ae8894ec..470683dc38c 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1223,7 +1223,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ groups: ["native"], profiles: extendedHarnessProfiles, environments: runnerEnvironments, tasks: [...openRouterBreadthTasks, localIntegrityTasks[1]!, extendedHarnessFileTask], expectedMatrixSize: 30, - definitionMetadata: { version: 3, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28", piFileEvidence: "seed-edit-single-execute-public-download-v2" }, + definitionMetadata: { version: 4, qualification: "pending", scheduling: "explicit-only", admission: "host-exact-candidate-and-model", authenticatedDiscoveryDate: "2026-09-28", piFileEvidence: "seed-edit-single-execute-public-download-v2", piFileArtifactTitle: "exact-filename" }, }, { id: "instruction-persistence", label: "Instruction Persistence", diff --git a/tests/runner-e2e/pi-file-evidence.test.ts b/tests/runner-e2e/pi-file-evidence.test.ts index 3049eeda59f..cd453423c7f 100644 --- a/tests/runner-e2e/pi-file-evidence.test.ts +++ b/tests/runner-e2e/pi-file-evidence.test.ts @@ -192,6 +192,12 @@ describe("Pi edit, validation and public artifact oracle", () => { for (const cell of cells) { const prompt = cell.task.buildPrompt("fixture"); expect(prompt.includes("register_deliverable")).toBe(cell.profile.qualificationCandidate === "pi"); + if (cell.profile.qualificationCandidate === "pi") { + const artifact = cell.task.buildMatchers("fixture", cell).find(matcher => matcher.kind === "artifact_exact"); + if (!artifact || artifact.kind !== "artifact_exact") throw new Error("Pi file task must require a registered artifact"); + expect(artifact.name).toBe(piFileContract("fixture").filename); + expect(prompt).toContain(`title ${artifact.name},`); + } } expect(runnerMatrix.filter(c => c.profile.qualificationCandidate === "pi")).toHaveLength(26); }); diff --git a/tests/runner-e2e/pi-file-evidence.ts b/tests/runner-e2e/pi-file-evidence.ts index 08b4f1b3422..7e5eeadb996 100644 --- a/tests/runner-e2e/pi-file-evidence.ts +++ b/tests/runner-e2e/pi-file-evidence.ts @@ -22,7 +22,7 @@ export function piFilePrompt(nonce: string) { return [ `The fixture has already created ${c.filename} in your execution workspace with exactly ${JSON.stringify(c.before)}. Read it, then use your native edit tool once to replace those bytes with exactly ${JSON.stringify(c.after)}. Do not substitute write, bash, or an API for the edit.`, `Run this exact native bash command exactly once, without an added shell prefix or suffix:\n${c.validationCommand}\nIt independently reads the file, fails on incorrect bytes, and prints the validation marker only after success. This is the only bash call allowed for this task; do not run any other shell commands.`, - `Deliver the edited text file as a downloadable attachment using the assigned register_deliverable semantic tool, with filename and workspace-relative contentRef ${c.filename}, contentType text/plain, byteSize ${c.byteSize}, sha256 ${c.sha256}, title Verified text, and a fresh idempotencyKey. Use the supplied byteSize and sha256 for the exact validated post-edit bytes; do not run extra commands to obtain metadata. Wait for the real registration receipt. Do not upload a different file or claim publication from a disk path.`, + `Deliver the edited text file as a downloadable attachment using the assigned register_deliverable semantic tool, with filename and workspace-relative contentRef ${c.filename}, contentType text/plain, byteSize ${c.byteSize}, sha256 ${c.sha256}, title ${c.filename}, and a fresh idempotencyKey. Use the supplied byteSize and sha256 for the exact validated post-edit bytes; do not run extra commands to obtain metadata. Wait for the real registration receipt. Do not upload a different file or claim publication from a disk path.`, `After successful validation and registration, call paperclip_finish with reportedWorkDisposition done, summary EXTENDED-FILE-${nonce}, the current completion contract revision and satisfied objective criterion, no remaining work, the actual validation command/result, and the registered artifact reference.`, `Wait for paperclip_finish to succeed, then emit exactly EXTENDED-FILE-${nonce} as your final response. Do not create unrelated files or work.`, ].join("\n");