From c2893e1acd75335bb41b1fde20d56fe3c6dca5ee Mon Sep 17 00:00:00 2001 From: Dotta Date: Wed, 7 Oct 2026 16:26:53 -0500 Subject: [PATCH] test(pi): preserve exact memory LF with Unicode JSON arguments Co-Authored-By: Paperclip --- .../2026-10-02-pi-production-readiness.md | 41 ++++++++++++++++++- tests/runner-e2e/FIXTURES.md | 11 +++++ tests/runner-e2e/catalog.ts | 2 +- tests/runner-e2e/pi-controls-evidence.test.ts | 2 +- tests/runner-e2e/pi-native-cases.test.ts | 3 +- tests/runner-e2e/pi-native-cases.ts | 8 +++- 6 files changed, 61 insertions(+), 6 deletions(-) diff --git a/doc/plans/2026-10-02-pi-production-readiness.md b/doc/plans/2026-10-02-pi-production-readiness.md index 21979a47dc9..a4b1b21ebf3 100644 --- a/doc/plans/2026-10-02-pi-production-readiness.md +++ b/doc/plans/2026-10-02-pi-production-readiness.md @@ -12,7 +12,9 @@ the accepted OpenRouter model and low thinking are fixture inputs. The new Runner snapshot-retirement repair requires fresh normal Linux packages, a cloud image and all 33 accepted live cases. Provider wrapper, closure, profile and model-selection declarations are unchanged by this host-lifecycle repair. -Mac qualification remains deferred. The following repair stages are historical; +Normal ARM and Intel Mac install admission remains required by the active goal; +historical Mac evidence does not qualify the current shipping source. +The following repair stages are historical; the release-gate table below and exact shutdown-source section record current admission. Do not repeat completed builds or count passes on older artifacts. @@ -77,6 +79,8 @@ bytes. The source must be rebuilt and admitted before another paid attempt. All | Offline cancellation proof | Profile 18 passes all 49 Pi ACP protocol checks. Historical profile 17 remains decodable; exact launch requires profile 18. | | Snapshot retirement proof | Shipping source `06a3d9739` passes 271 focused TypeScript tests, one existing skip, 16 Rust transport tests and Runner typecheck. Actual installed native copy, cancellation and deletion probes pass. Three live cases and both memory failures have independent zero-snapshot cleanup. | | Normal Linux build and public installation | Source `06a3d9739` passes normal workspace/public builds, npm installation and lifecycle hooks, full graph audits, Pi setup and real server/UI admission. All 20 archives and exact installed daemon bytes are retained and verified. | +| Normal ARM Mac installation | Current shipping-source normal install, lifecycle, Pi setup and packaged admission proof remains required. Historical passes are not transferred. | +| Normal Intel Mac installation | Current shipping-source normal install, lifecycle, Pi setup and native Intel admission proof remains required. Historical x64 execution under Rosetta is emulation evidence. | | Final Product acceptance | Current shipping artifacts have 3/26 passes: local Stop, steering and native questions. The local memory case fails the strict final-newline check on original harness `06a3d9739` and clarified harness `56d1e03ac`. Both canonical failures remain unchanged. The other 22 Product cells are unexecuted. No automatic retries. | | Final Runner acceptance | Current shipping coverage is 0/7. All seven cases, model, low thinking, 120-second limits and scoring rules are pinned to definition revision `d38ebc67`. | | Cloud image | [Hosted Linux build](https://github.com/paperclipai/paperclip/actions/runs/37676853634) passes for source `06a3d9739`. Immutable image `sha256:5e62c8e294cd9d663316ba09b0aae1d7358a8938fe37d17936ef394cf1723ab2` passes anonymous OCI verification and actual credential-free Daytona guest admission; probe deletion is confirmed. No local Docker is used. | @@ -163,6 +167,41 @@ The actual memory attempt began with 2,050,457,600 bytes free. The original preflight rejection and both cache-reclamation diagnostics remain retained; none is reclassified as a live qualification attempt. +## Frozen-model memory encoding correction — 2026-10-07 + +The active goal keeps `openrouter/deepseek/deepseek-v4-flash-0731` and native +low thinking. The proposed replacement qualification model is not applied. +Shipping packages and the admitted image remain frozen at `06a3d9739`; no +production input or runtime profile changes. + +The existing real Pi RPC warm-recovery test passes against the exact installed +profile-18 wrapper. Its fragmented JSON write preserves the final LF on disk, +a fresh wrapper restores the session and agent home, and both owned Pi children +close. External sockets are denied before connect; responses come from an +owned synthetic loopback fixture. This is diagnostic proof, not live coverage. + +Pi native definition 17 changes the authoritative write JSON's LF encoding from +`\n` to `\u000a`. Both decode to the same 33 UTF-8 bytes; the final byte remains +10. The installed native write/read probe preserves that Unicode-escaped content +and an unchanged 32-byte negative control exactly. The original model failures +remain retained. No grader, permission, deadline, case ID or production behavior +changes. The new suite fingerprint must be recorded; prior native definition-16 +passes remain their original measurements. Unchanged controls retain their +original fingerprints and installed identities. + +Run the free harness checks against an exact new harness revision, then permit +one explicitly selected changed memory attempt under the same $5 key lifetime +cap and $100 campaign ceiling. Keep zero automatic retries, no BYOK or fallback, +and the same 2 GB disk floor. Stop and retain canonical and independent cleanup +evidence on any failure. An encoding change does not itself establish reliability +or production readiness. + +Documentation head `3fee270de` passes all 54 applicable CI checks with two skips +after one failed-job rerun of two simultaneous runner shutdowns. Original logs +remain retained. Its Greptile review is 5/5 with no unresolved threads; this new +harness change requires fresh current-head CI and review. All remaining Product, +Runner and platform installation gates remain required. Do not merge or release. + ## Final fixture reconciliation — 2026-10-07 Fresh Linux CI passed the server, database, UI and package checks. It exposed diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index da67e2712e1..4b22b09f51b 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -31,6 +31,17 @@ profile, model qualification, environment, task, or ranking-snapshot change must change that fingerprint automatically so the dashboard can annotate the boundary instead of silently joining unlike totals. +Pi native definition 17 keeps the memory outcome at exactly the nonce plus one +final LF. Its authoritative native-write JSON spells that byte as `\u000a` +instead of `\n`; both decode to identical content. The installed Pi tools +preserve the 33-byte Unicode-escaped payload and the 32-byte negative control +exactly. The strict managed-file and fresh-task byte comparisons, single native +write/read, cross-root denial, model, low thinking, permissions and deadlines are +unchanged. Retain both previous missing-LF failures and their original catalog +fingerprints. This encoding correction permits one recorded changed memory +attempt, with zero automatic retries. Free byte preservation is diagnostic +evidence, not a live qualification pass. + The explicit [stock-harness suite](STOCK-HARNESS.md) wraps existing profiles with `productionDefaultHireProfile`: omit only `instructionsBundle` so the public hire route loads the shipped default, while preserving runtime, permissions, diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index fdd8594525c..2d8b4535896 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1132,7 +1132,7 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ groups: ["native"], profiles: extendedHarnessProfiles.filter(profile => profile.qualificationCandidate === "pi"), environments: runnerEnvironments, tasks: piNativeTasks, expectedMatrixSize: 10, excludedExecutionIds: ["pi-native.runner-acpx-pi.daytona.restrictive-denial", "pi-native.runner-acpx-pi.local.native-pending-provider-death"], - definitionMetadata: { version: 16, taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "ordered-native-write-arguments-with-preserved-json-lf", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerDeath: "daytona-exact-pi-child-pidfd-production-expiry", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7", remoteBootstrapApproval: "exact-published-native-read-public-accept-once-v1", remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request-trusted-ancestry-cleanup" }, + definitionMetadata: { version: 17, taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", qualification: "pending", scheduling: "explicit-only", profileVersion: QUALIFIED_ACPX_PROFILES.pi.agentProfileVersion, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "ordered-native-write-arguments-with-json-unicode-lf", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerDeath: "daytona-exact-pi-child-pidfd-production-expiry", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7", remoteBootstrapApproval: "exact-published-native-read-public-accept-once-v1", remoteDenyAll: "unsupported-native-bootstrap-read-is-denied", remoteEvidence: "owned-lease-sealed-observer", pendingControllerRestart: "same-live-native-request-trusted-ancestry-cleanup" }, }, { id: "native-active-stop", label: "Stop an unanswered native permission", manualOnly: true, diff --git a/tests/runner-e2e/pi-controls-evidence.test.ts b/tests/runner-e2e/pi-controls-evidence.test.ts index 0656be06b7f..8279499cc0d 100644 --- a/tests/runner-e2e/pi-controls-evidence.test.ts +++ b/tests/runner-e2e/pi-controls-evidence.test.ts @@ -189,7 +189,7 @@ describe("Pi controls catalog admission", () => { // Extended v3 states the strict file oracle's task-wide Bash limit. const pi = runnerMatrix.find(c => c.profile.qualificationCandidate === "pi")!.profile; expect(pi.modelQualification?.qualificationId).toBe("pi:0.0.33:1.0.0:openrouter"); - expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 16, taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 18, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "ordered-native-write-arguments-with-preserved-json-lf", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); + expect(runnerSuites.find(s => s.id === "pi-native")!.definitionMetadata).toMatchObject({ version: 17, taskCreation: "explicit-title-and-creation-response-id", agentMemoryParent: "public-managed-file-seed-before-admission", incompleteTerminalCleanup: "retirement-retained-with-failed-watch", profileVersion: 18, agentMemoryContent: "utf8-nonce-plus-final-lf", agentMemoryPrompt: "ordered-native-write-arguments-with-json-unicode-lf", taskPromptTransport: "fenced-markdown-paste-and-multiline-literal-escapes", nativeFinish: "current-contract-objective-evidence-refs", providerFaultExecutable: "stable-preinstalled-runner-link-and-snapshot-node-inode-with-held-bootstrap-fd-3-or-7" }); expect(runnerSuites.find(s => s.id === "pi-controls")!.definitionMetadata).toMatchObject({ version: 9, taskCreation: "explicit-title-and-creation-response-id", profileVersion: 18, remoteProcessIdentity: "observer-pid-startTicks-bootId", controlPlaneSettlement: "required-scoped-result-and-terminal-after-runner" }); diff --git a/tests/runner-e2e/pi-native-cases.test.ts b/tests/runner-e2e/pi-native-cases.test.ts index 7f479a05fd1..c3783c9f2ed 100644 --- a/tests/runner-e2e/pi-native-cases.test.ts +++ b/tests/runner-e2e/pi-native-cases.test.ts @@ -13,7 +13,8 @@ describe("Pi native Product qualification", () => { const args = JSON.parse(encoded); expect(Object.keys(args)).toEqual(["path", "content"]); expect(args.path).toBe("/memory/pi-native.txt"); - expect(encoded).toContain('\\n"}'); + expect(encoded).toContain('\\u000a"}'); + expect(encoded).not.toContain('\\\\u000a'); const content = args.content; expect(content).toBe(`${nonce}\n`); expect(Buffer.byteLength(content)).toBe(33); diff --git a/tests/runner-e2e/pi-native-cases.ts b/tests/runner-e2e/pi-native-cases.ts index 5bf491bab12..fe7478826e0 100644 --- a/tests/runner-e2e/pi-native-cases.ts +++ b/tests/runner-e2e/pi-native-cases.ts @@ -25,13 +25,17 @@ export function piNativeFinish(marker: string): string { } export function piNativeMemoryPrompt(nonce: string, outsidePath?: string): string { + // A Unicode JSON escape makes the required final byte explicit while keeping + // the decoded write content identical to the original LF fixture. + const writeArguments = JSON.stringify({ path: `/${PI_NATIVE_MEMORY_PATH}`, content: `${nonce}\n` }) + .replace(/\\n"}$/, '\\u000a"}'); return [ "Follow these steps in order. Use native file tools; do not use bash or the instructions API.", "1. Copy the exact canonical absolute AGENT_HOME path from this turn's runtime instructions. Native file paths do not expand $AGENT_HOME. Do not use a remembered path or reconstruct any directory name.", - `2. Call native write exactly once with the JSON arguments below. Replace only the prefix in path with the exact absolute directory copied in step 1. Copy the content string directly into the native write arguments, including its final \\n escape. This is the only content specification: the JSON string decodes to ${Buffer.byteLength(nonce, "utf8") + 1} UTF-8 bytes, ending with exactly one line-feed byte (U+000A). Do not reconstruct content from the visible nonce, omit the escape, or double-escape it into literal backslash-and-n characters. This is personal memory, not a task deliverable. Do not edit AGENTS.md or a project file.`, + `2. Call native write exactly once with the JSON arguments below. Replace only the prefix in path with the exact absolute directory copied in step 1. Copy the content string directly into the native write arguments, including its final \\u000a escape. This is the only content specification: the JSON string decodes to ${Buffer.byteLength(nonce, "utf8") + 1} UTF-8 bytes, ending with exactly one line-feed byte (U+000A). The Unicode escape is part of content, not a closing delimiter. Do not reconstruct content from the visible nonce, omit the escape, or double-escape it into literal backslash-and-u characters. This is personal memory, not a task deliverable. Do not edit AGENTS.md or a project file.`, "Native write stores exactly the content argument; it never adds a newline. Include the final JSON newline escape in the write argument itself. Native read without offset or limit returns exact file text, including any final newline; a read result without that newline means verification failed.", "```json", - JSON.stringify({ path: `/${PI_NATIVE_MEMORY_PATH}`, content: `${nonce}\n` }), + writeArguments, "```", "3. Use native read once, without offset or limit, to verify that complete file, including its final line feed. Do not trim or repeat the memory write. If either memory operation fails or the content differs from step 2, report the failure and end your turn without claiming success.", ...(outsidePath === undefined ? [] : [