From 40c1ab36cdac70c85dc8fe5e912431834fdb691b Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 28 Oct 2026 00:54:04 -0500 Subject: [PATCH] feat(evals): register explicit Daytona feedback persistence (#17986) ## Thinking Path > - Paperclip manages AI agents and their work. > - Product E2E evals check real server, browser, runner, and provider behavior. > - Feedback persistence already checks saved complaints and suggestions with exact bodies and run attribution. > - Its catalog registers only local cells, so a Daytona campaign cannot select the same workflow. > - This PR registers the existing workflow for the two native Daytona profiles and tests their gateway bindings. > - Remote cells stay explicit-only and unqualified until a bounded live run passes. ## Linked Issues or Issue Description Refs #17980 and #17975. No duplicate open PR or issue was found in the Daytona feedback search. **What existing behavior does this improve?** The `feedback-persistence` Product E2E suite and its explicit execution selectors. **Current behavior** At master `d54f776ca0`, the suite has only local Claude and Codex cells. Selecting the matching Daytona ID fails before admission. The gateway manifest already supports these native remote profiles. **Proposed behavior** Register four cells: local and Daytona for each native profile. Reuse the existing saved-body and completed-tool oracle. Require an immutable image and Daytona credentials. Keep one attempt, one expected work run, five-minute local and inherited 15-minute remote case timeouts, and the existing 60-minute provider creation TTL. **Reason and benefit** A bounded remote qualification can check both a simple task and real feedback persistence. Local evidence remains separate from remote acceptance. **Breaking changes** Explicit `--suite feedback-persistence` now lists four cells instead of two. `--all` still excludes the suite. Daily and weekly manifests still select their existing exact local IDs. ## What Changed - Register the two native Daytona feedback cells and increment the suite definition version. - Address the remote cold-start review: inherit the base task's 15-minute Daytona deadline and retain the five-minute local limit. The regression failed before the fix and now passes. - Check immutable-image requirements, credentials, lifetime, and shared gateway connections in existing contract tests. - Fix the config type in the TTL catalog test added by #17980. Keep its runtime lifetime assertions. - Document remote selection and its unqualified status in the existing eval guides. ## Verification - Before the catalog change, the registration regression failed: one failed and 16 passed. - Focused catalog, feedback, operations, ledger, manifest, and summary tests: 114 passed across six files. - Read-only discovery lists exactly the two Daytona feedback IDs. - The Node test phase passes separately: 137 tests. The retained Vitest database failure prevents the unit launcher from reaching that phase. - Full Product support suite: 4,523 passed, three skipped, and one embedded PostgreSQL bootstrap failure. The same database failure was reproduced on a clean baseline before this PR. - Fresh Linux Product typecheck passed at the review-fix commit: `GOMEMLIMIT=2GiB GOGC=50 GOMAXPROCS=2 node node_modules/typescript/bin/tsc -p tests/runner-e2e/tsconfig.json`. The default run and the ineffective JavaScript-heap attempt exited 137 and remain retained. The compiler launches a native child process; its memory controls let the unchanged semantic check complete. Host typecheck still has 15 stale installed DB/Runner export errors; the two TTL test typing errors are fixed. All 58 latest-head checks are acceptable (success, neutral, or skipped). Both fresh reviews are 5/5, with no unresolved review threads. - The updated credential-free Linux controller discovers both planned remote IDs and launches Chromium 151.0.7922.34. Runtime build inputs are unchanged from the prior controller. The Daytona image content ID is still `b55f03dc9c633db300b70206a691ff45a8e779e56a3c546f9358e3282d3597c9`. - No model requests, sandbox allocation, or paid schedule changes. ## Risks - Selecting the whole explicit suite now includes remote cells and requires Daytona configuration. Use exact IDs for a bounded paid campaign. - Registration and contract tests do not prove live remote behavior or token accounting. Remote qualification remains pending. - The shared dependency checkout limits the local host typecheck. The fresh Linux semantic check passed. Earlier compiler OOM failures and the database bootstrap failure remain retained. ## Model Used - OpenAI GPT-6 through Codex, with tool use and code execution. The exact serving model ID and context window are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- doc/evals.md | 4 +++- tests/runner-e2e/FIXTURES.md | 13 +++++++++---- tests/runner-e2e/OPERATIONS-COVERAGE.md | 5 +++++ tests/runner-e2e/OPERATIONS.md | 17 ++++++++++------- tests/runner-e2e/README.md | 13 +++++++++---- tests/runner-e2e/catalog.test.ts | 11 ++++++----- tests/runner-e2e/catalog.ts | 4 ++-- tests/runner-e2e/feedback-persistence.test.ts | 15 ++++++++++++++- tests/runner-e2e/feedback-persistence.ts | 2 +- tests/runner-e2e/operations.test.ts | 2 ++ 10 files changed, 61 insertions(+), 25 deletions(-) diff --git a/doc/evals.md b/doc/evals.md index a61e4d6f6e3..6cb2ce4b3da 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -539,5 +539,7 @@ API and download oracles; no live qualification is implied by fixture registrati The explicit-only [feedback persistence case](../tests/runner-e2e/README.md#feedback-persistence-explicit-only) checks real native complaint/suggestion submissions and retained bodies with -run-attributed tool receipts. It is manual, uses the normal Product E2E report, +run-attributed tool receipts on local and Daytona execution. Its four native +Claude/Codex cells are explicit-only. Remote registrations remain unqualified +until retained live attempts prove them. It uses the normal Product E2E report and keeps synthetic feedback distinct from actionable product findings. diff --git a/tests/runner-e2e/FIXTURES.md b/tests/runner-e2e/FIXTURES.md index 377b12913a3..87a2ad25b25 100644 --- a/tests/runner-e2e/FIXTURES.md +++ b/tests/runner-e2e/FIXTURES.md @@ -1021,14 +1021,18 @@ The oracle rejects a missing or pending terminal intent. ## Feedback persistence (explicit only) -`--suite feedback-persistence` selects two local native Claude/Codex cells. +`--suite feedback-persistence` selects four native Claude/Codex cells across local and Daytona execution. Each creates one standard task through the browser, asks for two distinct synthetic feedback items, and requires the real `submit_complaint` and -`submit_suggestion` tools. The five-minute case expects one task work run and -disables automatic retries. Company and agent accounted-work budgets are $10. +`submit_suggestion` tools. The case expects one task work run and disables +automatic retries. Local attempts have five minutes; Daytona inherits the +15-minute remote deadline. Company and agent accounted-work budgets are $10. Setup probes and any background work must also pass through the separately budgeted gateway when using an operations manifest. Local compute is unmetered. -Cost and duration forecasts remain unmeasured until a retained live attempt. +Daytona cells require an immutable image and the existing provider credential. +Their ephemeral sandboxes carry the 60-minute provider lifetime. Historical local +measurements remain separate; Daytona cost and time are unknown until measured. +These registrations do not qualify remote behavior or change daily/weekly selections. The runner E2E maintainers own this manual case; `--all` excludes it. After collection and fixture teardown, the oracle reads the retained evidence @@ -1043,5 +1047,6 @@ quality or survival after an abrupt worker kill. ```sh pnpm test:e2e:runner -- --list --suite feedback-persistence +pnpm test:e2e:runner -- --list --suite feedback-persistence --environment daytona pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1 ``` diff --git a/tests/runner-e2e/OPERATIONS-COVERAGE.md b/tests/runner-e2e/OPERATIONS-COVERAGE.md index 1b75a563435..6cc961daca6 100644 --- a/tests/runner-e2e/OPERATIONS-COVERAGE.md +++ b/tests/runner-e2e/OPERATIONS-COVERAGE.md @@ -61,6 +61,11 @@ current `pnpm test:e2e:runner -- --list` before choosing work. In particular, Everyday Workflows, provider connections, feedback persistence, interruption, and case handoffs must not be inferred from a passing `--all` campaign. +The explicit feedback suite also registers native Claude/Codex cells on Daytona. +Their model usage, duration, feedback persistence and cleanup remain unqualified +until retained live attempts prove them. These cells do not enter the daily or +weekly selections, and `--all` continues to exclude the suite. + Keep the broader rotating lane disabled until its exact cells have a metered route, complete retained attempts, a measured cost/time distribution, and a budget that includes failures. Prefer targeted continuation, approval, and diff --git a/tests/runner-e2e/OPERATIONS.md b/tests/runner-e2e/OPERATIONS.md index 5d45ece6627..1a860869e37 100644 --- a/tests/runner-e2e/OPERATIONS.md +++ b/tests/runner-e2e/OPERATIONS.md @@ -11,11 +11,13 @@ Eval audit, and remaining activation gaps. ## Supported admission -The initial path admits local `core-compatibility` and `feedback-persistence` cells for `runner-codex`, -`runner-acpx-claude`, `legacy-codex`, and `legacy-claude`. Each cell must have one -provider credential and an exact priced model. Other suites, remote environments, -subscription profiles, hosted tools, multimodal inputs, premium service modes, -and separate judges fail admission. This whitelist is implementation scope, not +The bounded path admits local `core-compatibility` and `feedback-persistence` cells +for `runner-codex`, `runner-acpx-claude`, `legacy-codex`, and `legacy-claude`. +Daytona admits the two native profiles in those suites when the manifest declares +`remoteGateway: "cloudflare"`. Each cell needs one exact priced model and its +provider credential; Daytona also needs its provider key. Other suites and remote +routes, subscription profiles, hosted tools, multimodal inputs, premium service +modes, and separate judges fail admission. This whitelist is implementation scope, not evidence that every admitted harness has passed live qualification. Before creating the journal or requesting credentials, manifest validation checks @@ -899,8 +901,9 @@ launch manifests must use the adopted cap. The migration cannot extend an expire deadline, change an existing cap, or replenish an allowance. An interrupted append remains an accounting failure and must be recovered before more paid work. -The explicit local `feedback-persistence` native Claude/Codex cells also accept -an operations manifest. They use the same metered connection, prices, campaign +The explicit `feedback-persistence` native Claude/Codex cells on local and Daytona +execution also accept an operations manifest. Remote cells require the Cloudflare +gateway declaration and immutable image. They use the same metered connection, prices, campaign journal, setup boundary, and durable attempt allowance as core compatibility. Collection and the independent saved-evidence oracle require no additional model call. The synthetic items need review as test data, not product incidents. diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index a97715ed1ed..2d1545a339f 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -2985,14 +2985,18 @@ from live qualification and lists remaining billing/reporting coverage gaps. ## Feedback persistence (explicit only) -`--suite feedback-persistence` selects two local native Claude/Codex cells. +`--suite feedback-persistence` selects four native Claude/Codex cells across local and Daytona execution. Each creates one standard task through the browser, asks for two distinct synthetic feedback items, and requires the real `submit_complaint` and -`submit_suggestion` tools. The five-minute case expects one task work run and -disables automatic retries. Company and agent accounted-work budgets are $10. +`submit_suggestion` tools. The case expects one task work run and disables +automatic retries. Local attempts have five minutes; Daytona inherits the +15-minute remote deadline. Company and agent accounted-work budgets are $10. Setup probes and any background work must also pass through the separately budgeted gateway when using an operations manifest. Local compute is unmetered. -Cost and duration forecasts remain unmeasured until a retained live attempt. +Daytona cells require an immutable image and the existing provider credential. +Their ephemeral sandboxes carry the 60-minute provider lifetime. Historical local +measurements remain separate; Daytona cost and time are unknown until measured. +These registrations do not qualify remote behavior or change daily/weekly selections. The runner E2E maintainers own this manual case; `--all` excludes it. After collection and fixture teardown, the oracle reads the retained evidence @@ -3007,6 +3011,7 @@ quality or survival after an abrupt worker kill. ```sh pnpm test:e2e:runner -- --list --suite feedback-persistence +pnpm test:e2e:runner -- --list --suite feedback-persistence --environment daytona pnpm test:e2e:runner -- --id feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion --max-automatic-retries 0 --max-parallel 1 ``` diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index bb499b6970c..009445d2723 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -219,10 +219,10 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 2, 28, 18, 2, 6, 6, 13, 10, 2, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, + 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 2, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 2, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(769); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(769); + expect(validateRunnerCatalog()).toHaveLength(771); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(771); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); @@ -259,12 +259,13 @@ describe("runner E2E catalog", () => { executionId: "bounded-lifetime", daytonaImage: `runner@sha256:${"a".repeat(64)}`, secretRefs: { DAYTONA_API_KEY: { type: "secret_ref", secretId: "22222222-2222-4222-8222-222222222222", version: "latest" } }, }); - const lifetimeMs = Number(payload.config?.ttlMinutes) * 60_000; + const config = payload.config as Record; + const lifetimeMs = Number(config.ttlMinutes) * 60_000; expect(Number.isFinite(lifetimeMs)).toBe(true); const cells = runnerMatrix.filter(entry => entry.environment === environment); expect(cells.length).toBeGreaterThan(0); for (const cell of cells) { - expect(lifetimeMs, cell.id).toBeGreaterThan(cell.task.attemptTimeoutMs.daytona + Number(payload.config?.timeoutMs)); + expect(lifetimeMs, cell.id).toBeGreaterThan(cell.task.attemptTimeoutMs.daytona + Number(config.timeoutMs)); } }); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 24691ef102c..d5f83321734 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1698,8 +1698,8 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ description: "Real complaint and suggestion submissions, durable bodies, and retained tool receipts.", groups: ["native"], profiles: runnerProfiles.filter(profile => ["runner-codex", "runner-acpx-claude"].includes(profile.id)), - environments: [localEnvironment], tasks: [feedbackPersistenceTask(runnerTasks.find(task => task.id === "message-marker")!)], expectedMatrixSize: 2, - definitionMetadata: { version: 1, owner: "runner-e2e-maintainers", scheduling: "explicit-only", + environments: runnerEnvironments, tasks: [feedbackPersistenceTask(runnerTasks.find(task => task.id === "message-marker")!)], expectedMatrixSize: 4, + definitionMetadata: { version: 2, owner: "runner-e2e-maintainers", scheduling: "explicit-only", sourceDigest: feedbackPersistenceDigest(runnerTasks.find(task => task.id === "message-marker")!), expectedWorkRuns: 1, automaticRetryPolicy: "single_attempt", budgetCents: 1_000, forecast: "unmeasured; gateway manifest required for bounded qualification", grading: "retained-exact-bodies-and-run-attributed-completed-tools", feedback: "synthetic-disposable-items" }, diff --git a/tests/runner-e2e/feedback-persistence.test.ts b/tests/runner-e2e/feedback-persistence.test.ts index ca178d36a78..9a81b4b0cb0 100644 --- a/tests/runner-e2e/feedback-persistence.test.ts +++ b/tests/runner-e2e/feedback-persistence.test.ts @@ -55,13 +55,26 @@ describe("feedback persistence oracle", () => { expect(gradeFeedbackPersistence({ ...fixture(), evidence }).some(check => !check.passed)).toBe(true); } }); - it("registers only two explicit local native cases with no automatic retry", () => { + it("registers explicit local and Daytona native cases with no automatic retry", () => { const selected = selectRunnerExecutions(parseRunnerSelectors(["--suite", "feedback-persistence"])); expect(selected.map(execution => execution.id)).toEqual([ "feedback-persistence.runner-codex.local.complaint-and-suggestion", + "feedback-persistence.runner-codex.daytona.complaint-and-suggestion", "feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion", + "feedback-persistence.runner-acpx-claude.daytona.complaint-and-suggestion", ]); expect(selected.every(execution => execution.task.automaticRetryPolicy === "single_attempt")).toBe(true); + expect(selected.every(execution => execution.task.attemptTimeoutMs.local === 5 * 60_000)).toBe(true); + for (const execution of selected.filter(entry => entry.environment.id === "daytona")) { + expect(execution.requiredCredentials).toEqual(expect.arrayContaining([execution.profile.credential, "DAYTONA_API_KEY"])); + expect(execution.requiredCredentials).toHaveLength(2); + expect(execution.task.expectedRunCount).toBe(1); + expect(execution.task.attemptTimeoutMs.daytona).toBe(15 * 60_000); + const secretRefs = { DAYTONA_API_KEY: { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const } }; + expect(() => execution.environment.buildEnvironment({ executionId: execution.id, secretRefs, daytonaImage: "runner:latest" })).toThrow("immutable"); + const payload = execution.environment.buildEnvironment({ executionId: execution.id, secretRefs, daytonaImage: `runner@sha256:${"a".repeat(64)}` }); + expect(payload).toMatchObject({ driver: "sandbox", config: { provider: "daytona", reuseLease: false, ttlMinutes: 60 } }); + } expect(runnerMatrix).toEqual(expect.arrayContaining(selected)); expect(selectRunnerExecutions(parseRunnerSelectors(["--all"])).some(execution => execution.suite.id === "feedback-persistence")).toBe(false); }); diff --git a/tests/runner-e2e/feedback-persistence.ts b/tests/runner-e2e/feedback-persistence.ts index fd4faf1ee76..af2c5c9febc 100644 --- a/tests/runner-e2e/feedback-persistence.ts +++ b/tests/runner-e2e/feedback-persistence.ts @@ -26,7 +26,7 @@ export function expectedFixtureFeedback(nonce: string) { export function feedbackPersistenceTask(base: RunnerTaskFixture): RunnerTaskFixture { return { ...base, id: "complaint-and-suggestion", label: "Retain submitted feedback", automaticRetryPolicy: "single_attempt", - attemptTimeoutMs: { local: 5 * 60_000, daytona: 5 * 60_000 }, + attemptTimeoutMs: { local: 5 * 60_000, daytona: base.attemptTimeoutMs.daytona }, buildPrompt(nonce) { const bodies = expectedFixtureFeedback(nonce); return [ diff --git a/tests/runner-e2e/operations.test.ts b/tests/runner-e2e/operations.test.ts index 74cff67c76f..7c2ef3f6abd 100644 --- a/tests/runner-e2e/operations.test.ts +++ b/tests/runner-e2e/operations.test.ts @@ -164,6 +164,8 @@ it.each([ cell.id, "feedback-persistence.runner-codex.local.complaint-and-suggestion", "feedback-persistence.runner-acpx-claude.local.complaint-and-suggestion", + "feedback-persistence.runner-codex.daytona.complaint-and-suggestion", + "feedback-persistence.runner-acpx-claude.daytona.complaint-and-suggestion", "core-compatibility.runner-acpx-claude.daytona.message-marker", ])("routes %s through a shared connection without ambient provider bindings", async executionId => { const selected = runnerMatrix.find(execution => execution.id === executionId)!;