From 513c03cb3e67ec08c45f4528db130d8dfdab3198 Mon Sep 17 00:00:00 2001 From: Dotta <34892728+cryppadotta@users.noreply.github.com> Date: Wed, 4 Nov 2026 14:40:45 -0600 Subject: [PATCH] fix(e2e): configure supported rolling agents with warm policy (#18994) ## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Rolling tests must exercise the lifecycle they claim to cover. > - The warm lane selected a warm environment. > - Its profiles also set an explicit per-turn agent policy, which took precedence. > - The test then waited for a live workspace that per-turn cleanup had correctly retired. > - This pull request sets warm agent policy for providers admitted by the production warm capability. > - The matrix now checks effective lifecycle and preserves per-turn coverage for every qualified native profile. ## Linked Issues or Issue Description Builds on merged #18982 for the source controller dependency link. This PR now targets master. **What happened?** The warm rolling fixture built a Codex agent with `lifecycleMode: per_turn` and an environment with `runnerLifecycleMode: warm`. The explicit agent policy won. The initial task completed, its disposable workspace retired, and the fixture waited for a live file before it reached deployment. **Expected behavior** Warm rolling, legacy migration, remote-file and controller-outage cells must create actual warm agents. Only providers supported by the production managed warm-session capability belong in that lane. All qualified native profiles retain per-turn coverage. **Steps to reproduce** Build the old warm Codex cell's agent and environment and compare their lifecycle policies. The new effective-policy regression fails on the old catalog and passes on this branch. No duplicate exact-scope warm-fixture PR was found. ## What Changed - Share the production managed warm capability filter and explicit warm agent policy across rolling, legacy migration, remote-file and controller-outage suites. - Check effective agent and environment policies and update matrix coverage/counts. - Read completed warm turns from the exact saved snapshot while independently requiring the retained, bound workspace location. - Keep both rolling lanes separate from the outage simulation and exercise their assembled supervisor requests. - Version the corrected definitions and keep every qualified native profile in the per-turn lanes. ## Verification - Baseline regression: failed against the unchanged catalog; retained before the fix. - Fixed catalog/controller selection: 39 checks passed. - The first real warm-policy attempt completed and saved its workspace but timed out because the file oracle demanded live content; retained trace responses prove the API returned the correct saved snapshot with a retained lease. - A new regression fails on the previous reader and passes after separating saved file source from retained versus released location. Focused catalog, reader and rolling checks: 81 passed. - The next warm attempt reached replacement and exposed a second fixture defect: warm also selected a SIGKILL outage, which the supervisor rejected alongside rolling mode. The failure remains retained; its four owned sandboxes, temporary key and local bucket were deleted. - An outage-selection regression failed before the fix; 57 outage/supervisor checks pass afterward, including assembled requests for both rolling lanes. The fresh warm Codex run passed all 56 checks in 8.5 minutes: original run/session/runner/lease retained, authenticated takeover, both predecessor homes removed, later native continuation and saved file readback. Two replacement receipts recorded 742 ordinary reads and 75 writes, all 75 read back, with zero HTTP probe failures. A transient `native_workspace_sync_out_failed` phase recovered before successful finalization and remains retained. This source-lane result does not claim released-image or hosted Railway qualification. - The invalid earlier attempt was interrupted before deployment. Its four archived disposable sandboxes were deleted and verified absent; its temporary key and local bucket were removed. - Broad local typecheck reached stale shared database declarations in this checkout's linked build dependencies. The fixture's own type error was corrected; clean-head CI remains required. No full build is claimed locally. - An incidental report-catalog test selected by a substring filter failed to generate its normalized output. That failure is retained separately; the focused catalog/controller checks pass. - Review follow-up: fixed the same policy conflict in the base file-storage, legacy migration and controller-outage suites. The roster now has 822 valid cells; unsupported warm combinations remain covered per-turn. Updated the explicit supervisor-request control to use a supported warm provider. Focused catalog, file-reader, migration, rolling and supervisor checks pass (133 checks). - Earlier full CI passed at dd5aa71 using the built-in GitHub-hosted fallback after self-hosted runner shutdowns. That result does not qualify this new head; fresh CI and review are required. The final head now has a 5/5 review and passing full CI (run 41901807639, attempt 2). Attempt 1 hit two unrelated port collisions: the unchanged managed Codex credential lease test and CLI company-server startup on a busy requested port. Both passed on one failed-job retry. Those failures remain retained. ## Risks This changes test setup and coverage. It does not change agent runtime defaults or provider capabilities. Warm cells use the production capability boundary, while every qualified native profile keeps its per-turn cell. ## Model Used OpenAI Codex, exact model ID `gpt-6.1-sol`, with reasoning, tool use and code execution. The host does not expose context window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked existing issues or described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name describes the change and contains no internal ticket id or instance-derived details - [x] I have run the relevant tests locally; failures and limits are retained above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip --- tests/runner-e2e/README.md | 30 +++++++++----- tests/runner-e2e/catalog.test.ts | 33 +++++++++++---- tests/runner-e2e/catalog.ts | 40 +++++++++++++------ tests/runner-e2e/controller-outage.ts | 3 +- .../controller-rolling-deployment.test.ts | 12 ++++-- tests/runner-e2e/everyday-flow.ts | 11 +++-- .../remote-workspace-storage.test.ts | 31 +++++++++++++- tests/runner-e2e/remote-workspace-storage.ts | 10 ++++- .../runner-e2e/server-restart-request.test.ts | 28 +++++++++---- 9 files changed, 151 insertions(+), 47 deletions(-) diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index bd446143639..8c0243c4fa9 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -63,8 +63,11 @@ chat. Other YAML entries remain planning inventory; registration is not live qualification. The explicit-only `daytona-restart-continuity` suite runs the existing -`recover-controller` journey with the recorded Codex and Claude staging models -on fresh reusable warm Daytona fixtures. The sandbox, Runner and provider history +`recover-controller` journey with the recorded Codex staging model +on a fresh reusable warm Daytona fixture. Warm controller-replacement and legacy +migration suites use the production managed warm-session capability to select +profiles and set both the agent and environment policy to warm. Claude, OpenCode +and Grok remain covered by the per-turn rolling and migration lanes. The sandbox, Runner and provider history must survive both controller replacements. After a completed task tool in an active revision turn, it keeps the owned development controller offline for 300 seconds. This exceeds the 60-second ownership lease and remains within the Runner's 420-second @@ -83,7 +86,7 @@ ownership/reconnect limits, archive/delete backstops and owned cleanup remain un The retained continuation accepts whitespace between its exact marker and the exact `queue_r` output token; readiness and final grading use the same matcher. -Definition version 7 uses the existing warm reusable environment with a dedicated +Definition version 8 uses an explicitly warm agent and the warm reusable environment with a dedicated configuration key. Only the post-completion replacement follow-up uses the normal board-authenticated task-comment API, with one persisted user comment and the unchanged follow-up request. Initial creation, active outage and queued continuation @@ -104,9 +107,10 @@ its DB checkpoint alone cannot replace those bytes. Per-turn application qualification under this disk loss awaits the provider-backup owner's durable storage contract; native controller recovery does not waive that dependency. -`rolling-web-deployment` definition 4 selects every registered native Daytona +`rolling-web-deployment` definition 6 selects every registered native Daytona profile in disposable per-turn environments; `rolling-web-deployment-warm` -uses the same profiles in reusable warm environments. Their +selects providers supported by the production managed warm-session capability +and explicitly sets both agent and environment policy to warm. Their provider-backup path uses encrypted object storage and a fenced database ledger. The earlier per-turn failure remains failed evidence; this definition requires a fresh passing run, including the strict post-replacement continuation oracle. @@ -115,7 +119,9 @@ the replacement time to become ready while useful agent work remains active. The observer rechecks the original live owner immediately before switching traffic and fails if that turn already settled. A completed turn cannot count as active takeover. Per-turn file inspection requires the exact committed saved -snapshot; the warm lane still requires the live workspace. +snapshot. Completed warm turns also read that exact saved snapshot, while their +workspace location must remain bound to the retained sandbox. The API's saved +file response is independent of whether the sandbox is retained or retired. Cold restores retain the published workspace directory name, including after the old allocation is destroyed, because provider recovery binds the working directory into session identity. Unpublished preparations retain distinct @@ -126,7 +132,7 @@ The ordinary Everyday matrix retains its 125-second active outage default. The two explicit continuity suites require 300 seconds and reject a shorter override. The explicit-only `remote-workspace-storage` suite reuses that controller-restart -journey for one native Codex Daytona cell. It verifies the default-enabled remote +journey for one explicitly warm native Codex Daytona cell. It verifies the default-enabled remote workspace setting through the ordinary instance API and creates a projectless task. It enables the existing experimental Files viewer through that API. The agent creates only synthetic source, an unpushed Git commit, and an untracked random @@ -141,7 +147,8 @@ after completion, a second replacement must admit a useful fourth native turn through the ordinary task-comment API with the same native and provider session. That request specifies one plain-text line containing its marker, one space, and the command's actual output; the expected output stays private to the evaluator. -The browser then reads the saved reference through Files again. This is same-task +The browser then reads the saved reference through Files again, with the sandbox +location still retained. This is same-task continuation, not a separate-task sharing test. The ordinary original-process/lease, receipt-prefix, queued-message, artifact, and strict retained-session oracles still apply. @@ -191,6 +198,9 @@ Postgres. Each controller has an independent compiled build and cache. The copie app retains the checkout's `cli/` and `packages/` dependency links, including the development Runner vendor entry. Shared workspace dependencies mean this is not released-container or Railway evidence. +The warm lane selects providers through the production managed warm-session +capability and explicitly sets the agent policy to warm. The per-turn lane +retains every registered provider's per-turn policy. The read-only recovery observer captures the live predecessor's identity just before handoff, then requires a new controller generation, authenticated Runner attachment, and new canonical events for the original execution. No fixture @@ -3519,7 +3529,9 @@ change credentials, or change evaluation and evidence limits. # Legacy workspace migration `legacy-workspace-migration` and `legacy-workspace-migration-warm` are explicit, -paid, two-turn Daytona workflows for registered native production profiles. They +paid, two-turn Daytona workflows. The per-turn lane covers every registered native +production profile; the warm lane selects supported managed warm providers and +sets the agent policy explicitly to warm. They create a private reference file and ask a question on a local-disk controller. The supervisor stops that controller, runs the packaged offline migration with owned PostgreSQL and S3, verifies the export, starts an empty-home controller, diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 044865f6cae..969b43626cc 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -33,6 +33,25 @@ import { } from "./selectors.js"; describe("runner E2E catalog", () => { + it("uses effective supported warm agent policy across controller replacement and migration suites", () => { + const warmSuites = ["rolling-web-deployment-warm", "legacy-workspace-migration-warm", "remote-workspace-storage", "daytona-restart-continuity"]; + const warm = runnerMatrix.filter(cell => warmSuites.includes(cell.suite.id)); + for (const id of warmSuites) expect(warm.some(cell => cell.suite.id === id && cell.profile.id === "runner-codex"), id).toBe(true); + const perTurn = runnerMatrix.filter(cell => cell.suite.id === "rolling-web-deployment"); + expect(warm.some(cell => ["runner-opencode", "runner-acpx-claude", "runner-acpx-grok"].includes(cell.profile.id))).toBe(false); + expect(perTurn.length).toBeGreaterThan(runnerMatrix.filter(cell => cell.suite.id === "rolling-web-deployment-warm").length); + for (const cell of [...warm, ...perTurn]) { + const secretRefs = Object.fromEntries([cell.profile.credential, "DAYTONA_API_KEY"].map(key => [key, + { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const }])); + const input = { executionId: cell.id, secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` }; + const agent = cell.profile.buildAgent({ ...input, environmentId: "env-1", environmentFixtureId: "daytona", workspacePath: "/workspace" }); + const environment = cell.environment.buildEnvironment(input); + const expected = warmSuites.includes(cell.suite.id) ? "warm" : "per_turn"; + expect(agent.adapterConfig).toMatchObject({ lifecycleMode: expected }); + expect(environment.config).toMatchObject({ runnerLifecycleMode: expected, reuseLease: expected === "warm" }); + expect(cell.suite.expectedMatrixSize).toBe(cell.suite.profiles.length * cell.suite.tasks.length); + } + }); it("keeps verified artifact transfer a single explicit Daytona run", () => { const id = "api-artifact-transfer.runner-codex.daytona.verified-artifact-copy"; const cells = selectRunnerExecutions(parseRunnerSelectors(["--id", id])); @@ -53,10 +72,9 @@ describe("runner E2E catalog", () => { const baseConfig = base.buildEnvironment(environmentInput).config as Record; const warmConfig = daytonaWarmEnvironment.buildEnvironment(environmentInput).config as Record; const cells = runnerMatrix.filter(entry => entry.suite.id === suite.id); - expect(cells).toHaveLength(2); + expect(cells).toHaveLength(1); expect(cells.map(cell => cell.id)).toEqual([ "daytona-restart-continuity.runner-codex.daytona.recover-controller", - "daytona-restart-continuity.runner-acpx-claude.daytona.recover-controller", ]); for (const cell of cells) { const config = cell.environment.buildEnvironment(environmentInput).config as Record; @@ -68,9 +86,8 @@ describe("runner E2E catalog", () => { expect(cell.environment.label).toContain("warm reusable"); const agentInput = { ...environmentInput, environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace" }; const adapterConfig = cell.profile.buildAgent(agentInput).adapterConfig as Record; - const original = runnerProfiles.find(profile => profile.id === cell.profile.id)!.buildAgent(agentInput).adapterConfig as Record; expect(adapterConfig.idleTimeoutMs).toBe(300_000); - expect(adapterConfig.lifecycleMode).toBe(original.lifecycleMode); + expect(adapterConfig.lifecycleMode).toBe("warm"); expect(cell.suite.manualOnly).toBe(true); expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(entry => entry.id)).toEqual([cell.id]); @@ -78,7 +95,7 @@ describe("runner E2E catalog", () => { } expect(baseConfig).toMatchObject({ autoStopInterval: 5, reuseLease: false, runnerLifecycleMode: "per_turn" }); expect(warmConfig).toMatchObject({ autoStopInterval: 5, reuseLease: true, runnerLifecycleMode: "warm" }); - expect(suite.definitionMetadata).toMatchObject({ version: 7, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + expect(suite.definitionMetadata).toMatchObject({ version: 8, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1 }); }); @@ -312,11 +329,11 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 9, 9, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(837); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(837); + expect(validateRunnerCatalog()).toHaveLength(822); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(822); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 18c1151aaf3..e9a8f8ad6af 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,4 +1,5 @@ import { ROLLING_WEB_DEPLOYMENT_SUITE, ROLLING_WEB_DEPLOYMENT_WARM_SUITE } from "./controller-rolling-deployment.js"; +import { supportsManagedNativeWarmSession } from "../../server/src/services/native-runtime/native-warm-capability.js"; import { FEEDBACK_PERSISTENCE_SUITE, feedbackPersistenceTask, feedbackPersistenceDigest } from "./feedback-persistence.js"; import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js"; import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js"; @@ -1205,10 +1206,21 @@ const daytonaControllerOutageEnvironment: EnvironmentFixture = { }, }; +function managedWarmProfiles(profiles: readonly RunnerProfileFixture[]): RunnerProfileFixture[] { + return profiles.filter(profile => { + const backend = profile.provider === "codex" ? "codex_app_server" : profile.provider === "opencode" ? "opencode_server" : "acpx_runtime"; + return supportsManagedNativeWarmSession({ backend, acpxAgent: "acpxAgent" in profile ? profile.acpxAgent : undefined }, + profile.credential === "COPILOT_GITHUB_TOKEN" ? { provider: "github", method: "api_key" } : undefined); + }).map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) { + const agent = profile.buildAgent(input); + return { ...agent, adapterConfig: { ...(agent.adapterConfig as Record), lifecycleMode: "warm" } }; + } })); +} + const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { id: REMOTE_WORKSPACE_STORAGE_SUITE, label: "Remote workspace storage and controller replacement", manualOnly: true, description: "Keep synthetic files and local Git work in the sandbox, commit saved versions to object storage, and preserve the original active execution through controller replacement.", - groups: [], profiles: [(() => { + groups: [], profiles: managedWarmProfiles([(() => { const profile = productionStoryProfile(repositoryModelProfiles[0]!.native); return { ...profile, buildAgent(input: AgentFixtureBuildInput) { const agent = profile.buildAgent(input); @@ -1216,13 +1228,13 @@ const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { delete adapterConfig.cwd; return { ...agent, adapterConfig }; } }; - })()], environments: [daytonaControllerOutageEnvironment], + })()]), environments: [daytonaControllerOutageEnvironment], tasks: everydayTasks.filter(task => task.id === "recover-controller").map(task => ({ ...task, label: "Recover sandbox-owned files after controller replacement", expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const, buildPrompt: (nonce: string) => remoteWorkspaceStoragePrompt(task.buildPrompt(nonce)), })), expectedMatrixSize: 1, - definitionMetadata: { version: 5, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + definitionMetadata: { version: 6, lifecycleCoverage: "warm-reusable", workspaceFileSource: "saved-exact-snapshot-with-retained-location", scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", remoteWorkspaceStorage: true, upgradeReadiness: "active-remote-workspace-recoverable-before-sigkill", @@ -1252,17 +1264,19 @@ const rollingDeploymentProfiles = [...runnerProfiles, ...extendedHarnessProfiles return { ...agent, adapterConfig }; } }; }); +const rollingWarmDeploymentProfiles = managedWarmProfiles(rollingDeploymentProfiles); + const rollingDeploymentSuites: RunnerSuiteFixture[] = [ { id: ROLLING_WEB_DEPLOYMENT_SUITE, environment: runnerEnvironments.find(environment => environment.id === "daytona")! }, { id: ROLLING_WEB_DEPLOYMENT_WARM_SUITE, environment: daytonaControllerOutageEnvironment }, ].map(({ id, environment }) => ({ ...remoteWorkspaceStorageSuite, id, label: `Rolling control-plane web deployment (${environment.configurationKey ? "warm" : "per-turn"})`, description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.", - profiles: rollingDeploymentProfiles, environments: [environment], - expectedMatrixSize: rollingDeploymentProfiles.length * remoteWorkspaceStorageSuite.tasks.length, - definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 4, controllerDowntimeMs: 0, + profiles: id === ROLLING_WEB_DEPLOYMENT_WARM_SUITE ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles, environments: [environment], + expectedMatrixSize: (id === ROLLING_WEB_DEPLOYMENT_WARM_SUITE ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles).length * remoteWorkspaceStorageSuite.tasks.length, + definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 6, controllerDowntimeMs: 0, activeWorkBoundary: "foreground-180-second-cli-reliability-soak-and-live-owner-recheck", - workspaceFileSource: environment.configurationKey ? "live" : "saved-exact-snapshot-after-per-turn-retirement", + workspaceFileSource: environment.configurationKey ? "saved-exact-snapshot-with-retained-location" : "saved-exact-snapshot-after-per-turn-retirement", lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", providerCoverage: "registered-native-daytona-profiles", replacement: "ready-before-traffic-switch-sigterm-delete-entire-home", upgradeReadiness: "active-remote-workspace-recoverable-before-handoff", @@ -1278,11 +1292,11 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ].map(({ id, environment }): RunnerSuiteFixture => ({ id, label: `Legacy native session migration (${environment.configurationKey ? "warm" : "per-turn"})`, manualOnly: true, description: "Resume an existing native session and workspace after the production offline migration and deletion of the original controller home.", - groups: [], profiles: rollingDeploymentProfiles, environments: [environment], + groups: [], profiles: environment.configurationKey ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles, environments: [environment], tasks: [{ ...necessaryQuestionTask, minimumExpectedRunCount: 2, turnTimeoutMs: 6 * 60_000, attemptTimeoutMs: { local: 15 * 60_000, daytona: 20 * 60_000 } }], - expectedMatrixSize: rollingDeploymentProfiles.length, - definitionMetadata: { version: 1, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000, + expectedMatrixSize: (environment.configurationKey ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles).length, + definitionMetadata: { version: 2, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000, lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", migration: "packaged-offline-entry-verified-export-delete-entire-source-home", authentication: "api-key", stagingQualification: false, grading: "same-native-and-provider-session-logical-and-remote-paths-reference-bytes-question-and-final-document" }, @@ -1300,11 +1314,11 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true, description: "Keep the original active native runner, reusable warm sandbox, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.", - groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), + groups: [], profiles: managedWarmProfiles(repositoryModelProfiles.map(({ native }) => productionStoryProfile(native))), environments: [daytonaControllerOutageEnvironment], tasks: everydayTasks.filter(task => task.id === "recover-controller") - .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, - definitionMetadata: { version: 7, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", + .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: managedWarmProfiles(repositoryModelProfiles.map(({ native }) => productionStoryProfile(native))).length, + definitionMetadata: { version: 8, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, }, diff --git a/tests/runner-e2e/controller-outage.ts b/tests/runner-e2e/controller-outage.ts index 32ec972c938..758f80a8c33 100644 --- a/tests/runner-e2e/controller-outage.ts +++ b/tests/runner-e2e/controller-outage.ts @@ -3,6 +3,7 @@ import type { ServingArtifact } from "./upgrade-artifacts.js"; import type { ControllerRecoveryReceipt } from "./controller-recovery-observer.js"; import { setTimeout as delay } from "node:timers/promises"; import type { MatrixExecution } from "./types.js"; +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; export interface ControllerOutage { signal: "SIGTERM" | "SIGKILL"; @@ -40,7 +41,7 @@ export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs } export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined { - if (execution.suite.id === "rolling-web-deployment") return undefined; + if (isRollingWebDeployment(execution)) return undefined; if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller" || execution.environment.id !== "daytona" || execution.profile.generation !== "native" || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined; diff --git a/tests/runner-e2e/controller-rolling-deployment.test.ts b/tests/runner-e2e/controller-rolling-deployment.test.ts index 2484abb0212..7a7e6c5815a 100644 --- a/tests/runner-e2e/controller-rolling-deployment.test.ts +++ b/tests/runner-e2e/controller-rolling-deployment.test.ts @@ -57,7 +57,7 @@ describe("rolling controller qualification", () => { const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"])); expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); }); - it("requires both lifecycle modes for every production-qualified native sandbox profile", () => { + it("covers every qualified native profile per turn and each supported warm profile", () => { const executions = ["rolling-web-deployment", "rolling-web-deployment-warm"] .flatMap(suite => selectRunnerExecutions(parseRunnerSelectors(["--suite", suite]))); const native = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile] @@ -74,13 +74,19 @@ describe("rolling controller qualification", () => { && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")) .map(kind)); expect(new Set(native.map(kind))).toEqual(registeredKinds); + const warmIds = new Set(["runner-codex", "runner-acpx-copilot"]); + expect(new Set(runnerSuites.find(suite => suite.id === "rolling-web-deployment-warm")!.profiles.map(profile => profile.id))).toEqual(warmIds); for (const profile of native) { const cells = executions.filter(execution => execution.profile.id === profile.id); - expect(cells).toHaveLength(2); + expect(cells.filter(cell => cell.suite.id === "rolling-web-deployment")).toHaveLength(1); + expect(cells).toHaveLength(warmIds.has(profile.id) ? 2 : 1); expect(cells.every(cell => !cell.profile.qualificationCandidate)).toBe(true); expect(new Set(cells.map(cell => cell.environment.configurationKey ?? "per_turn"))) - .toEqual(new Set(["per_turn", "warm-controller-outage-v1"])); + .toEqual(new Set(warmIds.has(profile.id) ? ["per_turn", "warm-controller-outage-v1"] : ["per_turn"])); } expect(executions.every(execution => execution.suite.definitionMetadata?.retainedControllerContinuation === true)).toBe(true); + for (const execution of executions) { + expect(controllerOutageForExecution(execution, {})).toBeUndefined(); + } }); }); diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index ccd87e9ef89..f10f675db26 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -186,8 +186,12 @@ export async function runEverydayFlow(input: Input) { const controllerOutage = controllerOutageForExecution(execution, process.env); const requireRemoteRestart = controllerOutage !== undefined || isRollingWebDeployment(execution); const remoteWorkspaceStorage = isRemoteWorkspaceStorageExecution(execution); - if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [], - expectedFileSource: execution.suite.definitionMetadata?.lifecycleCoverage === "per-turn-disposable" ? "saved" : "live" }; + if (remoteWorkspaceStorage) { + const perTurn = execution.suite.definitionMetadata?.lifecycleCoverage === "per-turn-disposable"; + ev.remoteWorkspaceStorage = { browserReadPhases: [], + expectedFileSource: perTurn || execution.suite.definitionMetadata?.lifecycleCoverage === "warm-reusable" ? "saved" : "live", + expectedLocation: perTurn ? "released" : "retained" }; + } const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -564,7 +568,8 @@ export async function runEverydayFlow(input: Input) { label: `remote workspace ${phase} saved version`, deadlineAt: input.deadlineAt, load: () => readRemoteWorkspaceStorageCheckpoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id, - expectedFileSource: ev.remoteWorkspaceStorage!.expectedFileSource }), + expectedFileSource: ev.remoteWorkspaceStorage!.expectedFileSource, + expectedLocation: ev.remoteWorkspaceStorage!.expectedLocation }), accept: observed => Boolean(observed.point), }); ev.remoteWorkspaceStorage![phase] = point.point!; diff --git a/tests/runner-e2e/remote-workspace-storage.test.ts b/tests/runner-e2e/remote-workspace-storage.test.ts index 4fa50e72462..9bc0ad0c03a 100644 --- a/tests/runner-e2e/remote-workspace-storage.test.ts +++ b/tests/runner-e2e/remote-workspace-storage.test.ts @@ -86,6 +86,34 @@ describe("remote workspace Product E2E qualification", () => { expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(false); }); + it("reads a completed warm turn from its saved snapshot while retaining the bound workspace location", async () => { + const f = checkpointApi(); + const savedFile = { ...f.file, resource: { ...f.file.resource, + source: { kind: "saved", snapshotId: f.receipt.snapshotId } } }; + const location = f.workspace.workspace.metadata.remoteWorkspace.location; + const originalGet = f.api.get; + const api = { get: async (route: string): Promise => { + if (route.includes("file-resources/content")) return savedFile as T; + if (route.endsWith("/leases")) return [{ id: location.leaseId, companyId: f.context.companyId, + issueId: f.context.issueId, providerLeaseId: location.providerLeaseId }] as T; + return originalGet(route); + } }; + const input = { ...f.context, api, expectedFileSource: "saved" as const, expectedLocation: "retained" as const }; + expect((await readRemoteWorkspaceStorageCheckpoint(input)).point).toMatchObject({ + fileSource: "saved", fileSnapshotId: f.receipt.snapshotId, locationState: "retained" }); + expect(await readRemoteWorkspaceStorageCheckpoint({ ...input, expectedLocation: "released" })).toEqual({}); + location.environmentId = id("5"); + expect(await readRemoteWorkspaceStorageCheckpoint(input)).toEqual({}); + const evidence = goodEvidence(); + evidence.expectedFileSource = "saved"; + evidence.expectedLocation = "retained"; + for (const point of [evidence.initial!, evidence.final!]) Object.assign(point, { + fileSource: "saved", fileSnapshotId: point.snapshotId, locationState: "retained" }); + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(true); + evidence.final!.locationState = "released"; + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(false); + }); + it.each(["run-id", "company", "agent", "native-issue", "runtime", "status", "schema", "workspace", "epoch", "snapshot", "save-state", "lease", "provider", "remote-root"] as const)( "rejects a full-run %s mismatch even when its public summary succeeds", async mismatch => { @@ -139,7 +167,7 @@ describe("remote workspace Product E2E qualification", () => { expect(controllerOutageForExecution(cells[0]!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(() => controllerOutageForExecution(cells[0]!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })).toThrow("shorter"); expect(cells[0]!.task.expectedRunCount).toBe(4); - expect(cells[0]!.suite.definitionMetadata).toMatchObject({ version: 5, savedRunReadback: "public-runId-summary-then-bound-native-detail", + expect(cells[0]!.suite.definitionMetadata).toMatchObject({ version: 6, lifecycleCoverage: "warm-reusable", workspaceFileSource: "saved-exact-snapshot-with-retained-location", savedRunReadback: "public-runId-summary-then-bound-native-detail", fileViewerSetup: "ordinary-setting-enabled-with-storage-default-preserved", retainedControllerContinuation: true, postControllerReplyFormat: "one-plain-text-line-marker-space-actual-command-output", retainedControllerFollowupTransport: "task_comment_api", autoStopIntervalMinutes: 15 }); @@ -151,6 +179,7 @@ describe("remote workspace Product E2E qualification", () => { const agent = cells[0]!.profile.buildAgent({ environmentId: id("9"), environmentFixtureId: "daytona", workspacePath: "/unavailable-host-folder", secretRefs: { OPENAI_API_KEY: { type: "secret_ref", secretId: id("8"), version: "latest" } }, executionId: "synthetic" }); expect((agent.adapterConfig as Record).cwd).toBeUndefined(); + expect(agent.adapterConfig).toMatchObject({ lifecycleMode: "warm" }); expect(cells[0]!.task.buildPrompt("synthetic")).toContain("fresh random 32-character lowercase hexadecimal reference"); }); diff --git a/tests/runner-e2e/remote-workspace-storage.ts b/tests/runner-e2e/remote-workspace-storage.ts index 3f17cb67ee8..2e2924a10c6 100644 --- a/tests/runner-e2e/remote-workspace-storage.ts +++ b/tests/runner-e2e/remote-workspace-storage.ts @@ -25,10 +25,12 @@ export interface RemoteWorkspaceStorageCheckpoint { proofSha256: string; fileSource: string; fileSnapshotId?: string; + locationState?: "retained" | "released"; } export interface RemoteWorkspaceStorageEvidence { expectedFileSource?: "live" | "saved"; + expectedLocation?: "retained" | "released"; initial?: RemoteWorkspaceStorageCheckpoint; final?: RemoteWorkspaceStorageCheckpoint; browserReadPhases: string[]; @@ -51,6 +53,7 @@ export function remoteWorkspaceProofDigest(content: string): string { export async function readRemoteWorkspaceStorageCheckpoint(input: { api: Pick; companyId: string; agentId: string; issueId: string; environmentId: string; expectedFileSource?: "live" | "saved"; + expectedLocation?: "retained" | "released"; }): Promise<{ point?: RemoteWorkspaceStorageCheckpoint; content?: string }> { const view = await input.api.get<{ issueId: string; workspace: ExecutionWorkspace | null }>(`/api/issues/${input.issueId}/workspace`); const workspace = view.workspace; @@ -59,9 +62,10 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { || workspace.providerType !== "cloud_sandbox" || !authorityResult.success) return {}; const authority = authorityResult.data; const expectedSource = input.expectedFileSource ?? "live"; + const expectedLocation = input.expectedLocation ?? (expectedSource === "saved" ? "released" : "retained"); if (!authority.snapshotId || authority.dirty || authority.writerRunId) return {}; - if (expectedSource === "live" && (!authority.location || authority.location.environmentId !== input.environmentId)) return {}; - if (expectedSource === "saved" && authority.location !== null) return {}; + if (expectedLocation === "retained" && (!authority.location || authority.location.environmentId !== input.environmentId)) return {}; + if (expectedLocation === "released" && authority.location !== null) return {}; const runs = await input.api.get(`/api/issues/${input.issueId}/runs`); const latest = runs.filter(run => run.runtimeMode === "native" && run.status === "succeeded" && run.agentId === input.agentId && typeof run.runId === "string") @@ -95,6 +99,7 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { authoritySchema: authority.schema, authorityDirty: authority.dirty, authorityWriter: authority.writerRunId, providerType: workspace.providerType, proofSha256: remoteWorkspaceProofDigest(file.content.data), fileSource: file.resource?.source?.kind, fileSnapshotId, + locationState: authority.location ? "retained" : "released", } }; } @@ -106,6 +111,7 @@ export function gradeRemoteWorkspaceStorage(evidence: RemoteWorkspaceStorageEvid && point.authoritySchema === "paperclip.remote-workspace.v1" && point.authorityDirty === false && point.authorityWriter === null && point.providerType === "cloud_sandbox" && point.fileSource === (evidence.expectedFileSource ?? "live") + && (!evidence.expectedLocation || point.locationState === evidence.expectedLocation) && (point.fileSource !== "saved" || point.fileSnapshotId === point.snapshotId) && /^[a-f0-9]{64}$/.test(point.proofSha256)); return [ diff --git a/tests/runner-e2e/server-restart-request.test.ts b/tests/runner-e2e/server-restart-request.test.ts index cf510935884..bfec299ead9 100644 --- a/tests/runner-e2e/server-restart-request.test.ts +++ b/tests/runner-e2e/server-restart-request.test.ts @@ -19,13 +19,27 @@ afterEach(async () => { await rm(root, { recursive: true, force: true }); }); -const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite => - ["codex", "acpx"].map(provider => { - const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller" - && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider); - if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`); - return execution; - })); +it.each(["rolling-web-deployment", "rolling-web-deployment-warm"])("accepts the assembled %s restart request", async suite => { + const execution = runnerMatrix.find(row => row.suite.id === suite && row.profile.id === "runner-codex")!; + const recoveryBinding = { companyId: "11111111-1111-4111-8111-111111111111", + issueId: "22222222-2222-4222-8222-222222222222", runId: "33333333-3333-4333-8333-333333333333" }; + await writeFile(requestPath, JSON.stringify({ requestId: "rolling-request", rollingDeployment: true, + discardControllerCache: true, controllerOutage: controllerOutageForExecution(execution, {}), recoveryBinding })); + const request = await readRestartRequest(requestPath, [execution.id]); + expect(request).toMatchObject({ rollingDeployment: true, discardControllerCache: true, recoveryBinding }); + expect(request).not.toHaveProperty("controllerOutage"); +}); + +const cells = [ + { suite: "everyday-workflows", provider: "codex" }, + { suite: "everyday-workflows", provider: "acpx" }, + { suite: "daytona-restart-continuity", provider: "codex" }, +].map(({ suite, provider }) => { + const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller" + && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider); + if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`); + return execution; +}); it.each(cells)("accepts the real supervisor request file for $id", async execution => { const outage = controllerOutageForExecution(execution, {})!;