diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index bd446143639..8c0243c4fa9 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -63,8 +63,11 @@ chat. Other YAML entries remain planning inventory; registration is not live qualification. The explicit-only `daytona-restart-continuity` suite runs the existing -`recover-controller` journey with the recorded Codex and Claude staging models -on fresh reusable warm Daytona fixtures. The sandbox, Runner and provider history +`recover-controller` journey with the recorded Codex staging model +on a fresh reusable warm Daytona fixture. Warm controller-replacement and legacy +migration suites use the production managed warm-session capability to select +profiles and set both the agent and environment policy to warm. Claude, OpenCode +and Grok remain covered by the per-turn rolling and migration lanes. The sandbox, Runner and provider history must survive both controller replacements. After a completed task tool in an active revision turn, it keeps the owned development controller offline for 300 seconds. This exceeds the 60-second ownership lease and remains within the Runner's 420-second @@ -83,7 +86,7 @@ ownership/reconnect limits, archive/delete backstops and owned cleanup remain un The retained continuation accepts whitespace between its exact marker and the exact `queue_r` output token; readiness and final grading use the same matcher. -Definition version 7 uses the existing warm reusable environment with a dedicated +Definition version 8 uses an explicitly warm agent and the warm reusable environment with a dedicated configuration key. Only the post-completion replacement follow-up uses the normal board-authenticated task-comment API, with one persisted user comment and the unchanged follow-up request. Initial creation, active outage and queued continuation @@ -104,9 +107,10 @@ its DB checkpoint alone cannot replace those bytes. Per-turn application qualification under this disk loss awaits the provider-backup owner's durable storage contract; native controller recovery does not waive that dependency. -`rolling-web-deployment` definition 4 selects every registered native Daytona +`rolling-web-deployment` definition 6 selects every registered native Daytona profile in disposable per-turn environments; `rolling-web-deployment-warm` -uses the same profiles in reusable warm environments. Their +selects providers supported by the production managed warm-session capability +and explicitly sets both agent and environment policy to warm. Their provider-backup path uses encrypted object storage and a fenced database ledger. The earlier per-turn failure remains failed evidence; this definition requires a fresh passing run, including the strict post-replacement continuation oracle. @@ -115,7 +119,9 @@ the replacement time to become ready while useful agent work remains active. The observer rechecks the original live owner immediately before switching traffic and fails if that turn already settled. A completed turn cannot count as active takeover. Per-turn file inspection requires the exact committed saved -snapshot; the warm lane still requires the live workspace. +snapshot. Completed warm turns also read that exact saved snapshot, while their +workspace location must remain bound to the retained sandbox. The API's saved +file response is independent of whether the sandbox is retained or retired. Cold restores retain the published workspace directory name, including after the old allocation is destroyed, because provider recovery binds the working directory into session identity. Unpublished preparations retain distinct @@ -126,7 +132,7 @@ The ordinary Everyday matrix retains its 125-second active outage default. The two explicit continuity suites require 300 seconds and reject a shorter override. The explicit-only `remote-workspace-storage` suite reuses that controller-restart -journey for one native Codex Daytona cell. It verifies the default-enabled remote +journey for one explicitly warm native Codex Daytona cell. It verifies the default-enabled remote workspace setting through the ordinary instance API and creates a projectless task. It enables the existing experimental Files viewer through that API. The agent creates only synthetic source, an unpushed Git commit, and an untracked random @@ -141,7 +147,8 @@ after completion, a second replacement must admit a useful fourth native turn through the ordinary task-comment API with the same native and provider session. That request specifies one plain-text line containing its marker, one space, and the command's actual output; the expected output stays private to the evaluator. -The browser then reads the saved reference through Files again. This is same-task +The browser then reads the saved reference through Files again, with the sandbox +location still retained. This is same-task continuation, not a separate-task sharing test. The ordinary original-process/lease, receipt-prefix, queued-message, artifact, and strict retained-session oracles still apply. @@ -191,6 +198,9 @@ Postgres. Each controller has an independent compiled build and cache. The copie app retains the checkout's `cli/` and `packages/` dependency links, including the development Runner vendor entry. Shared workspace dependencies mean this is not released-container or Railway evidence. +The warm lane selects providers through the production managed warm-session +capability and explicitly sets the agent policy to warm. The per-turn lane +retains every registered provider's per-turn policy. The read-only recovery observer captures the live predecessor's identity just before handoff, then requires a new controller generation, authenticated Runner attachment, and new canonical events for the original execution. No fixture @@ -3519,7 +3529,9 @@ change credentials, or change evaluation and evidence limits. # Legacy workspace migration `legacy-workspace-migration` and `legacy-workspace-migration-warm` are explicit, -paid, two-turn Daytona workflows for registered native production profiles. They +paid, two-turn Daytona workflows. The per-turn lane covers every registered native +production profile; the warm lane selects supported managed warm providers and +sets the agent policy explicitly to warm. They create a private reference file and ask a question on a local-disk controller. The supervisor stops that controller, runs the packaged offline migration with owned PostgreSQL and S3, verifies the export, starts an empty-home controller, diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 044865f6cae..969b43626cc 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -33,6 +33,25 @@ import { } from "./selectors.js"; describe("runner E2E catalog", () => { + it("uses effective supported warm agent policy across controller replacement and migration suites", () => { + const warmSuites = ["rolling-web-deployment-warm", "legacy-workspace-migration-warm", "remote-workspace-storage", "daytona-restart-continuity"]; + const warm = runnerMatrix.filter(cell => warmSuites.includes(cell.suite.id)); + for (const id of warmSuites) expect(warm.some(cell => cell.suite.id === id && cell.profile.id === "runner-codex"), id).toBe(true); + const perTurn = runnerMatrix.filter(cell => cell.suite.id === "rolling-web-deployment"); + expect(warm.some(cell => ["runner-opencode", "runner-acpx-claude", "runner-acpx-grok"].includes(cell.profile.id))).toBe(false); + expect(perTurn.length).toBeGreaterThan(runnerMatrix.filter(cell => cell.suite.id === "rolling-web-deployment-warm").length); + for (const cell of [...warm, ...perTurn]) { + const secretRefs = Object.fromEntries([cell.profile.credential, "DAYTONA_API_KEY"].map(key => [key, + { type: "secret_ref" as const, secretId: "22222222-2222-4222-8222-222222222222", version: "latest" as const }])); + const input = { executionId: cell.id, secretRefs, daytonaImage: `fixture@sha256:${"a".repeat(64)}` }; + const agent = cell.profile.buildAgent({ ...input, environmentId: "env-1", environmentFixtureId: "daytona", workspacePath: "/workspace" }); + const environment = cell.environment.buildEnvironment(input); + const expected = warmSuites.includes(cell.suite.id) ? "warm" : "per_turn"; + expect(agent.adapterConfig).toMatchObject({ lifecycleMode: expected }); + expect(environment.config).toMatchObject({ runnerLifecycleMode: expected, reuseLease: expected === "warm" }); + expect(cell.suite.expectedMatrixSize).toBe(cell.suite.profiles.length * cell.suite.tasks.length); + } + }); it("keeps verified artifact transfer a single explicit Daytona run", () => { const id = "api-artifact-transfer.runner-codex.daytona.verified-artifact-copy"; const cells = selectRunnerExecutions(parseRunnerSelectors(["--id", id])); @@ -53,10 +72,9 @@ describe("runner E2E catalog", () => { const baseConfig = base.buildEnvironment(environmentInput).config as Record; const warmConfig = daytonaWarmEnvironment.buildEnvironment(environmentInput).config as Record; const cells = runnerMatrix.filter(entry => entry.suite.id === suite.id); - expect(cells).toHaveLength(2); + expect(cells).toHaveLength(1); expect(cells.map(cell => cell.id)).toEqual([ "daytona-restart-continuity.runner-codex.daytona.recover-controller", - "daytona-restart-continuity.runner-acpx-claude.daytona.recover-controller", ]); for (const cell of cells) { const config = cell.environment.buildEnvironment(environmentInput).config as Record; @@ -68,9 +86,8 @@ describe("runner E2E catalog", () => { expect(cell.environment.label).toContain("warm reusable"); const agentInput = { ...environmentInput, environmentId: "env-1", environmentFixtureId: "daytona" as const, workspacePath: "/workspace" }; const adapterConfig = cell.profile.buildAgent(agentInput).adapterConfig as Record; - const original = runnerProfiles.find(profile => profile.id === cell.profile.id)!.buildAgent(agentInput).adapterConfig as Record; expect(adapterConfig.idleTimeoutMs).toBe(300_000); - expect(adapterConfig.lifecycleMode).toBe(original.lifecycleMode); + expect(adapterConfig.lifecycleMode).toBe("warm"); expect(cell.suite.manualOnly).toBe(true); expect(cell.task.automaticRetryPolicy).toBe("single_attempt"); expect(selectRunnerExecutions(parseRunnerSelectors(["--id", cell.id])).map(entry => entry.id)).toEqual([cell.id]); @@ -78,7 +95,7 @@ describe("runner E2E catalog", () => { } expect(baseConfig).toMatchObject({ autoStopInterval: 5, reuseLease: false, runnerLifecycleMode: "per_turn" }); expect(warmConfig).toMatchObject({ autoStopInterval: 5, reuseLease: true, runnerLifecycleMode: "warm" }); - expect(suite.definitionMetadata).toMatchObject({ version: 7, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + expect(suite.definitionMetadata).toMatchObject({ version: 8, controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1 }); }); @@ -312,11 +329,11 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 9, 9, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 9, 2, 9, 2, 4, 1, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(837); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(837); + expect(validateRunnerCatalog()).toHaveLength(822); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(822); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index 18c1151aaf3..e9a8f8ad6af 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1,4 +1,5 @@ import { ROLLING_WEB_DEPLOYMENT_SUITE, ROLLING_WEB_DEPLOYMENT_WARM_SUITE } from "./controller-rolling-deployment.js"; +import { supportsManagedNativeWarmSession } from "../../server/src/services/native-runtime/native-warm-capability.js"; import { FEEDBACK_PERSISTENCE_SUITE, feedbackPersistenceTask, feedbackPersistenceDigest } from "./feedback-persistence.js"; import { followupStopTask, followupStopDigest } from "./followup-stop-resume.js"; import { finalizationRecoveryTask, finalizationRecoveryDigest } from "./finalization-recovery.js"; @@ -1205,10 +1206,21 @@ const daytonaControllerOutageEnvironment: EnvironmentFixture = { }, }; +function managedWarmProfiles(profiles: readonly RunnerProfileFixture[]): RunnerProfileFixture[] { + return profiles.filter(profile => { + const backend = profile.provider === "codex" ? "codex_app_server" : profile.provider === "opencode" ? "opencode_server" : "acpx_runtime"; + return supportsManagedNativeWarmSession({ backend, acpxAgent: "acpxAgent" in profile ? profile.acpxAgent : undefined }, + profile.credential === "COPILOT_GITHUB_TOKEN" ? { provider: "github", method: "api_key" } : undefined); + }).map(profile => ({ ...profile, buildAgent(input: AgentFixtureBuildInput) { + const agent = profile.buildAgent(input); + return { ...agent, adapterConfig: { ...(agent.adapterConfig as Record), lifecycleMode: "warm" } }; + } })); +} + const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { id: REMOTE_WORKSPACE_STORAGE_SUITE, label: "Remote workspace storage and controller replacement", manualOnly: true, description: "Keep synthetic files and local Git work in the sandbox, commit saved versions to object storage, and preserve the original active execution through controller replacement.", - groups: [], profiles: [(() => { + groups: [], profiles: managedWarmProfiles([(() => { const profile = productionStoryProfile(repositoryModelProfiles[0]!.native); return { ...profile, buildAgent(input: AgentFixtureBuildInput) { const agent = profile.buildAgent(input); @@ -1216,13 +1228,13 @@ const remoteWorkspaceStorageSuite: RunnerSuiteFixture = { delete adapterConfig.cwd; return { ...agent, adapterConfig }; } }; - })()], environments: [daytonaControllerOutageEnvironment], + })()]), environments: [daytonaControllerOutageEnvironment], tasks: everydayTasks.filter(task => task.id === "recover-controller").map(task => ({ ...task, label: "Recover sandbox-owned files after controller replacement", expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const, buildPrompt: (nonce: string) => remoteWorkspaceStoragePrompt(task.buildPrompt(nonce)), })), expectedMatrixSize: 1, - definitionMetadata: { version: 5, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, + definitionMetadata: { version: 6, lifecycleCoverage: "warm-reusable", workspaceFileSource: "saved-exact-snapshot-with-retained-location", scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", remoteWorkspaceStorage: true, upgradeReadiness: "active-remote-workspace-recoverable-before-sigkill", @@ -1252,17 +1264,19 @@ const rollingDeploymentProfiles = [...runnerProfiles, ...extendedHarnessProfiles return { ...agent, adapterConfig }; } }; }); +const rollingWarmDeploymentProfiles = managedWarmProfiles(rollingDeploymentProfiles); + const rollingDeploymentSuites: RunnerSuiteFixture[] = [ { id: ROLLING_WEB_DEPLOYMENT_SUITE, environment: runnerEnvironments.find(environment => environment.id === "daytona")! }, { id: ROLLING_WEB_DEPLOYMENT_WARM_SUITE, environment: daytonaControllerOutageEnvironment }, ].map(({ id, environment }) => ({ ...remoteWorkspaceStorageSuite, id, label: `Rolling control-plane web deployment (${environment.configurationKey ? "warm" : "per-turn"})`, description: "Overlap independent ephemeral web servers while a remote execution stays active; keep ordinary reads, saves, files and a later native continuation working.", - profiles: rollingDeploymentProfiles, environments: [environment], - expectedMatrixSize: rollingDeploymentProfiles.length * remoteWorkspaceStorageSuite.tasks.length, - definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 4, controllerDowntimeMs: 0, + profiles: id === ROLLING_WEB_DEPLOYMENT_WARM_SUITE ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles, environments: [environment], + expectedMatrixSize: (id === ROLLING_WEB_DEPLOYMENT_WARM_SUITE ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles).length * remoteWorkspaceStorageSuite.tasks.length, + definitionMetadata: { ...remoteWorkspaceStorageSuite.definitionMetadata, version: 6, controllerDowntimeMs: 0, activeWorkBoundary: "foreground-180-second-cli-reliability-soak-and-live-owner-recheck", - workspaceFileSource: environment.configurationKey ? "live" : "saved-exact-snapshot-after-per-turn-retirement", + workspaceFileSource: environment.configurationKey ? "saved-exact-snapshot-with-retained-location" : "saved-exact-snapshot-after-per-turn-retirement", lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", providerCoverage: "registered-native-daytona-profiles", replacement: "ready-before-traffic-switch-sigterm-delete-entire-home", upgradeReadiness: "active-remote-workspace-recoverable-before-handoff", @@ -1278,11 +1292,11 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ ].map(({ id, environment }): RunnerSuiteFixture => ({ id, label: `Legacy native session migration (${environment.configurationKey ? "warm" : "per-turn"})`, manualOnly: true, description: "Resume an existing native session and workspace after the production offline migration and deletion of the original controller home.", - groups: [], profiles: rollingDeploymentProfiles, environments: [environment], + groups: [], profiles: environment.configurationKey ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles, environments: [environment], tasks: [{ ...necessaryQuestionTask, minimumExpectedRunCount: 2, turnTimeoutMs: 6 * 60_000, attemptTimeoutMs: { local: 15 * 60_000, daytona: 20 * 60_000 } }], - expectedMatrixSize: rollingDeploymentProfiles.length, - definitionMetadata: { version: 1, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000, + expectedMatrixSize: (environment.configurationKey ? rollingWarmDeploymentProfiles : rollingDeploymentProfiles).length, + definitionMetadata: { version: 2, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000, lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", migration: "packaged-offline-entry-verified-export-delete-entire-source-home", authentication: "api-key", stagingQualification: false, grading: "same-native-and-provider-session-logical-and-remote-paths-reference-bytes-question-and-final-document" }, @@ -1300,11 +1314,11 @@ export const runnerSuites: readonly RunnerSuiteFixture[] = [ { id: "daytona-restart-continuity", label: "Daytona warm controller outage continuity", manualOnly: true, description: "Keep the original active native runner, reusable warm sandbox, task tools and saved files through five minutes without a controller, then continue the completed session after another replacement.", - groups: [], profiles: repositoryModelProfiles.map(({ native }) => productionStoryProfile(native)), + groups: [], profiles: managedWarmProfiles(repositoryModelProfiles.map(({ native }) => productionStoryProfile(native))), environments: [daytonaControllerOutageEnvironment], tasks: everydayTasks.filter(task => task.id === "recover-controller") - .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: 2, - definitionMetadata: { version: 7, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", + .map(task => ({ ...task, expectedRunCount: task.expectedRunCount + 1, automaticRetryPolicy: "single_attempt" as const })), expectedMatrixSize: managedWarmProfiles(repositoryModelProfiles.map(({ native }) => productionStoryProfile(native))).length, + definitionMetadata: { version: 8, scheduling: "explicit-only", controllerDowntimeMs: 300_000, autoStopIntervalMinutes: 15, retainedControllerContinuation: true, retainedControllerFollowupTransport: "task_comment_api", maximumAttemptsPerCell: 1, budgetMonthlyCents: 1000, instructions: "production", grading: "same-active-process-lease-native-prefix-and-workspace-receipt", authentication: "api-key", stagingAuthentication: "subscription", stagingQualification: false }, }, diff --git a/tests/runner-e2e/controller-outage.ts b/tests/runner-e2e/controller-outage.ts index 32ec972c938..758f80a8c33 100644 --- a/tests/runner-e2e/controller-outage.ts +++ b/tests/runner-e2e/controller-outage.ts @@ -3,6 +3,7 @@ import type { ServingArtifact } from "./upgrade-artifacts.js"; import type { ControllerRecoveryReceipt } from "./controller-recovery-observer.js"; import { setTimeout as delay } from "node:timers/promises"; import type { MatrixExecution } from "./types.js"; +import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; export interface ControllerOutage { signal: "SIGTERM" | "SIGKILL"; @@ -40,7 +41,7 @@ export function configuredControllerOutage(env: NodeJS.ProcessEnv, defaultHoldMs } export function controllerOutageForExecution(execution: MatrixExecution, env: NodeJS.ProcessEnv): ControllerOutage | undefined { - if (execution.suite.id === "rolling-web-deployment") return undefined; + if (isRollingWebDeployment(execution)) return undefined; if (execution.task.flow !== "everyday_workflow" || execution.task.id !== "recover-controller" || execution.environment.id !== "daytona" || execution.profile.generation !== "native" || (execution.profile.provider !== "codex" && execution.profile.provider !== "acpx")) return undefined; diff --git a/tests/runner-e2e/controller-rolling-deployment.test.ts b/tests/runner-e2e/controller-rolling-deployment.test.ts index 2484abb0212..7a7e6c5815a 100644 --- a/tests/runner-e2e/controller-rolling-deployment.test.ts +++ b/tests/runner-e2e/controller-rolling-deployment.test.ts @@ -57,7 +57,7 @@ describe("rolling controller qualification", () => { const [outage] = selectRunnerExecutions(parseRunnerSelectors(["--suite", "remote-workspace-storage"])); expect(controllerOutageForExecution(outage!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); }); - it("requires both lifecycle modes for every production-qualified native sandbox profile", () => { + it("covers every qualified native profile per turn and each supported warm profile", () => { const executions = ["rolling-web-deployment", "rolling-web-deployment-warm"] .flatMap(suite => selectRunnerExecutions(parseRunnerSelectors(["--suite", suite]))); const native = [...runnerProfiles, ...extendedHarnessProfiles, kimiRunnerProfile] @@ -74,13 +74,19 @@ describe("rolling controller qualification", () => { && (!profile.qualificationCandidate || QUALIFIED_ACPX_PROFILES[profile.qualificationCandidate].qualificationStatus !== "pending")) .map(kind)); expect(new Set(native.map(kind))).toEqual(registeredKinds); + const warmIds = new Set(["runner-codex", "runner-acpx-copilot"]); + expect(new Set(runnerSuites.find(suite => suite.id === "rolling-web-deployment-warm")!.profiles.map(profile => profile.id))).toEqual(warmIds); for (const profile of native) { const cells = executions.filter(execution => execution.profile.id === profile.id); - expect(cells).toHaveLength(2); + expect(cells.filter(cell => cell.suite.id === "rolling-web-deployment")).toHaveLength(1); + expect(cells).toHaveLength(warmIds.has(profile.id) ? 2 : 1); expect(cells.every(cell => !cell.profile.qualificationCandidate)).toBe(true); expect(new Set(cells.map(cell => cell.environment.configurationKey ?? "per_turn"))) - .toEqual(new Set(["per_turn", "warm-controller-outage-v1"])); + .toEqual(new Set(warmIds.has(profile.id) ? ["per_turn", "warm-controller-outage-v1"] : ["per_turn"])); } expect(executions.every(execution => execution.suite.definitionMetadata?.retainedControllerContinuation === true)).toBe(true); + for (const execution of executions) { + expect(controllerOutageForExecution(execution, {})).toBeUndefined(); + } }); }); diff --git a/tests/runner-e2e/everyday-flow.ts b/tests/runner-e2e/everyday-flow.ts index ccd87e9ef89..f10f675db26 100644 --- a/tests/runner-e2e/everyday-flow.ts +++ b/tests/runner-e2e/everyday-flow.ts @@ -186,8 +186,12 @@ export async function runEverydayFlow(input: Input) { const controllerOutage = controllerOutageForExecution(execution, process.env); const requireRemoteRestart = controllerOutage !== undefined || isRollingWebDeployment(execution); const remoteWorkspaceStorage = isRemoteWorkspaceStorageExecution(execution); - if (remoteWorkspaceStorage) ev.remoteWorkspaceStorage = { browserReadPhases: [], - expectedFileSource: execution.suite.definitionMetadata?.lifecycleCoverage === "per-turn-disposable" ? "saved" : "live" }; + if (remoteWorkspaceStorage) { + const perTurn = execution.suite.definitionMetadata?.lifecycleCoverage === "per-turn-disposable"; + ev.remoteWorkspaceStorage = { browserReadPhases: [], + expectedFileSource: perTurn || execution.suite.definitionMetadata?.lifecycleCoverage === "warm-reusable" ? "saved" : "live", + expectedLocation: perTurn ? "released" : "retained" }; + } const providerChoice = caseId === "provider-decline" || caseId === "provider-second"; const nativeProviderCase = caseId === "provider-native"; let aggregatorFixture: Awaited> | undefined; @@ -564,7 +568,8 @@ export async function runEverydayFlow(input: Input) { label: `remote workspace ${phase} saved version`, deadlineAt: input.deadlineAt, load: () => readRemoteWorkspaceStorageCheckpoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: parent!.id, environmentId: fixtures.environment.id, - expectedFileSource: ev.remoteWorkspaceStorage!.expectedFileSource }), + expectedFileSource: ev.remoteWorkspaceStorage!.expectedFileSource, + expectedLocation: ev.remoteWorkspaceStorage!.expectedLocation }), accept: observed => Boolean(observed.point), }); ev.remoteWorkspaceStorage![phase] = point.point!; diff --git a/tests/runner-e2e/remote-workspace-storage.test.ts b/tests/runner-e2e/remote-workspace-storage.test.ts index 4fa50e72462..9bc0ad0c03a 100644 --- a/tests/runner-e2e/remote-workspace-storage.test.ts +++ b/tests/runner-e2e/remote-workspace-storage.test.ts @@ -86,6 +86,34 @@ describe("remote workspace Product E2E qualification", () => { expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(false); }); + it("reads a completed warm turn from its saved snapshot while retaining the bound workspace location", async () => { + const f = checkpointApi(); + const savedFile = { ...f.file, resource: { ...f.file.resource, + source: { kind: "saved", snapshotId: f.receipt.snapshotId } } }; + const location = f.workspace.workspace.metadata.remoteWorkspace.location; + const originalGet = f.api.get; + const api = { get: async (route: string): Promise => { + if (route.includes("file-resources/content")) return savedFile as T; + if (route.endsWith("/leases")) return [{ id: location.leaseId, companyId: f.context.companyId, + issueId: f.context.issueId, providerLeaseId: location.providerLeaseId }] as T; + return originalGet(route); + } }; + const input = { ...f.context, api, expectedFileSource: "saved" as const, expectedLocation: "retained" as const }; + expect((await readRemoteWorkspaceStorageCheckpoint(input)).point).toMatchObject({ + fileSource: "saved", fileSnapshotId: f.receipt.snapshotId, locationState: "retained" }); + expect(await readRemoteWorkspaceStorageCheckpoint({ ...input, expectedLocation: "released" })).toEqual({}); + location.environmentId = id("5"); + expect(await readRemoteWorkspaceStorageCheckpoint(input)).toEqual({}); + const evidence = goodEvidence(); + evidence.expectedFileSource = "saved"; + evidence.expectedLocation = "retained"; + for (const point of [evidence.initial!, evidence.final!]) Object.assign(point, { + fileSource: "saved", fileSnapshotId: point.snapshotId, locationState: "retained" }); + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(true); + evidence.final!.locationState = "released"; + expect(gradeRemoteWorkspaceStorage(evidence).every(check => check.passed)).toBe(false); + }); + it.each(["run-id", "company", "agent", "native-issue", "runtime", "status", "schema", "workspace", "epoch", "snapshot", "save-state", "lease", "provider", "remote-root"] as const)( "rejects a full-run %s mismatch even when its public summary succeeds", async mismatch => { @@ -139,7 +167,7 @@ describe("remote workspace Product E2E qualification", () => { expect(controllerOutageForExecution(cells[0]!, {})).toEqual({ signal: "SIGKILL", holdMs: 300_000 }); expect(() => controllerOutageForExecution(cells[0]!, { PAPERCLIP_RUNNER_E2E_CONTROLLER_OUTAGE_MS: "1000" })).toThrow("shorter"); expect(cells[0]!.task.expectedRunCount).toBe(4); - expect(cells[0]!.suite.definitionMetadata).toMatchObject({ version: 5, savedRunReadback: "public-runId-summary-then-bound-native-detail", + expect(cells[0]!.suite.definitionMetadata).toMatchObject({ version: 6, lifecycleCoverage: "warm-reusable", workspaceFileSource: "saved-exact-snapshot-with-retained-location", savedRunReadback: "public-runId-summary-then-bound-native-detail", fileViewerSetup: "ordinary-setting-enabled-with-storage-default-preserved", retainedControllerContinuation: true, postControllerReplyFormat: "one-plain-text-line-marker-space-actual-command-output", retainedControllerFollowupTransport: "task_comment_api", autoStopIntervalMinutes: 15 }); @@ -151,6 +179,7 @@ describe("remote workspace Product E2E qualification", () => { const agent = cells[0]!.profile.buildAgent({ environmentId: id("9"), environmentFixtureId: "daytona", workspacePath: "/unavailable-host-folder", secretRefs: { OPENAI_API_KEY: { type: "secret_ref", secretId: id("8"), version: "latest" } }, executionId: "synthetic" }); expect((agent.adapterConfig as Record).cwd).toBeUndefined(); + expect(agent.adapterConfig).toMatchObject({ lifecycleMode: "warm" }); expect(cells[0]!.task.buildPrompt("synthetic")).toContain("fresh random 32-character lowercase hexadecimal reference"); }); diff --git a/tests/runner-e2e/remote-workspace-storage.ts b/tests/runner-e2e/remote-workspace-storage.ts index 3f17cb67ee8..2e2924a10c6 100644 --- a/tests/runner-e2e/remote-workspace-storage.ts +++ b/tests/runner-e2e/remote-workspace-storage.ts @@ -25,10 +25,12 @@ export interface RemoteWorkspaceStorageCheckpoint { proofSha256: string; fileSource: string; fileSnapshotId?: string; + locationState?: "retained" | "released"; } export interface RemoteWorkspaceStorageEvidence { expectedFileSource?: "live" | "saved"; + expectedLocation?: "retained" | "released"; initial?: RemoteWorkspaceStorageCheckpoint; final?: RemoteWorkspaceStorageCheckpoint; browserReadPhases: string[]; @@ -51,6 +53,7 @@ export function remoteWorkspaceProofDigest(content: string): string { export async function readRemoteWorkspaceStorageCheckpoint(input: { api: Pick; companyId: string; agentId: string; issueId: string; environmentId: string; expectedFileSource?: "live" | "saved"; + expectedLocation?: "retained" | "released"; }): Promise<{ point?: RemoteWorkspaceStorageCheckpoint; content?: string }> { const view = await input.api.get<{ issueId: string; workspace: ExecutionWorkspace | null }>(`/api/issues/${input.issueId}/workspace`); const workspace = view.workspace; @@ -59,9 +62,10 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { || workspace.providerType !== "cloud_sandbox" || !authorityResult.success) return {}; const authority = authorityResult.data; const expectedSource = input.expectedFileSource ?? "live"; + const expectedLocation = input.expectedLocation ?? (expectedSource === "saved" ? "released" : "retained"); if (!authority.snapshotId || authority.dirty || authority.writerRunId) return {}; - if (expectedSource === "live" && (!authority.location || authority.location.environmentId !== input.environmentId)) return {}; - if (expectedSource === "saved" && authority.location !== null) return {}; + if (expectedLocation === "retained" && (!authority.location || authority.location.environmentId !== input.environmentId)) return {}; + if (expectedLocation === "released" && authority.location !== null) return {}; const runs = await input.api.get(`/api/issues/${input.issueId}/runs`); const latest = runs.filter(run => run.runtimeMode === "native" && run.status === "succeeded" && run.agentId === input.agentId && typeof run.runId === "string") @@ -95,6 +99,7 @@ export async function readRemoteWorkspaceStorageCheckpoint(input: { authoritySchema: authority.schema, authorityDirty: authority.dirty, authorityWriter: authority.writerRunId, providerType: workspace.providerType, proofSha256: remoteWorkspaceProofDigest(file.content.data), fileSource: file.resource?.source?.kind, fileSnapshotId, + locationState: authority.location ? "retained" : "released", } }; } @@ -106,6 +111,7 @@ export function gradeRemoteWorkspaceStorage(evidence: RemoteWorkspaceStorageEvid && point.authoritySchema === "paperclip.remote-workspace.v1" && point.authorityDirty === false && point.authorityWriter === null && point.providerType === "cloud_sandbox" && point.fileSource === (evidence.expectedFileSource ?? "live") + && (!evidence.expectedLocation || point.locationState === evidence.expectedLocation) && (point.fileSource !== "saved" || point.fileSnapshotId === point.snapshotId) && /^[a-f0-9]{64}$/.test(point.proofSha256)); return [ diff --git a/tests/runner-e2e/server-restart-request.test.ts b/tests/runner-e2e/server-restart-request.test.ts index cf510935884..bfec299ead9 100644 --- a/tests/runner-e2e/server-restart-request.test.ts +++ b/tests/runner-e2e/server-restart-request.test.ts @@ -19,13 +19,27 @@ afterEach(async () => { await rm(root, { recursive: true, force: true }); }); -const cells = ["everyday-workflows", "daytona-restart-continuity"].flatMap(suite => - ["codex", "acpx"].map(provider => { - const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller" - && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider); - if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`); - return execution; - })); +it.each(["rolling-web-deployment", "rolling-web-deployment-warm"])("accepts the assembled %s restart request", async suite => { + const execution = runnerMatrix.find(row => row.suite.id === suite && row.profile.id === "runner-codex")!; + const recoveryBinding = { companyId: "11111111-1111-4111-8111-111111111111", + issueId: "22222222-2222-4222-8222-222222222222", runId: "33333333-3333-4333-8333-333333333333" }; + await writeFile(requestPath, JSON.stringify({ requestId: "rolling-request", rollingDeployment: true, + discardControllerCache: true, controllerOutage: controllerOutageForExecution(execution, {}), recoveryBinding })); + const request = await readRestartRequest(requestPath, [execution.id]); + expect(request).toMatchObject({ rollingDeployment: true, discardControllerCache: true, recoveryBinding }); + expect(request).not.toHaveProperty("controllerOutage"); +}); + +const cells = [ + { suite: "everyday-workflows", provider: "codex" }, + { suite: "everyday-workflows", provider: "acpx" }, + { suite: "daytona-restart-continuity", provider: "codex" }, +].map(({ suite, provider }) => { + const execution = runnerMatrix.find(row => row.suite.id === suite && row.task.id === "recover-controller" + && row.environment.id === "daytona" && row.profile.generation === "native" && row.profile.provider === provider); + if (!execution) throw new Error(`Missing real ${suite}/${provider} continuity cell`); + return execution; +}); it.each(cells)("accepts the real supervisor request file for $id", async execution => { const outage = controllerOutageForExecution(execution, {})!;