diff --git a/doc/evals.md b/doc/evals.md index f59035c2843..8db280a6219 100644 --- a/doc/evals.md +++ b/doc/evals.md @@ -661,3 +661,9 @@ run-attributed tool receipts on local and Daytona execution. Its four native Claude/Codex cells are explicit-only. Remote registrations remain unqualified until retained live attempts prove them. It uses the normal Product E2E report and keeps synthetic feedback distinct from actionable product findings. +# Legacy workspace preservation + +The explicit [legacy workspace migration Product E2E suites](../tests/runner-e2e/README.md#legacy-workspace-migration) +verify a retained session through the production offline migration and an empty +replacement controller. They complement the rolling deployment suites and do +not claim uninterrupted service during initial enrollment. diff --git a/doc/plans/2026-11-04-preserve-native-workspace-migration.md b/doc/plans/2026-11-04-preserve-native-workspace-migration.md new file mode 100644 index 00000000000..0f316804906 --- /dev/null +++ b/doc/plans/2026-11-04-preserve-native-workspace-migration.md @@ -0,0 +1,119 @@ +# Preserve native sessions when enrolling a legacy workspace + +A retained native session binds both its Core workspace address and the provider's +sandbox address. Migrating files to object storage must preserve these addresses; +a new `.paperclip-workspaces` directory alone is not session continuity. + +The offline workspace owner records selected native source bindings in its +successful migration operation. It checks the source again before publication. +The workspace points to that operation; arbitrary metadata is insufficient. +Each binding retains company/workspace, agent, task, normalized session, source +run, logical address, and the shared provider address. Conflicting or missing +identity blocks publication without deleting source files. + +Runtime selection uses the original logical address only for the exact retained +session. The remote workspace owner restores at the original provider address, +refuses another base path or an occupied unowned root, and may publish into a +verified empty directory on a newly allocated sandbox. The environment owner +includes the migration operation in its reusable lease fingerprint, so existing +warm policy obtains a fresh allocation and then reuses it normally. A verified +migration binding permits restoration of the exact retained warm backup into +that new allocation. Existing backup digest and identity checks remain required. + +Verification covers real PostgreSQL, filesystem restore after removal of the old +controller workspace, unchanged provider bytes, warm and per-turn policy, forged +receipts, wrong targets, occupied roots, and reusable-lease replacement. The live +acceptance gate is an ordinary agent continuation through the packaged offline +migration, followed by a rolling deployment with active ownership transfer. +Named staging enrollment remains pending until this proof and the PR gates pass. + +The first live enrollment also found that already-packaged local skill objects +were skipped by the older database-body migration. The skill storage owner now +copies each verified local archive and manifest to a new S3 upload claim, reads +back their exact bytes, then publishes the new receipt with a compare-and-swap. +Package and version IDs stay unchanged. Failed copies retain the original +receipt, local objects and durable upload journal. The maintenance entry pages +through this owner before its existing package and version verification. + +Fresh provider allocations can contain provider-owned runtime files. The plugin +lease metadata exposes a bounded `workspaceAllocationFiles` receipt containing +the exact provider lease, remote root, relative paths and SHA-256 hashes. Daytona +owns its sentinel encoding; workspace restore only consumes that normalized +receipt. The restore owner admits only the declared regular files, preserves +those bytes alongside the restored workspace, and refuses unknown files, changed +hashes and links. A parked allocation directory makes interrupted publication +retryable. Providers without the optional receipt retain empty-root behavior. + +The all-Runner prerequisite merged as `612d9931a6`. The second live Codex migration +at `469d28314f` verified all ten local skill objects, the original workspace and +provider backup, and reached a ready maintenance receipt. Its next turn exposed +the allocation-marker restore failure; that attempt remains failed. The exact +owned sandboxes, temporary credentials, database and object bucket were cleaned. +The original task remains staging enrollment, active handoff, and admission of +new work with existing warm/per-turn policies; staging has not yet been enrolled. + +The `91db767eb8` live attempt successfully restored the workspace and preserved +its reference bytes, but the next turn incorrectly selected a fresh provider +session. Its recorded admission reason was an adapter-configuration change. +A no-provider reproduction isolated the cause: importing unchanged managed +instructions creates a durable head, after which normal fingerprints omit the +old local instruction paths. Session preparation now compares the complete old +fingerprint using the verified imported content hash and the still-recorded +paths. It does not waive content, model, credential or other configuration +changes. Ordinary future turns retain the location-independent fingerprint. + +Migration qualification also waits for a live question to become a saved +handoff and for its task lock to release before requesting offline enrollment. +The maintenance owner's active-run rejection remains unchanged. + +The same failed attempt also exposed a fixture conflict: it requested a generated +reference file but forbade delivering it, and completion stayed blocked. The +migration journey now asks for that exact reference as an attachment after the +answer, alongside the confirmation. The byte/session/path preservation oracle +is unchanged; initial delivery remains forbidden and final task completion is +still required. This is fixture setup, not a relaxation of product completion. + +Live Codex at `2d5bb9a6d1` retained the same native and provider session, both +workspace addresses and exact reference bytes after old-home destruction. It +finished the task and delivered its confirmation and reference attachment. +The original campaign failed one generic output-count check because it counted +the explicitly requested reference as a second primary output. The migration +owner now grades that attachment separately and excludes only its independently +verified original bytes from the primary-output count; duplicates and unrelated +files remain failures. Regrading the retained evidence passes all 21 checks. +The original failed disposition remains recorded. Forty-four grader/readiness +controls pass; further live provider and warm-policy qualification follows. + +OpenCode per-turn and Codex warm at harness `94bb917947` passed all 21 product +checks. Their diagnostic observer held the disposable database open at cleanup; +the observer now closes on the final case receipt, and an ordinary PostgreSQL +drop reproduces both the failure and fix without a provider. Owned sandboxes, +keys and buckets were independently cleaned. Original campaign failures remain +recorded. Full repository build passed at `94bb917947`; full repository typecheck +and 38 focused harness tests passed at `b12ecd823e`. + +The next Grok migration preserved the workspace and session selection but failed +at the Runner's strict attachment check. Imported instructions and skills had +new registered copy paths. Grok still used immutable-instruction attachment, +while Claude already allowed only the authenticated canonical asset-path suffix +to move. A no-provider Grok regression reproduces that exact rejection. Grok now +uses the same narrow asset-path policy. Content, skill digests, workspace/session +identity, unknown context policy, same-run mutations and unauthenticated run +rotation remain rejected. All 57 ACPX provider tests pass after the fix. A fresh +Linux Runner and real Grok continuation must still verify the provider boundary. + +The released ACPX Codex profile had the same immutable-path default. Its +no-provider regression also reproduces the attachment rejection, and the same +narrow policy fixes it. All 58 ACPX provider tests pass. Unknown profiles retain +their prior restrictive default. Claude at `b12ecd823e` passed both strict +migration assertions, including the downloaded reference, but its full browser +workflow failed on an unnecessary confirmation and incorrect booking text. +That failed user-flow result remains recorded separately from preservation. + +Fresh Grok (`b05ba3728a`) and ACPX Codex (`24fabf1d81`) migration campaigns each +passed all 21 checks, including real provider continuation and cleanup. Default +regression coverage now exercises the actual runtime-selection database path +with matching, mismatched, and absent migration receipts. An executor-level +test restores real backup files into a created warm lease only for the exact +migrated session, verifies the retained provider identity, and checks the new +lease's backup stamp. Neither test replaces the migration decision helper. diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs index bcc1625dc88..d1bef5eaec3 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_backend.rs @@ -4115,6 +4115,18 @@ mod tests { authenticated_run_grant_attachment("claude"); } + #[cfg(unix)] + #[test] + fn grok_attachment_rotates_only_authenticated_asset_paths() { + authenticated_run_grant_attachment("grok"); + } + + #[cfg(unix)] + #[test] + fn codex_attachment_rotates_only_authenticated_asset_paths() { + authenticated_run_grant_attachment("codex"); + } + #[cfg(unix)] fn authenticated_run_grant_attachment(agent: &str) { let directory = temporary_directory(&format!("{agent}-cross-run-attach")); @@ -4164,7 +4176,7 @@ mod tests { "mcp": {"assignmentSetId": "assignment-1", "digest": "e".repeat(64), "bindingId": "old-run-binding"}, "futurePolicy": {"companyId": "company-1"}, }); - if agent == "claude" { + if matches!(agent, "claude" | "grok" | "codex") { descriptor_value["instructions"] = json!(format!( "Pinned prompt. Custom entry mentioning /old-bundle.\n\n{}", registered_asset_suffix(&descriptor_value["runtimeContext"]).unwrap() @@ -4301,7 +4313,7 @@ mod tests { json!("/new-bundle"); descriptor_value["runtimeContext"]["skills"][0]["bundle"]["rootPath"] = json!("/new-skill"); descriptor_value["runtimeContext"]["mcp"]["bindingId"] = json!("new-run-binding"); - if agent == "claude" { + if matches!(agent, "claude" | "grok" | "codex") { descriptor_value["instructions"] = json!(format!( "Pinned prompt. Custom entry mentioning /old-bundle.\n\n{}", registered_asset_suffix(&descriptor_value["runtimeContext"]).unwrap() diff --git a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_capabilities.rs b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_capabilities.rs index f8c48c170f7..7d0ffb3fd5c 100644 --- a/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_capabilities.rs +++ b/packages/paperclip-runner/runner/crates/runner-core/src/acpx_provider_capabilities.rs @@ -12,7 +12,7 @@ pub(crate) enum RunAttachmentPolicy { pub(crate) fn run_attachment_policy(agent: &str) -> RunAttachmentPolicy { match agent { "cursor" | "copilot" | "pi" | "kimi" => RunAttachmentPolicy::AuthenticatedRunGrants, - "claude" => RunAttachmentPolicy::AuthenticatedAssetPaths, + "claude" | "grok" | "codex" => RunAttachmentPolicy::AuthenticatedAssetPaths, _ => RunAttachmentPolicy::ImmutableInstructions, } } @@ -39,11 +39,17 @@ mod tests { run_attachment_policy("claude"), RunAttachmentPolicy::AuthenticatedAssetPaths ); - for agent in ["codex", "grok", "unknown"] { - assert_eq!( - run_attachment_policy(agent), - RunAttachmentPolicy::ImmutableInstructions - ); - } + assert_eq!( + run_attachment_policy("grok"), + RunAttachmentPolicy::AuthenticatedAssetPaths + ); + assert_eq!( + run_attachment_policy("codex"), + RunAttachmentPolicy::AuthenticatedAssetPaths + ); + assert_eq!( + run_attachment_policy("unknown"), + RunAttachmentPolicy::ImmutableInstructions + ); } } diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts index ee370a9c890..11e0c65cff4 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.test.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { execFileSync } from "node:child_process"; import { promises as fs } from "node:fs"; import os from "node:os"; @@ -1347,7 +1348,7 @@ describe("Daytona sandbox provider plugin", () => { }); }); - it("acquires a lease from a created sandbox", async () => { + it.each([true, false])("records allocation bytes for reuseLease=%s", async reuseLease => { process.env.DAYTONA_API_KEY = "host-key"; const sandbox = createMockSandbox(); mockCreate.mockResolvedValue(sandbox); @@ -1363,7 +1364,7 @@ describe("Daytona sandbox provider plugin", () => { config: { image: "node:20", timeoutMs: 300000, - reuseLease: true, + reuseLease, }, }); @@ -1374,13 +1375,21 @@ describe("Daytona sandbox provider plugin", () => { shellCommand: "bash", sandboxId: "sandbox-123", remoteCwd: "/home/daytona/paperclip-workspace", - reuseLease: true, + reuseLease, workspaceSentinel: { path: "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json", result: "written", }, }, }); + const uploaded = sandbox.fs.uploadFile.mock.calls.find(call => call[1] === + "/home/daytona/paperclip-workspace/.paperclip-runtime/reusable-sandbox-lease.json")?.[0]; + expect(Buffer.isBuffer(uploaded)).toBe(true); + expect(lease?.metadata?.workspaceAllocationFiles).toEqual({ providerLeaseId: "sandbox-123", + remoteCwd: "/home/daytona/paperclip-workspace", files: [{ + relativePath: ".paperclip-runtime/reusable-sandbox-lease.json", + sha256: createHash("sha256").update(uploaded as Buffer).digest("hex"), + }] }); expect(sandbox.fs.createFolder).toHaveBeenCalledWith( "/home/daytona/paperclip-workspace/.paperclip-runtime", "755", diff --git a/packages/plugins/sandbox-providers/daytona/src/plugin.ts b/packages/plugins/sandbox-providers/daytona/src/plugin.ts index 55f1aa8eaf0..8938696d932 100644 --- a/packages/plugins/sandbox-providers/daytona/src/plugin.ts +++ b/packages/plugins/sandbox-providers/daytona/src/plugin.ts @@ -33,6 +33,7 @@ import type { PluginEnvironmentGetInteractiveSetupParams, PluginEnvironmentInteractiveSetupSession, PluginEnvironmentLease, + PluginEnvironmentWorkspaceAllocationFiles, PluginEnvironmentProbeParams, PluginEnvironmentProbeResult, PluginEnvironmentRealizeWorkspaceParams, @@ -197,6 +198,7 @@ interface DaytonaDriverConfig { type WorkspaceSentinelResult = { path: string; token: string | null; + allocationFiles?: PluginEnvironmentWorkspaceAllocationFiles; runId?: string; providerLeaseId?: string; result: "written" | "matched" | "missing" | "mismatch" | "skipped"; @@ -633,23 +635,26 @@ async function writeWorkspaceSentinel(input: { return { path: sentinelPath, token: null, result: "skipped" }; } await input.sandbox.fs.createFolder(path.posix.dirname(sentinelPath), "755"); + const content = Buffer.from(JSON.stringify({ + version: 1, + token, + companyId: input.params.companyId, + environmentId: input.params.environmentId, + agentId: input.params.agentId, + executionWorkspaceId: input.params.executionWorkspaceId, + ...(input.params.executionWorkspaceId ? {} : { projectlessIssueId: input.params.issueId }), + adapterType: input.params.adapterType ?? null, + provider: "daytona", + writtenAt: new Date().toISOString(), + }, null, 2), "utf8"); await input.sandbox.fs.uploadFile( - Buffer.from(JSON.stringify({ - version: 1, - token, - companyId: input.params.companyId, - environmentId: input.params.environmentId, - agentId: input.params.agentId, - executionWorkspaceId: input.params.executionWorkspaceId, - ...(input.params.executionWorkspaceId ? {} : { projectlessIssueId: input.params.issueId }), - adapterType: input.params.adapterType ?? null, - provider: "daytona", - writtenAt: new Date().toISOString(), - }, null, 2), "utf8"), + content, sentinelPath, input.timeoutSeconds, ); return { path: sentinelPath, token, result: "written", + allocationFiles: { providerLeaseId: input.sandbox.id, remoteCwd: input.remoteCwd, + files: [{ relativePath: WORKSPACE_SENTINEL_RELATIVE_PATH, sha256: createHash("sha256").update(content).digest("hex") }] }, ...(!input.config.reuseLease ? { runId: input.params.runId!, providerLeaseId: input.sandbox.id } : {}) }; } @@ -734,6 +739,7 @@ function leaseMetadata(input: { ...(input.config.disk != null ? { disk: input.config.disk } : {}), ...(input.config.gpu != null ? { gpu: input.config.gpu } : {}), ...(input.workspaceSentinel ? { workspaceSentinel: input.workspaceSentinel } : {}), + ...(input.workspaceSentinel?.allocationFiles ? { workspaceAllocationFiles: input.workspaceSentinel.allocationFiles } : {}), }; } diff --git a/packages/plugins/sdk/src/index.ts b/packages/plugins/sdk/src/index.ts index 50f50d8929b..d70d8988d02 100644 --- a/packages/plugins/sdk/src/index.ts +++ b/packages/plugins/sdk/src/index.ts @@ -180,6 +180,7 @@ export type { PluginEnvironmentProbeParams, PluginEnvironmentProbeResult, PluginEnvironmentLease, + PluginEnvironmentWorkspaceAllocationFiles, PluginEnvironmentAcquireLeaseParams, PluginEnvironmentResumeLeaseParams, PluginEnvironmentReleaseLeaseParams, diff --git a/packages/plugins/sdk/src/protocol.ts b/packages/plugins/sdk/src/protocol.ts index bf43887f757..5ca50907f2b 100644 --- a/packages/plugins/sdk/src/protocol.ts +++ b/packages/plugins/sdk/src/protocol.ts @@ -618,6 +618,12 @@ export interface PluginEnvironmentProbeResult { metadata?: Record; } +export interface PluginEnvironmentWorkspaceAllocationFiles { + providerLeaseId: string; + remoteCwd: string; + files: Array<{ relativePath: string; sha256: string }>; +} + export interface PluginEnvironmentLease { providerLeaseId: string | null; metadata?: Record; diff --git a/server/src/__tests__/environment-runtime.test.ts b/server/src/__tests__/environment-runtime.test.ts index 9231fb7a089..45aa4153326 100644 --- a/server/src/__tests__/environment-runtime.test.ts +++ b/server/src/__tests__/environment-runtime.test.ts @@ -17,6 +17,7 @@ import { } from "@paperclipai/adapter-utils/ssh"; import { activityLog, + workspaceOperations, costEvents, agents, builtInManagedResources, @@ -238,6 +239,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { await stopSshEnvLabFixture(path.join(root, "state.json")).catch(() => undefined); await rm(root, { recursive: true, force: true }).catch(() => undefined); } + await db.delete(workspaceOperations); await db.delete(activityLog); await db.delete(environmentLeases); await db.delete(issues); @@ -4441,7 +4443,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); }); - it("does not resume released reusable plugin sandbox leases after provider config drift", async () => { + it.each(["provider config", "workspace migration"])("does not resume released reusable plugin sandbox leases after %s drift", async drift => { const pluginId = randomUUID(); const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment(); const providerConfig = { @@ -4488,12 +4490,13 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { await insertReusableProject(projectId, companyId); await insertReusableExecutionWorkspace(executionWorkspaceId, companyId, projectId); + let acquisitions = 0; const workerManager = { isRunning: vi.fn((id: string) => id === pluginId), call: vi.fn(async (_pluginId: string, method: string, params: any) => { if (method === "environmentAcquireLease") { return { - providerLeaseId: `lease-${params.config.image}`, + providerLeaseId: `lease-${params.config.image}${++acquisitions > 1 ? "-replacement" : ""}`, metadata: { provider: "fake-plugin", image: params.config.image, @@ -4532,13 +4535,27 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }); await runtimeWithPlugin.releaseRunLeases(runId); + if (drift === "workspace migration") { + const operationId = randomUUID(); + await db.insert(workspaceOperations).values({ id: operationId, companyId, executionWorkspaceId, + phase: "remote_workspace_migration", status: "succeeded", metadata: { + schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: { + schema: "paperclip.native-workspace-migration.v1", remoteCwd: "/workspace", sessions: [{ + sourceRunId: runId, agentId, issueId: randomUUID(), sessionId: randomUUID(), logicalCwd: "/old/controller/workspace", + }], + }, + } }); + await db.update(executionWorkspaces).set({ metadata: { nativeWorkspaceMigration: { + schema: "paperclip.native-workspace-migration.v1", operationId, + } } }).where(eq(executionWorkspaces.id, executionWorkspaceId)); + } const nextRunId = randomUUID(); await insertRunningManualRun(nextRunId, companyId, agentId); const updatedEnvironment: Environment = { ...environment, config: { ...providerConfig, - image: "template-b", + image: drift === "provider config" ? "template-b" : "template-a", }, }; await environmentService(db).update(environment.id, { @@ -4557,7 +4574,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => { }, }); - expect(second.lease.providerLeaseId).toBe("lease-template-b"); + expect(second.lease.providerLeaseId).toBe(`lease-${updatedEnvironment.config.image}-replacement`); + expect(second.lease.metadata?.reusableSandboxLease).not.toEqual(first.lease.metadata?.reusableSandboxLease); expect(workerManager.call).not.toHaveBeenCalledWith( pluginId, "environmentResumeLease", diff --git a/server/src/__tests__/ephemeral-migration.test.ts b/server/src/__tests__/ephemeral-migration.test.ts index a2687c0e4ec..93b8f1e51ee 100644 --- a/server/src/__tests__/ephemeral-migration.test.ts +++ b/server/src/__tests__/ephemeral-migration.test.ts @@ -23,6 +23,11 @@ import { verifyAgentFileObjects } from "../services/agent-file-maintenance.js"; import { getStorageService } from "../storage/index.js"; import { resolveDecisionSigningSecret } from "../services/decision-signing.js"; import { getRunLogStore } from "../services/run-log-store.js"; +import { companySkillService } from "../services/company-skills.js"; +import { skillPackageStore } from "../services/skill-package-store.js"; +import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js"; +import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js"; +import { companySkills, companySkillRepositorySnapshots } from "@paperclipai/db"; // Explicit real-dependency qualification. Never contacts off-host storage in // the default test path; use a disposable S3-compatible service and PostgreSQL. @@ -55,6 +60,8 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId)); await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId)); await db.delete(agents).where(eq(agents.companyId, companyId)); + await db.delete(companySkills).where(eq(companySkills.companyId, companyId)); + await db.delete(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, companyId)); await db.delete(companies).where(eq(companies.id, companyId)); let token: string | undefined; do { @@ -100,6 +107,46 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll await s3.send(new PutObjectCommand({ Bucket: bucket, Key: file.objectKey, Body: "corrupt" })); await expect(verifyAgentFileObjects(db, getStorageService(), AbortSignal.timeout(30_000))).rejects.toThrow(/agent_file_object_(size|digest)_mismatch/); }, 120_000); + it("preserves original skill objects and identity when destination readback fails", async () => { + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk"); + const skills = await companySkillService(db).list(companyId); + const local = getStorageService(), store = skillPackageStore(db, local); + const before = await Promise.all(skills.map(async skill => { + const version = await store.version(companyId, skill.id, skill.currentVersionId!); + return { skillId: skill.id, version: version!, files: await store.versionFiles(version!) }; + })); + expect(before.length).toBeGreaterThan(0); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3"); + const remote = getStorageService(); + await expect(migrateLocalSkillPackageObjects(db, { source: local, destination: { ...remote, + getObject: async () => { throw new Error("Synthetic destination readback failed"); } }, writersStopped: true, + signal: AbortSignal.timeout(30_000) })).rejects.toThrow("Synthetic destination readback failed"); + for (const point of before) { + expect((await store.descriptor(companyId, point.version.packageSnapshotId!)).storage?.provider).toBe("local_disk"); + expect(await store.versionFiles(point.version)).toEqual(point.files); + } + }, 120_000); + it("verifies bundled skill migration after removal of the old server home", async () => { + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk"); + const skills = await companySkillService(db).list(companyId), local = skillPackageStore(db); + const before = await Promise.all(skills.map(async skill => { + const version = await local.version(companyId, skill.id, skill.currentVersionId!); + return { skillId: skill.id, version: version!, files: await local.versionFiles(version!) }; + })); + vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3"); + const result = await migrate(); + expect(result.eligibility.eligible).toBe(true); + expect(result.counts.skillObjects).toBe(before.length); + await fs.rm(home, { recursive: true, force: true }); + const remote = skillPackageStore(db); + for (const point of before) { + const version = await remote.version(companyId, point.skillId, point.version.id); + expect(version?.packageSnapshotId).toBe(point.version.packageSnapshotId); + expect(await remote.versionFiles(version!)).toEqual(point.files); + } + expect((await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 })).results + .every(row => ["remote_storage", "database_release_snapshot"].includes(row.status))).toBe(true); + }, 120_000); it("migrates overlapping task folders through maintenance and restores both from S3 after volume loss", async () => { vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32)); const parent = path.join(home, "project"), child = path.join(parent, ".paperclip/worktrees/task"); diff --git a/server/src/__tests__/heartbeat-workspace-session.test.ts b/server/src/__tests__/heartbeat-workspace-session.test.ts index 7603852eb7c..790c59bd656 100644 --- a/server/src/__tests__/heartbeat-workspace-session.test.ts +++ b/server/src/__tests__/heartbeat-workspace-session.test.ts @@ -1,3 +1,4 @@ +import { createHash } from "node:crypto"; import { execFile as execFileCallback } from "node:child_process"; import fs from "node:fs/promises"; import os from "node:os"; @@ -2732,6 +2733,28 @@ describe("effective run session config freshness", () => { expect(freshness(await metadata("/replacement-controller", "sha256:entry-v1", "skill-version-2")).changedCategories).toContain("runtimeSkills"); }); + it("retains a legacy managed-instruction session after verified import removes the old home", async () => { + const root = await fs.mkdtemp(path.join(os.tmpdir(), "legacy-instruction-session-")); + const body = "Preserve this exact agent identity and instructions.\n"; + await fs.writeFile(path.join(root, "AGENTS.md"), body); + const config = { model: "gpt-5.4-mini", instructionsBundleMode: "managed", instructionsRootPath: root, + instructionsFilePath: path.join(root, "AGENTS.md"), instructionsEntryFile: "AGENTS.md" }; + const before = await buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: config }); + await fs.rm(root, { recursive: true }); + const managedInstructions = { entryFile: "AGENTS.md", contentHash: "sha256:" + createHash("sha256").update(body).digest("hex") }; + const metadata = (contentHash = managedInstructions.contentHash, model = config.model, retainManagedInstructionPaths = false) => + buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: { ...config, model }, + managedInstructions: { ...managedInstructions, contentHash }, retainManagedInstructionPaths }); + const after = await metadata(); + const decide = (current: SessionConfigMetadata, compatible: SessionConfigMetadata[] = []) => resolveTaskSessionConfigFreshness({ + hasTaskSession: true, configuredModel: config.model, taskSessionParams: sessionParamsWithConfigMetadata(before), + configMetadata: current, compatibleConfigMetadata: compatible }); + expect(decide(after)).toMatchObject({ reset: true, changedCategories: ["adapterConfig"] }); + expect(decide(after, [await metadata(undefined, undefined, true)]).reset).toBe(false); + expect(decide(await metadata("sha256:changed"), [await metadata("sha256:changed", undefined, true)]).reset).toBe(true); + expect(decide(await metadata(undefined, "different-model"), [await metadata(undefined, "different-model", true)]).reset).toBe(true); + }); + it("does not read unbounded legacy instructions paths for config fingerprints", async () => { const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-session-fingerprint-")); const instructionsPath = path.join(root, "AGENTS.md"); diff --git a/server/src/maintenance/ephemeral-migration-data.ts b/server/src/maintenance/ephemeral-migration-data.ts index 44acf0a3635..c45a5731fe3 100644 --- a/server/src/maintenance/ephemeral-migration-data.ts +++ b/server/src/maintenance/ephemeral-migration-data.ts @@ -1,3 +1,7 @@ +import { loadConfig } from "../config.js"; +import { createStorageService } from "../storage/service.js"; +import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js"; +import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js"; import { heartbeatRuns, type Db } from "@paperclipai/db"; import { offlineMigrationExecutionWhere } from "../services/offline-migration-execution.js"; import { getStorageService } from "../storage/index.js"; @@ -73,6 +77,15 @@ export async function migrateEphemeralServerData(db: Db, input: { const packages = skillPackageMaintenance(db); const skills = skillInstallationMaintenance(db); + phase("skill_objects"); + const localSkillStorage = createStorageService(createLocalDiskStorageProvider(loadConfig().storageLocalDiskBaseDir)); + after = undefined; + for (;;) { + const page = await migrateLocalSkillPackageObjects(db, { source: localSkillStorage, destination: storage, writersStopped: true, signal, after }); + counts.skillObjects = (counts.skillObjects ?? 0) + page.migrated; + if (page.inspected < 50 || !page.after) break; + after = page.after; + } for (const [name, migrate] of [["skill_packages", packages.migrateSnapshots], ["skill_versions", packages.migrateVersions]] as const) { phase(name); after = undefined; diff --git a/server/src/services/environment-runtime.ts b/server/src/services/environment-runtime.ts index 6f53cbe45e9..77c6dff225d 100644 --- a/server/src/services/environment-runtime.ts +++ b/server/src/services/environment-runtime.ts @@ -1,3 +1,4 @@ +import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js"; import { HttpError } from "../errors.js"; import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js"; import { createOwnedSandbox, recoverSandboxCreations, recoverableTaskCreation } from "./sandbox-creation-operations.js"; @@ -964,6 +965,8 @@ async function buildReusableSandboxLeaseFingerprint(input: { companyId: input.companyId, environment: input.environment, }); + const workspaceMigration = input.executionWorkspaceId + ? await readNativeWorkspaceMigration(input.db, { companyId: input.companyId, id: input.executionWorkspaceId }) : null; return createEffectiveRunConfigFingerprints({ lease: { companyId: input.companyId, @@ -972,6 +975,7 @@ async function buildReusableSandboxLeaseFingerprint(input: { driver: input.environment.driver, }, executionWorkspaceId: input.executionWorkspaceId, + ...(workspaceMigration ? { workspaceMigrationGeneration: workspaceMigration.operationId } : {}), ...(input.executionWorkspaceId ? {} : { projectlessIssueId: input.issueId }), agentId: input.agentId, adapterType: input.adapterType, diff --git a/server/src/services/heartbeat/runtime-selection.test.ts b/server/src/services/heartbeat/runtime-selection.test.ts index 3c7c3bab0d5..b8f8697c386 100644 --- a/server/src/services/heartbeat/runtime-selection.test.ts +++ b/server/src/services/heartbeat/runtime-selection.test.ts @@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto"; import { mkdtemp, rm, writeFile } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; -import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db"; +import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db"; import { eq, sql } from "drizzle-orm"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js"; @@ -82,6 +82,54 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () = const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id)); const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false }; + it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => { + const sameSession = receipt === "matching"; + const { input, run, agent, issue } = await fixture(); + const sessionId = randomUUID(), operationId = randomUUID(); + const logicalCwd = "/paperclip/legacy/task-workspace"; + const marker = { schema: "paperclip.native-workspace-migration.v1", operationId }; + const [workspace] = await db.insert(executionWorkspaces).values({ + companyId: agent.companyId, sourceIssueId: issue.id, name: "Migrated workspace", + mode: "shared_workspace", strategyType: "project_primary", cwd: "/tmp/ephemeral/task-workspace", + providerType: "cloud_sandbox", metadata: receipt === "absent" ? {} : { nativeWorkspaceMigration: marker }, + }).returning(); + await db.insert(workspaceOperations).values({ + id: operationId, companyId: agent.companyId, executionWorkspaceId: workspace!.id, + phase: "remote_workspace_migration", status: "succeeded", metadata: { + schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), + nativeSessionContinuity: { schema: marker.schema, remoteCwd: sandbox.remoteCwd, sessions: [{ + sourceRunId: randomUUID(), agentId: agent.id, issueId: issue.id, + sessionId: sameSession ? sessionId : randomUUID(), logicalCwd, + }] }, + }, + }); + input.workspace.persistedExecutionWorkspace = workspace!; + input.workspace.executionWorkspace.cwd = logicalCwd; + input.workspace.executionTarget = sandbox; + const prior = await selectHeartbeatRuntime(db, input); + if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection"); + const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } }; + const priorProfile = (await read(run.id)).runnerProfileJson; + await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId, + runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: { + backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread", + providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId }, + } }, + }).where(eq(heartbeatRuns.id, run.id)); + const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id, + issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId, + createdAt: new Date(run.createdAt.getTime() + 1000), + }).returning(); + input.run = nextRun!; + input.workspace.executionWorkspace.cwd = workspace!.cwd!; + const selected = await selectHeartbeatRuntime(db, input); + if (!selected.selected || !selected.nativeExecution) throw new Error("Expected migrated native selection"); + expect(selected.nativeExecution.session.normalizedSessionId).toBe(sessionId); + expect(selected.nativeExecution.workspace.cwd).toBe(sameSession ? logicalCwd : workspace!.cwd); + expect((await read(nextRun!.id)).runnerProfileJson?.nativeExecutionInput).toEqual(selected.nativeExecution); + expect((await read(nextRun!.id)).nativeSessionId).toBe(selected.nativeExecution.session.normalizedSessionId); + }); + it.each([false, true])("persists the Cursor provider API placement and completion authority without a Computer lease (ephemeral: %s)", async ephemeral => { vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral)); const { input, run, issue } = await fixture(); diff --git a/server/src/services/heartbeat/runtime-selection.ts b/server/src/services/heartbeat/runtime-selection.ts index d9613c3dc9b..20d3c40ac8d 100644 --- a/server/src/services/heartbeat/runtime-selection.ts +++ b/server/src/services/heartbeat/runtime-selection.ts @@ -1,3 +1,4 @@ +import { readNativeWorkspaceMigration, migratedNativeExecutionCwd } from "../native-runtime/native-workspace-migration.js"; import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js"; import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js"; import { museRunnerBroker } from "../muse-runner-broker.js"; @@ -762,6 +763,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele codexCliVersion: codexCliVersion!, }); } + const workspaceMigration = persistedExecutionWorkspace?.metadata?.nativeWorkspaceMigration + ? await readNativeWorkspaceMigration(db, { companyId: agent.companyId, id: nativeExecutionWorkspaceId }) : null; const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) => buildNativeExecutionInput({ agentKeyId: agentIdentity?.keyId, @@ -817,7 +820,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele // transient workspace to the run id so the native input remains durable and replayable // without fabricating a project-scoped execution_workspaces row. id: nativeExecutionWorkspaceId, - cwd: executionWorkspace.cwd, + cwd: migratedNativeExecutionCwd(workspaceMigration, { agentId: agent.id, issueId: issueRef.id, sessionId: normalizedSessionId }) ?? executionWorkspace.cwd, repoUrl: executionWorkspace.repoUrl, repoRef: executionWorkspace.repoRef, branchName: executionWorkspace.branchName, diff --git a/server/src/services/heartbeat/session-preparation.ts b/server/src/services/heartbeat/session-preparation.ts index 832c5ebcefd..04ec0654fec 100644 --- a/server/src/services/heartbeat/session-preparation.ts +++ b/server/src/services/heartbeat/session-preparation.ts @@ -152,7 +152,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre : null, }; const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput); - let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; + const compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = []; + const importedManagedInstructions = Boolean(taskSession && sessionConfigMetadataInput.managedInstructions?.contentHash + && readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint); + if (importedManagedInstructions) { + compatibleConfigMetadata.push(await buildEffectiveRunSessionConfigMetadata({ + ...sessionConfigMetadataInput, retainManagedInstructionPaths: true, + })); + } if (managedAiRuntime && aiBinding && taskSession && readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) { const revisions = requestedAiBinding?.mode === "router" @@ -166,7 +173,12 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity, sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions, }); - compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }))); + compatibleConfigMetadata.push(...await Promise.all(candidates.flatMap(candidate => [ + buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }), + ...(importedManagedInstructions ? [buildEffectiveRunSessionConfigMetadata({ + ...sessionConfigMetadataInput, ...candidate, retainManagedInstructionPaths: true, + })] : []), + ]))); } const configuredModel = readConfiguredModelFromAdapterConfig(runtimeConfig); diff --git a/server/src/services/heartbeat/workspaces.ts b/server/src/services/heartbeat/workspaces.ts index 467c8317338..86023d53494 100644 --- a/server/src/services/heartbeat/workspaces.ts +++ b/server/src/services/heartbeat/workspaces.ts @@ -2839,6 +2839,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: { effectiveAdapterConfig: Record; managedAiHome?: string; managedInstructions?: { entryFile: string; contentHash: string | null } | null; + retainManagedInstructionPaths?: boolean; agentRuntimeConfig: unknown; issueOverrides: unknown; workspaceConfig: unknown; @@ -2858,7 +2859,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: { ...(input.managedInstructions.contentHash !== null ? { contentHash: input.managedInstructions.contentHash } : {}), } : await resolveInstructionsConfigFingerprintMetadata(input.effectiveAdapterConfig); const effectiveAdapterConfig = { ...managedAiSessionFingerprintConfig(input.effectiveAdapterConfig, input.managedAiHome) }; - if (input.managedInstructions) { + if (input.managedInstructions && !input.retainManagedInstructionPaths) { delete effectiveAdapterConfig.instructionsRootPath; delete effectiveAdapterConfig.instructionsFilePath; } diff --git a/server/src/services/native-runtime/native-session-executor.test.ts b/server/src/services/native-runtime/native-session-executor.test.ts index 2de774f90ea..ce7da62cb9b 100644 --- a/server/src/services/native-runtime/native-session-executor.test.ts +++ b/server/src/services/native-runtime/native-session-executor.test.ts @@ -44,6 +44,7 @@ import { budgetReservations, heartbeatRuns, environmentLeases, + executionWorkspaces, heartbeatRunEvents, issueThreadInteractions, issues, @@ -2677,6 +2678,14 @@ describe("verified native harness backups", () => { ).toBe(false); }); + it("requires verified migration continuity before restoring a retained warm session into a fresh sandbox", () => { + const acquisition = { acquisitionOutcome: "created" as const, reusableLeaseConfigured: true, backupAvailable: true }; + expect(shouldRestoreNativeHarnessBackupIntoSandbox(acquisition)).toBe(false); + expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true })).toBe(true); + expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, backupAvailable: false })).toBe(false); + expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, acquisitionOutcome: "resumed" })).toBe(false); + }); + it("accepts a complete digest-matched backup and rejects corruption", async () => { const root = await mkdtemp(join(tmpdir(), "paperclip-harness-backup-")); try { @@ -6660,6 +6669,7 @@ function leaseDb( : []; const query = { then: Promise.resolve(rows).then.bind(Promise.resolve(rows)), + leftJoin: () => query, where: () => query, orderBy: () => query, for: () => query, @@ -16811,6 +16821,100 @@ describe("runnerd provider runtime wiring", () => { expect(runnerEnvironment).toMatchObject(paths); }); + it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => { + const remoteCwd = join(isolatedStateDirectory, "remote"); + const current: NativeExecutionInputV1 = { ...execution, + binding: { ...execution.binding, companyId: randomUUID(), agentId: randomUUID(), issueId: randomUUID(), + runId: randomUUID(), executionWorkspaceId: randomUUID() }, + workspace: { ...execution.workspace, cwd: "/paperclip/legacy/task-workspace" }, + session: { ...execution.session, normalizedSessionId: randomUUID(), lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } }, + }; + const operationId = randomUUID(), runnerInstanceId = "migrated-warm-runner"; + const receipt = { operationId, + workspaceMetadata: { nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId } }, + metadata: { schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: { + schema: "paperclip.native-workspace-migration.v1", remoteCwd, sessions: [{ sourceRunId: randomUUID(), + agentId: current.binding.agentId, issueId: current.binding.issueId, + sessionId: matching ? current.session.normalizedSessionId : randomUUID(), logicalCwd: current.workspace.cwd, + }], + } }, + }; + const updates: Array<{ table: unknown; values: Record }> = []; + const db = leaseDb(current, {}, {}, updates), ordinarySelect = db.select.bind(db); + db.select = ((fields?: Record) => ({ from: (table: unknown) => { + if (table !== executionWorkspaces && table !== environmentLeases) return ordinarySelect(fields as never).from(table as never); + const rows = table === executionWorkspaces ? [receipt] : [{ metadata: {}, providerLeaseId: "sandbox-created", + companyId: current.binding.companyId, heartbeatRunId: current.binding.runId }]; + const query = { then: Promise.resolve(rows).then.bind(Promise.resolve(rows)), + leftJoin: () => query, where: () => query, limit: () => query }; + return query; + } })) as unknown as Db["select"]; + const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => { + if (command.args?.[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({ + schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd", + packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2, + durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"], + }) }; + if (command.args?.[0] === "--version") return { exitCode: 0, timedOut: false, stdout: "codex-cli 0.160.0", stderr: "" }; + if (command.args?.[1]?.includes("base64")) { + try { return { exitCode: 0, timedOut: false, stdout: execFileSync("sh", command.args, { encoding: "utf8" }), stderr: "" }; } + catch { return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; } + } + return { exitCode: 0, timedOut: false, stdout: "", stderr: "" }; + }); + // The sandbox boundary copies actual backup bytes; migration admission and + // the executor's manifest, restored identity, and lease stamp checks stay real. + const syncIn = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => { + for (const operation of operations) for (const file of operation.files) { + expect(file.targetPath.startsWith(`${remoteCwd}/`)).toBe(true); + await mkdir(dirname(file.targetPath), { recursive: true }); + await cp(file.sourcePath, file.targetPath, { recursive: true }); + } + }); + await createRunnerdBackend({ db, execution: current, runnerInstanceId, runnerIngressAuthorized: true, + runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd, environmentId: "environment", + leaseId: "lease-created", providerKey: "daytona", reusableLeaseConfigured: true, + effectiveCapabilities: { runnerWebSocketIngress: true }, + sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "sandbox-created" }, + runner: { execute: remoteExecute, syncIn }, + } as never, + }); + state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!(); + const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise }; + const backupRoot = join(options.stateDirectory!, "failover-backups/current"); + for (const directory of resolveNativeHarnessPersistenceProfile(current).directories) + await mkdir(join(backupRoot, directory.name), { recursive: true }); + const runnerState = durableRunnerState({ runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId, + runnerInstanceId, environmentLeaseId: "lease-created" }, "suspended"); + await writeFile(join(backupRoot, "runner/runner-state.json"), JSON.stringify(runnerState)); + await writeFile(join(backupRoot, "runner/codex-provider-state.json"), JSON.stringify({ + schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "prepared", threadId: "retained-provider-thread", + providerSessionId: null, activeProviderTurnId: null, config: { provider: "codex", driver: "codex_app_server" }, + })); + await writeFile(join(backupRoot, "codex-home/sentinel.txt"), "retained provider home"); + await writeFile(join(backupRoot, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({ + backupRoot, execution: current, runnerInstanceId, sourceProviderLeaseId: "prior-sandbox", + providerSessionIdentity: { providerSessionId: "retained-provider-thread", providerBackendSessionId: null, providerSessionIdentity: null }, + }))); + if (matching) { + await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined(); + const remoteSession = join(remoteCwd, ".paperclip-runtime/paperclip-runner/sessions", + createHash("sha256").update(current.session.normalizedSessionId!).digest("hex")); + expect(JSON.parse(await readFile(join(remoteSession, "runner/runner-state.json"), "utf8"))).toEqual(runnerState); + expect(await readFile(join(remoteSession, "filesystem/codex-home/sentinel.txt"), "utf8")).toBe("retained provider home"); + expect(updates.filter(update => update.table === environmentLeases)).toEqual([expect.objectContaining({ + values: expect.objectContaining({ metadata: { nativeHarnessBackup: expect.objectContaining({ + normalizedSessionId: current.session.normalizedSessionId, runnerInstanceId, + sourceProviderLeaseId: "prior-sandbox", authorizedProviderLeaseId: "sandbox-created", + }) } }), + })]); + } else { + await expect(options.prepareExternalRunnerState()).rejects.toThrow("backup_without_reusable_lease"); + expect(syncIn).not.toHaveBeenCalled(); + expect(updates.filter(update => update.table === environmentLeases)).toEqual([]); + } + }); + it("archives failover evidence with an explicitly replaced provider session", async () => { const remoteCwd = join(isolatedStateDirectory, "remote"); const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => { diff --git a/server/src/services/native-runtime/native-session-executor.ts b/server/src/services/native-runtime/native-session-executor.ts index 7c85342d3bf..a38c49122ba 100644 --- a/server/src/services/native-runtime/native-session-executor.ts +++ b/server/src/services/native-runtime/native-session-executor.ts @@ -1,3 +1,4 @@ +import { canRestoreMigratedNativeSession } from "./native-workspace-migration.js"; import { prepareCursorCloudRuntime } from "./cursor-cloud-runtime.js"; import { resolveLocalRuntimeApiUrl } from "../../runtime-api.js"; import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js"; @@ -6638,10 +6639,11 @@ export function shouldRestoreNativeHarnessBackupIntoSandbox(input: { acquisitionOutcome: "created" | "resumed" | "replacement" | null; reusableLeaseConfigured: boolean | null | undefined; backupAvailable: boolean; + migratedSessionContinuation?: boolean; }): boolean { return ( input.acquisitionOutcome === "created" && - input.reusableLeaseConfigured === false && + (input.reusableLeaseConfigured === false || input.migratedSessionContinuation === true) && input.backupAvailable ); } @@ -13880,6 +13882,11 @@ async function createRunnerdBackendWithinSessionClaim( reusableLeaseConfigured: remoteTarget.reusableLeaseConfigured, backupAvailable, + migratedSessionContinuation: sandboxLeaseAcquisition?.outcome === "created" && backupAvailable + && remoteTarget.reusableLeaseConfigured === true && await canRestoreMigratedNativeSession(input.db, { + ...input.execution.binding, workspaceId: input.execution.binding.executionWorkspaceId, + sessionId: nativeSessionKey(input.execution), logicalCwd: input.execution.workspace.cwd, remoteCwd: remoteTarget.remoteCwd, + }), }); await input.trace?.record({ name: "sandbox.lease.acquisition", diff --git a/server/src/services/native-runtime/native-workspace-migration.ts b/server/src/services/native-runtime/native-workspace-migration.ts new file mode 100644 index 00000000000..f60ece66d75 --- /dev/null +++ b/server/src/services/native-runtime/native-workspace-migration.ts @@ -0,0 +1,83 @@ +import path from "node:path"; +import { and, eq, sql } from "drizzle-orm"; +import { executionWorkspaces, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db"; +import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js"; +import { findNativeSessionResumeRun } from "./native-session-resume.js"; +import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js"; + +const SCHEMA = "paperclip.native-workspace-migration.v1"; +const object = (value: unknown): Record => value && typeof value === "object" && !Array.isArray(value) ? value as Record : {}; +const absolute = (value: unknown): value is string => typeof value === "string" && value !== "/" + && !value.includes("\0") && path.posix.isAbsolute(value) && path.posix.normalize(value) === value; +const uuid = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value); +type Workspace = Pick; +type SessionBinding = { sourceRunId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string }; +export type NativeWorkspaceMigration = { schema: typeof SCHEMA; remoteCwd: string; sessions: SessionBinding[] }; + +export async function inspectNativeWorkspaceMigration(db: Db, workspace: Workspace): Promise { + const rows = await db.execute<{ agentId: string; issueId: string; sessionId: string; createdAt: Date }>(sql` + select s.agent_id as "agentId", r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt" + from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id + and r.company_id = s.company_id and r.agent_id = s.agent_id + where s.company_id = ${workspace.companyId} and s.adapter_type = 'paperclip_runner' + and r.native_issue_id is not null and r.native_session_id is not null + and r.runner_profile_json#>>'{nativeExecutionInput,binding,executionWorkspaceId}' = ${workspace.id} + order by s.id limit 1001`); + if (rows.length > 1000) throw new Error("native_workspace_migration_session_limit"); + const sessions: SessionBinding[] = []; + let remoteCwd: string | null = null; + for (const row of rows) { + const source = await findNativeSessionResumeRun(db, { companyId: workspace.companyId, agentId: row.agentId, + issueId: row.issueId, normalizedSessionId: row.sessionId, beforeCreatedAt: new Date(row.createdAt), includeProgressBarrier: true }); + if (!source) continue; + const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput); + const [saved] = await db.select({ sync: sql`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` }) + .from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, source.id))); + const sync = readNativeWorkspaceSyncReference(saved?.sync); + if (execution.binding.executionWorkspaceId !== workspace.id || execution.binding.companyId !== workspace.companyId + || execution.binding.agentId !== row.agentId || execution.binding.issueId !== row.issueId + || execution.session.normalizedSessionId !== row.sessionId || !absolute(execution.workspace.cwd) + || !sync || sync.workspaceId !== workspace.id || !absolute(sync.remoteCwd)) throw new Error("native_workspace_migration_identity_unproven"); + if (remoteCwd && remoteCwd !== sync.remoteCwd) throw new Error("native_workspace_migration_remote_path_conflict"); + remoteCwd = sync.remoteCwd; + sessions.push({ sourceRunId: source.id, agentId: row.agentId, issueId: row.issueId, sessionId: row.sessionId, logicalCwd: execution.workspace.cwd }); + } + return remoteCwd ? { schema: SCHEMA, remoteCwd, sessions } : null; +} + +export async function readNativeWorkspaceMigration(db: Pick, workspace: Workspace) { + const [row] = await db.select({ workspaceMetadata: executionWorkspaces.metadata, metadata: workspaceOperations.metadata, operationId: workspaceOperations.id }) + .from(executionWorkspaces).leftJoin(workspaceOperations, and( + eq(workspaceOperations.companyId, executionWorkspaces.companyId), eq(workspaceOperations.executionWorkspaceId, executionWorkspaces.id), + sql`${workspaceOperations.id} = case when ${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId' ~ '^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$' + then (${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId')::uuid end`, + eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "succeeded"), + )).where(and(eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, workspace.id))); + if (!row?.workspaceMetadata?.nativeWorkspaceMigration) return null; + const migration = object(row.metadata?.nativeSessionContinuity); + if (object(row.workspaceMetadata.nativeWorkspaceMigration).schema !== SCHEMA || !row.operationId || row.metadata?.schema !== "paperclip.remote-workspace-migration.v1" || !uuid(row.metadata.snapshotId) + || migration.schema !== SCHEMA || !absolute(migration.remoteCwd) || !Array.isArray(migration.sessions) + || !migration.sessions.length || migration.sessions.length > 1000 + || !migration.sessions.every(value => { + const session = object(value); + return [session.sourceRunId, session.agentId, session.issueId, session.sessionId].every(uuid) && absolute(session.logicalCwd); + })) throw new Error("native_workspace_migration_receipt_invalid"); + return { ...(migration as NativeWorkspaceMigration), operationId: row.operationId }; +} + +export function nativeWorkspaceMigrationMarker(operationId: string) { + return { schema: SCHEMA, operationId }; +} + +export function migratedNativeExecutionCwd(migration: NativeWorkspaceMigration | null, input: { agentId: string; issueId: string; sessionId: string }) { + return migration?.sessions.find(session => session.agentId === input.agentId && session.issueId === input.issueId + && session.sessionId === input.sessionId)?.logicalCwd ?? null; +} + +export async function canRestoreMigratedNativeSession(db: Db, input: { + companyId: string; workspaceId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string | null; remoteCwd: string; +}) { + if (input.logicalCwd === null) return false; + const migration = await readNativeWorkspaceMigration(db, { companyId: input.companyId, id: input.workspaceId }); + return migration?.remoteCwd === input.remoteCwd && migratedNativeExecutionCwd(migration, input) === input.logicalCwd; +} diff --git a/server/src/services/remote-workspace-allocation.ts b/server/src/services/remote-workspace-allocation.ts new file mode 100644 index 00000000000..5eb04dca7c5 --- /dev/null +++ b/server/src/services/remote-workspace-allocation.ts @@ -0,0 +1,18 @@ +import type { PluginEnvironmentWorkspaceAllocationFiles } from "@paperclipai/plugin-sdk"; +import type { EnvironmentLease } from "@paperclipai/shared"; + +export function workspaceAllocationFiles(lease: EnvironmentLease, remoteCwd: string) { + const raw = lease.metadata?.workspaceAllocationFiles; + if (raw === undefined) return []; + const receipt = raw as PluginEnvironmentWorkspaceAllocationFiles | null; + if (!receipt || receipt.providerLeaseId !== lease.providerLeaseId || receipt.remoteCwd !== remoteCwd + || !Array.isArray(receipt.files) || receipt.files.length > 8 + || receipt.files.some(file => !file || typeof file.relativePath !== "string" + || !/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath) + || file.relativePath === ".paperclip-runtime/workspace-restored.json" + || typeof file.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(file.sha256)) + || new Set(receipt.files.map(file => file.relativePath)).size !== receipt.files.length) { + throw new Error("remote_workspace_allocation_receipt_invalid"); + } + return receipt.files; +} diff --git a/server/src/services/remote-workspace-migration.ts b/server/src/services/remote-workspace-migration.ts index f95ed723b3f..9ffc19581b4 100644 --- a/server/src/services/remote-workspace-migration.ts +++ b/server/src/services/remote-workspace-migration.ts @@ -22,6 +22,8 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, LEGACY_WORKSPACE_RECOVERY_MIGRATION, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js"; import { offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers } from "./remote-workspace-migration-tree.js"; +import { inspectNativeWorkspaceMigration, nativeWorkspaceMigrationMarker } from "./native-runtime/native-workspace-migration.js"; + const SCHEMA = "paperclip.remote-workspace-migration.v1"; type Workspace = typeof executionWorkspaces.$inferSelect; type ReadDb = Pick; @@ -233,6 +235,7 @@ export async function migrateRemoteWorkspace(input: { const current = await inspect(tx as unknown as Db, input); if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null, legacyRecoverySources: [] as LegacyWorkspaceRecoverySource[] }; if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`); + const nativeSessionContinuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace); const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source); const sourceFingerprint = fingerprint(workspace); const marker = current.marker; @@ -241,6 +244,7 @@ export async function migrateRemoteWorkspace(input: { const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId), eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration"))); if (!operation) throw new Error("remote_workspace_migration_operation_missing"); + if (canonical(operation.metadata?.nativeSessionContinuity ?? null) !== canonical(nativeSessionContinuity)) throw new Error("native_workspace_migration_source_changed"); // Older operations only admitted the task's current workspace. Preserve // that binding when resuming an import journaled before this field existed. const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding") @@ -254,6 +258,7 @@ export async function migrateRemoteWorkspace(input: { const [operation] = await tx.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id, issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running", metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding, + ...(nativeSessionContinuity ? { nativeSessionContinuity } : {}), nestedWorkspaceScope: { version: 1, state: "pending" }, ...(current.legacyRecoverySources.length ? { legacyRecoverySources: current.legacyRecoverySources } : {}), ...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}), @@ -347,9 +352,12 @@ export async function migrateRemoteWorkspace(input: { || saved.sourceOperationId !== child.operationId) throw new Error("nested_workspace_snapshot_changed"); } if (canonical(validation.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)) throw new Error("remote_workspace_migration_source_changed"); + const continuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace); + if (canonical(continuity) !== canonical(operation.metadata?.nativeSessionContinuity ?? null)) throw new Error("native_workspace_migration_source_changed"); const authority: RemoteWorkspaceAuthority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: ref.id, source: reserved.source, location: null, writerRunId: null, dirty: false }; const metadata: Record = { ...workspace.metadata, remoteWorkspace: authority }; delete metadata.remoteWorkspaceMigration; + if (continuity) metadata.nativeWorkspaceMigration = nativeWorkspaceMigrationMarker(operation.id); if (reserved.legacyRecoverySources.length) { metadata.legacyWorkspaceRecoveryMigration = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: operation.id }; for (const source of reserved.legacyRecoverySources) { diff --git a/server/src/services/remote-workspace-transport.test.ts b/server/src/services/remote-workspace-transport.test.ts index 523e8c40848..642272c4d22 100644 --- a/server/src/services/remote-workspace-transport.test.ts +++ b/server/src/services/remote-workspace-transport.test.ts @@ -161,6 +161,53 @@ describe("remote workspace transport safety with real sandbox commands", () => { } finally { await captured.cleanup(); } }); + it.each(["owned", "changed", "unrelated", "symlink", "interrupted"])("preserves only verified allocation files during a migrated restore: %s", async scenario => { + await fs.writeFile(path.join(workspace, "saved.txt"), "retained user work"); + const { remote, execute } = target(); + const captured = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() }); + try { + const root = path.join(scratch, "allocated"), relativePath = ".paperclip-runtime/reusable-sandbox-lease.json"; + const bytes = JSON.stringify({ version: 1, token: "exact-allocation-token", provider: "daytona" }); + await fs.mkdir(path.join(root, ".paperclip-runtime"), { recursive: true }); + const sentinel = path.join(root, relativePath); + await fs.writeFile(sentinel, bytes); + const input = { target: remote, remoteCwd: root, archivePath: captured.archivePath, manifestDigest: captured.manifestDigest, + restorationIdentity: "d".repeat(64), allowEmptyRoot: true, + allocationFiles: [{ relativePath, sha256: createHash("sha256").update(bytes).digest("hex") }] }; + if (scenario === "changed") await fs.writeFile(sentinel, "foreign marker"); + if (scenario === "unrelated") await fs.writeFile(path.join(root, ".paperclip-runtime", "unrelated.txt"), "keep"); + if (scenario === "symlink") { + const outside = path.join(scratch, "outside-marker"); await fs.writeFile(outside, bytes); + await fs.unlink(sentinel); await fs.symlink(outside, sentinel); + } + if (["changed", "unrelated", "symlink"].includes(scenario)) { + await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow(); + expect(await fs.readFile(sentinel, "utf8")).toBe(scenario === "changed" ? "foreign marker" : bytes); + expect(await fs.lstat(path.join(root, "saved.txt")).catch(() => null)).toBeNull(); + return; + } + if (scenario === "interrupted") { + const run = execute.getMockImplementation()!; let interrupt = true; + execute.mockImplementation(async command => { + if (interrupt && command.args?.[1]?.includes("remote_workspace_restore_failed")) { + interrupt = false; + return run({ ...command, args: command.args.map((arg, index) => index === 1 + ? arg.replace("try{await fs.rename(staged,root);}", "throw Error('interrupted');try{await fs.rename(staged,root);}") : arg) }); + } + return run(command); + }); + await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow(); + expect(await fs.readFile(path.join(root + ".allocation-" + input.restorationIdentity, relativePath), "utf8")).toBe(bytes); + } + await restoreRemoteWorkspaceDirectory(input); + expect(await fs.readFile(sentinel, "utf8")).toBe(bytes); + expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("retained user work"); + await fs.writeFile(path.join(root, "saved.txt"), "later user edits"); + await restoreRemoteWorkspaceDirectory(input); + expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("later user edits"); + } finally { await captured.cleanup(); } + }); + it("keeps a completed restore available when the staging cleanup reply is lost", async () => { await fs.writeFile(path.join(workspace, "saved.txt"), "durable synthetic files\n"); const { remote, execute } = target(); diff --git a/server/src/services/remote-workspace-transport.ts b/server/src/services/remote-workspace-transport.ts index fc95be3b5db..96cc69f0b65 100644 --- a/server/src/services/remote-workspace-transport.ts +++ b/server/src/services/remote-workspace-transport.ts @@ -279,7 +279,8 @@ export async function captureRemoteWorkspaceArchive(input: { } export async function restoreRemoteWorkspaceDirectory(input: { - target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string; + target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string; allowEmptyRoot?: boolean; + allocationFiles?: Array<{ relativePath: string; sha256: string }>; }) { if (!/^[a-f0-9]{64}$/.test(input.restorationIdentity)) throw new Error("remote_workspace_restore_identity_invalid"); const id = input.restorationIdentity; @@ -292,25 +293,54 @@ export async function restoreRemoteWorkspaceDirectory(input: { }]); // Publish the restored folder and its identity together. A lost successful // reply can then adopt the same files without repeating a destructive restore. - const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'); - (async()=>{const [helper,archive,root,digest,identity]=process.argv.slice(1); - if(!/^[a-f0-9]{64}$/.test(identity))throw Error('invalid_identity'); - const prior=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e}); - if(prior){ - if(!prior.isDirectory()||prior.isSymbolicLink()||await fs.realpath(root)!==root)throw Error('unsafe_root'); - const runtime=path.join(root,'.paperclip-runtime'),marker=path.join(runtime,'workspace-restored.json'); - const dir=await fs.lstat(runtime);if(!dir.isDirectory()||dir.isSymbolicLink())throw Error('unsafe_receipt'); - const file=await fs.lstat(marker);if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt'); + const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'),crypto=require('node:crypto'); + (async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles]=process.argv.slice(1),files=JSON.parse(rawFiles); + if(!/^[a-f0-9]{64}$/.test(identity)||!Array.isArray(files)||files.length>8)throw Error('invalid_identity'); + for(const file of files)if(!/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath) + ||file.relativePath==='.paperclip-runtime/workspace-restored.json'||!/^[a-f0-9]{64}$/.test(file.sha256))throw Error('invalid_allocation'); + if(new Set(files.map(file=>file.relativePath)).size!==files.length)throw Error('invalid_allocation'); + const stat=async p=>fs.lstat(p).catch(e=>{if(e.code==='ENOENT')return null;throw e}); + const directory=async p=>{const s=await fs.lstat(p);if(!s.isDirectory()||s.isSymbolicLink()||await fs.realpath(p)!==p)throw Error('unsafe_root')}; + const allocation=async p=>{ + await directory(p);const names=await fs.readdir(p); + if(names.length===0&&files.length===0)return []; + if(names.length!==1||names[0]!=='.paperclip-runtime'||!files.length)throw Error('occupied_root'); + const runtime=path.join(p,'.paperclip-runtime');await directory(runtime); + const contents=await fs.readdir(runtime),expected=files.map(file=>path.basename(file.relativePath)); + if(contents.length!==expected.length||contents.some(name=>!expected.includes(name)))throw Error('occupied_root'); + return Promise.all(files.map(async file=>{const name=path.join(p,file.relativePath),s=await fs.lstat(name); + if(!s.isFile()||s.isSymbolicLink()||s.nlink!==1||s.size>16384)throw Error('unsafe_allocation'); + const content=await fs.readFile(name);if(crypto.createHash('sha256').update(content).digest('hex')!==file.sha256)throw Error('allocation_changed'); + return {name:path.basename(name),content};})); + }; + const parked=root+'.allocation-'+identity; + if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);} + const prior=await stat(root);if(prior)await directory(root); + const marker=path.join(root,'.paperclip-runtime','workspace-restored.json'); + if(prior&&await stat(marker)){ + await directory(path.dirname(marker));const file=await fs.lstat(marker); + if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt'); const receipt=JSON.parse(await fs.readFile(marker,'utf8')); - if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');return; + if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed'); + if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}return; } + if(prior&&allowEmpty!=='true')throw Error('occupied_root'); + const retained=prior?await allocation(root):[]; + if(!prior&&files.length)throw Error('allocation_missing'); const staged=await fs.mkdtemp(root+'.restoring-'); - await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576}); - const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700}); - await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600}); - if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');await fs.rename(staged,root); + try{ + await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576}); + const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700}); + for(const file of retained)await fs.writeFile(path.join(runtime,file.name),file.content,{flag:'wx',mode:0o600}); + await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600}); + if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root'); + if(prior){await allocation(root);if(await stat(parked))throw Error('occupied_parking');await fs.rename(root,parked);} + try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(parked,root);throw e;} + if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});} + }finally{await fs.rm(staged,{recursive:true,force:true});} })().catch(()=>{process.stderr.write('remote_workspace_restore_failed');process.exitCode=1});`; - await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd, input.manifestDigest, input.restorationIdentity]); + await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd, + input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? [])]); // Publish the restored location before cleanup can fail, stall, or lose its // reply. The owner journals this receipt and then reclaims disposable staging. return { operationId, stagingRoot: parent, cleanupPending: true }; diff --git a/server/src/services/remote-workspaces.integration.test.ts b/server/src/services/remote-workspaces.integration.test.ts index 566934be7dd..c5eeef78d43 100644 --- a/server/src/services/remote-workspaces.integration.test.ts +++ b/server/src/services/remote-workspaces.integration.test.ts @@ -1,4 +1,7 @@ -import { randomUUID } from "node:crypto"; +import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js"; +import { readNativeWorkspaceMigration, migratedNativeExecutionCwd, canRestoreMigratedNativeSession } from "./native-runtime/native-workspace-migration.js"; +import { buildNativeHarnessBackupManifest, verifyNativeHarnessBackup } from "./native-runtime/native-session-executor.js"; +import { createHash, randomUUID } from "node:crypto"; import { execFile, spawn } from "node:child_process"; import net from "node:net"; import { promisify } from "node:util"; @@ -8,7 +11,7 @@ import path from "node:path"; import { pathToFileURL } from "node:url"; import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest"; import { and, eq, sql } from "drizzle-orm"; -import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments, +import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments, executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db"; import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target"; import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime"; @@ -90,6 +93,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, ids.company)); await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, ids.company)); await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, ids.company)); + await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, ids.company)); await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, ids.company)); await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, ids.company)); await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.companyId, ids.company)); @@ -437,6 +441,101 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem try { expect(await fs.readFile(path.join(browse.root, "report.md"), "utf8")).toContain("useful saved result"); } finally { await browse.cleanup(); } }, 60_000); + it.each(["per_turn", "warm", "occupied", "foreign-target", "foreign-allocation", "forged-receipt"])("preserves retained native workspace identity through migration: %s", async scenario => { + const original = path.join(root, "old-controller", "workspace"); await fs.mkdir(original, { recursive: true }); + await fs.writeFile(path.join(original, "saved.txt"), "retained work"); + const remote = await target("legacy-provider-root"); + const [local] = await db.insert(executionWorkspaces).values({ companyId: ids.company, sourceIssueId: ids.issue, + mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Retained session", + cwd: original, providerRef: original }).returning(); + const sessionId = randomUUID(), runnerInstanceId = randomUUID(); + const execution = parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1", + provider: { kind: "codex", model: null }, binding: { companyId: ids.company, agentId: ids.agent, + issueId: ids.issue, runId: ids.run, executionWorkspaceId: local.id }, + task: { identifier: "RWP-1", title: "Retained work", description: null, prompt: "Continue.", workMode: "standard" }, + workspace: { cwd: original, repoUrl: null, repoRef: null, branchName: null }, + session: { normalizedSessionId: sessionId, driverKind: "codex_app_server", protocolVersion: 1, + lifecyclePolicy: { mode: scenario === "warm" ? "warm" : "per_turn", idleTimeoutMs: scenario === "warm" ? 120000 : null } }, + completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1", + contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } }, + credentialBindings: [], interactionResponses: [], + }); + if (execution.schema !== "paperclip.native-execution-input.v1") throw new Error("Unexpected fixture schema"); + await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), nativeSessionId: sessionId, + runnerInstanceId, runnerProfileJson: { nativeExecutionInput: execution, sessionCheckpoint: { sessionId }, nativeWorkspaceSync: { + schema: "paperclip.native-workspace-sync/v3", state: "finalized", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64), + finalHostSha256: "c".repeat(64), workspaceId: local.id, leaseId: ids.lease, providerLeaseId: `allocation-${ids.lease}`, + remoteCwd: remote.remoteCwd, resourceDisposition: "destroy", + } } }).where(eq(heartbeatRuns.id, ids.run)); + await db.insert(agentTaskSessions).values({ companyId: ids.company, agentId: ids.agent, adapterType: "paperclip_runner", + taskKey: ids.issue, lastRunId: ids.run }); + await db.update(issues).set({ executionWorkspaceId: local.id }).where(eq(issues.id, ids.issue)); + const backupRoot = path.join(root, "private-state"), current = path.join(backupRoot, "failover-backups", "current"); + await fs.mkdir(path.join(current, "runner"), { recursive: true }); await fs.mkdir(path.join(current, "codex-home")); + await fs.writeFile(path.join(current, "runner", "state.json"), "unchanged opaque provider state"); + await fs.writeFile(path.join(current, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({ backupRoot: current, + execution, runnerInstanceId, sourceProviderLeaseId: "retained-allocation", providerSessionIdentity: { + providerSessionId: "retained-thread", providerBackendSessionId: "retained-backend", providerSessionIdentity: null } }))); + expect(verifyNativeHarnessBackup({ root: backupRoot, execution, runnerInstanceId })).not.toBeNull(); + await migrateRemoteWorkspace({ db, storage, companyId: ids.company, workspaceId: local.id, source: { kind: "empty" }, apply: true }); + const migrated = await workspace(local.id); + const receipt = await readNativeWorkspaceMigration(db, migrated); + expect(receipt?.remoteCwd).toBe(remote.remoteCwd); + const continuation = { companyId: ids.company, workspaceId: local.id, agentId: ids.agent, issueId: ids.issue, sessionId, logicalCwd: original, remoteCwd: remote.remoteCwd }; + expect(await canRestoreMigratedNativeSession(db, continuation)).toBe(true); + expect(await canRestoreMigratedNativeSession(db, { ...continuation, sessionId: randomUUID() })).toBe(false); + expect(await canRestoreMigratedNativeSession(db, { ...continuation, remoteCwd: "/foreign" })).toBe(false); + expect(migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId })).toBe(original); + expect(migratedNativeExecutionCwd(receipt, { agentId: randomUUID(), issueId: ids.issue, sessionId })).toBeNull(); + await fs.rm(path.join(root, "old-controller"), { recursive: true }); + ids.run = randomUUID(); + await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" }); + await db.update(environmentLeases).set({ heartbeatRunId: ids.run }).where(eq(environmentLeases.id, ids.lease)); + const reserved = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId: ids.run, + agentId: ids.agent, environmentId: ids.environment, existingWorkspaceId: local.id, projectId: null, projectWorkspaceId: null, source: { kind: "empty" } }); + const preparation = { db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: remote }; + const allocationPath = ".paperclip-runtime/reusable-sandbox-lease.json"; + const allocationBytes = JSON.stringify({ version: 1, token: "owned-allocation", companyId: ids.company, + environmentId: ids.environment, agentId: ids.agent, executionWorkspaceId: local.id, provider: "daytona" }); + await fs.mkdir(path.join(remote.remoteCwd, ".paperclip-runtime")); + await fs.writeFile(path.join(remote.remoteCwd, allocationPath), allocationBytes); + preparation.lease.metadata = { ...preparation.lease.metadata, workspaceAllocationFiles: { + providerLeaseId: preparation.lease.providerLeaseId, remoteCwd: remote.remoteCwd, + files: [{ relativePath: allocationPath, sha256: createHash("sha256").update(allocationBytes).digest("hex") }], + } }; + + if (scenario === "forged-receipt") { + await db.update(executionWorkspaces).set({ metadata: { ...reserved.metadata, + nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId: randomUUID() } } }).where(eq(executionWorkspaces.id, local.id)); + await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_receipt_invalid"); return; + } + if (scenario === "occupied") { + await fs.writeFile(path.join(remote.remoteCwd, "unrelated.txt"), "do not overwrite"); + await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow(); + expect(await fs.readFile(path.join(remote.remoteCwd, "unrelated.txt"), "utf8")).toBe("do not overwrite"); return; + } + if (scenario === "foreign-allocation") { + const receipt = preparation.lease.metadata.workspaceAllocationFiles as { providerLeaseId: string }; + receipt.providerLeaseId = "another-allocation"; + await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("remote_workspace_allocation_receipt_invalid"); + expect(await fs.readFile(path.join(remote.remoteCwd, allocationPath), "utf8")).toBe(allocationBytes); return; + } + if (scenario === "foreign-target") { + remote.remoteCwd = path.join(root, "different-provider-root"); + await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_target_changed"); return; + } + const prepared = await prepareRemoteWorkspace(preparation); + expect(prepared.remoteRoot).toBe(remote.remoteCwd); + expect(await fs.readFile(path.join(prepared.remoteRoot, allocationPath), "utf8")).toBe(allocationBytes); + expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("retained work"); + const successor = { ...execution, workspace: { ...execution.workspace, + cwd: migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId }) ?? prepared.remoteRoot } }; + expect(verifyNativeHarnessBackup({ root: backupRoot, execution: successor, runnerInstanceId })).not.toBeNull(); + expect(verifyNativeHarnessBackup({ root: backupRoot, execution: { ...successor, workspace: { ...successor.workspace, + cwd: prepared.remoteRoot } }, runnerInstanceId })).toBeNull(); + expect(await fs.readFile(path.join(current, "runner", "state.json"), "utf8")).toBe("unchanged opaque provider state"); + }, 60_000); + it.each(["path", "file-url"] as const)("cold restores an imported secondary %s repository and acquires only a new descriptor", async sourceKind => { const origin = path.join(root, "import-origin"), original = path.join(root, "old-checkout"); await fs.mkdir(origin); await fs.mkdir(original); diff --git a/server/src/services/remote-workspaces.ts b/server/src/services/remote-workspaces.ts index d651a716096..6efda0aa5af 100644 --- a/server/src/services/remote-workspaces.ts +++ b/server/src/services/remote-workspaces.ts @@ -1,3 +1,5 @@ +import { workspaceAllocationFiles } from "./remote-workspace-allocation.js"; +import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js"; import { createHash, randomUUID } from "node:crypto"; import { existsSync } from "node:fs"; import { mkdtemp, rm, statfs } from "node:fs/promises"; @@ -352,7 +354,10 @@ export async function prepareRemoteWorkspace(input: { const sameAllocation = initial.location?.providerLeaseId === input.lease.providerLeaseId && initial.location?.environmentId === input.lease.environmentId; if (initial.location && !sameAllocation && initial.dirty) throw conflict("The prior workspace has unsaved work; recover it before replacing its allocation."); - const remoteRoot = sameAllocation ? initial.location!.remoteCwd : path.posix.join(target.remoteCwd, ".paperclip-workspaces", + const migration = input.workspace.metadata?.nativeWorkspaceMigration + ? await readNativeWorkspaceMigration(input.db, input.workspace) : null; + if (migration && migration.remoteCwd !== target.remoteCwd) throw new Error("native_workspace_migration_target_changed"); + const remoteRoot = sameAllocation ? initial.location!.remoteCwd : migration?.remoteCwd ?? path.posix.join(target.remoteCwd, ".paperclip-workspaces", publishedWorkspaceDirectory(input.workspace, initial) ?? `${input.workspace.id}-${initial.epoch}-${input.lease.id}`); const preparationIdentity = createHash("sha256").update(JSON.stringify([input.workspace.companyId, input.workspace.id, input.runId, initial.epoch, input.lease.id, input.lease.providerLeaseId, remoteRoot, initial.source])).digest("hex"); @@ -369,7 +374,8 @@ export async function prepareRemoteWorkspace(input: { const archivePath = path.join(scratch, "workspace.pcws"); await loadWorkspaceSnapshotArchive({ storage: input.storage ?? getStorageService(), reference: snapshot, targetPath: archivePath }); restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest, - restorationIdentity: preparationIdentity }); + restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null, + allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] }); } finally { await rm(scratch, { recursive: true, force: true }); } } else { try { diff --git a/server/src/services/skill-package-storage-migration.ts b/server/src/services/skill-package-storage-migration.ts new file mode 100644 index 00000000000..9fbd967cabc --- /dev/null +++ b/server/src/services/skill-package-storage-migration.ts @@ -0,0 +1,66 @@ +import { createReadStream, createWriteStream } from "node:fs"; +import { promises as fs } from "node:fs"; +import { randomUUID } from "node:crypto"; +import os from "node:os"; +import path from "node:path"; +import { pipeline } from "node:stream/promises"; +import { Writable } from "node:stream"; +import { and, asc, eq, gt, sql } from "drizzle-orm"; +import { companySkillRepositorySnapshots as snapshots, type Db, type SkillPackageStorageReceipt } from "@paperclipai/db"; +import type { StorageService } from "../storage/index.js"; +import { boundedHashStream } from "./skill-package-files.js"; +import { skillPackageStore } from "./skill-package-store.js"; + +export async function migrateLocalSkillPackageObjects(db: Db, input: { + source: StorageService; destination: StorageService; writersStopped: boolean; signal: AbortSignal; after?: string; +}) { + if (!input.writersStopped || process.env.PAPERCLIP_EPHEMERAL_MIGRATION !== "true" + || input.source.provider !== "local_disk" || input.destination.provider !== "s3") throw new Error("offline_skill_object_migration_required"); + const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, storage: snapshots.storage }) + .from(snapshots).where(and(sql`${snapshots.storage}->>'provider' = 'local_disk'`, + input.after ? gt(snapshots.id, input.after) : undefined)).orderBy(asc(snapshots.id)).limit(50); + const source = skillPackageStore(db, input.source); + for (const row of rows) { + input.signal.throwIfAborted(); + const original = row.storage!; + if (!await source.verifyDurable(row.companyId, row.id, input.signal)) throw new Error("local_skill_object_verification_failed"); + const token = randomUUID(), prefix = `${row.companyId}/skill-packages/${token}`; + const upload = { token, legacy: true, provider: "s3" as const, archiveKey: `${prefix}/files.tar.gz`, + manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() }; + const unchanged = sql`${snapshots.storage} = ${JSON.stringify(original)}::jsonb`; + const [claimed] = await db.update(snapshots).set({ upload }).where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged, + sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning({ id: snapshots.id }); + if (!claimed) throw new Error("skill_object_migration_owner_busy"); + const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-object-migration-")); + try { + for (const object of [ + { source: original.archiveKey, target: upload.archiveKey, bytes: original.archiveBytes, sha256: original.archiveSha256, + filename: "files.tar.gz", contentType: "application/gzip" }, + { source: original.manifestKey, target: upload.manifestKey, bytes: original.manifestBytes, sha256: original.manifestSha256, + filename: "manifest.json", contentType: "application/json" }, + ]) { + const filename = path.join(directory, object.filename); + const read = await input.source.getObject(row.companyId, object.source, { signal: input.signal }); + const originalDigest = boundedHashStream(object.bytes); + await pipeline(read.stream, originalDigest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal: input.signal }); + const actual = originalDigest.result(); + if (actual.bytes !== object.bytes || actual.sha256 !== object.sha256) throw new Error("local_skill_object_changed"); + await input.destination.putFile({ companyId: row.companyId, namespace: "skill-packages", objectKey: object.target, + originalFilename: object.filename, contentType: object.contentType, body: createReadStream(filename), + byteSize: object.bytes, sha256: object.sha256, signal: input.signal }); + const saved = await input.destination.getObject(row.companyId, object.target, { signal: input.signal }); + const remoteDigest = boundedHashStream(object.bytes); + await pipeline(saved.stream, remoteDigest.stream, new Writable({ write(_chunk, _encoding, done) { done(); } }), { signal: input.signal }); + const verified = remoteDigest.result(); + if (verified.bytes !== object.bytes || verified.sha256 !== object.sha256) throw new Error("remote_skill_object_verification_failed"); + } + input.signal.throwIfAborted(); + const receipt: SkillPackageStorageReceipt = { ...original, provider: "s3", archiveKey: upload.archiveKey, manifestKey: upload.manifestKey }; + const [published] = await db.update(snapshots).set({ storage: receipt, upload: null, migrationAt: null }) + .where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged, + sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id }); + if (!published) throw new Error("skill_object_migration_owner_changed"); + } finally { await fs.rm(directory, { recursive: true, force: true }); } + } + return { inspected: rows.length, migrated: rows.length, after: rows.at(-1)?.id ?? null }; +} diff --git a/tests/runner-e2e/README.md b/tests/runner-e2e/README.md index 4974e9ed25f..b7c0a0110b9 100644 --- a/tests/runner-e2e/README.md +++ b/tests/runner-e2e/README.md @@ -3492,3 +3492,19 @@ separate BROKEN gap shape and must not be used for this recovery bundle. Final coverage must match all selected case IDs and preserve the separate gap count. This mode prepares data only. It does not upload, start jobs, reopen launches, change credentials, or change evaluation and evidence limits. +# Legacy workspace migration + +`legacy-workspace-migration` and `legacy-workspace-migration-warm` are explicit, +paid, two-turn Daytona workflows for registered native production profiles. They +create a private reference file and ask a question on a local-disk controller. +The supervisor stops that controller, runs the packaged offline migration with +owned PostgreSQL and S3, verifies the export, starts an empty-home controller, +and deletes the original home. The browser answers the existing question. The +oracle requires the same native and provider session, workspace addresses and +file bytes, plus the exact final document. Provider session identifiers are +hashed in the added evidence. The company budget is $10 and automatic retries +are disabled. Existing fixture teardown owns sandboxes, credentials, processes, +database and storage; failed private evidence is retained under the usual policy. + +This proves preservation during initial enrollment, which has an offline step. +Use the separate rolling deployment suites for uninterrupted deployment proof. diff --git a/tests/runner-e2e/catalog.test.ts b/tests/runner-e2e/catalog.test.ts index 32c66c5450c..044865f6cae 100644 --- a/tests/runner-e2e/catalog.test.ts +++ b/tests/runner-e2e/catalog.test.ts @@ -312,11 +312,11 @@ describe("runner E2E catalog", () => { expect(localIntegrityTasks).toHaveLength(2); expect(openRouterBreadthTasks).toHaveLength(3); expect(runnerSuites.map((suite) => suite.expectedMatrixSize)).toEqual([ - 1, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, + 1, 9, 9, 9, 9, 4, 2, 1, 3, 2, 2, 4, 4, 4, 4, 4, 4, 8, 10, 6, 63, 12, 6, 4, 2, 4, 2, 30, 3, 16, 16, 20, 2, 1, 6, 8, 46, 2, 23, 15, 52, 6, 6, 20, 26, 52, 5, 28, 18, 2, 6, 6, 13, 10, 4, 48, 16, 10, 2, 1, 1, 4, 116, 2, 1, 2, ]); - expect(validateRunnerCatalog()).toHaveLength(819); - expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(819); + expect(validateRunnerCatalog()).toHaveLength(837); + expect(new Set(runnerMatrix.map((entry) => entry.id)).size).toBe(837); expect( runnerMatrix.filter((entry) => entry.suite.id === "core-compatibility"), ).toHaveLength(48); diff --git a/tests/runner-e2e/catalog.ts b/tests/runner-e2e/catalog.ts index ecf24df00fd..18c1151aaf3 100644 --- a/tests/runner-e2e/catalog.ts +++ b/tests/runner-e2e/catalog.ts @@ -1272,6 +1272,21 @@ const rollingDeploymentSuites: RunnerSuiteFixture[] = [ export const runnerSuites: readonly RunnerSuiteFixture[] = [ remoteWorkspaceStorageSuite, ...rollingDeploymentSuites, + ...[ + { id: "legacy-workspace-migration", environment: runnerEnvironments.find(environment => environment.id === "daytona")! }, + { id: "legacy-workspace-migration-warm", environment: daytonaControllerOutageEnvironment }, + ].map(({ id, environment }): RunnerSuiteFixture => ({ + id, label: `Legacy native session migration (${environment.configurationKey ? "warm" : "per-turn"})`, manualOnly: true, + description: "Resume an existing native session and workspace after the production offline migration and deletion of the original controller home.", + groups: [], profiles: rollingDeploymentProfiles, environments: [environment], + tasks: [{ ...necessaryQuestionTask, minimumExpectedRunCount: 2, turnTimeoutMs: 6 * 60_000, + attemptTimeoutMs: { local: 15 * 60_000, daytona: 20 * 60_000 } }], + expectedMatrixSize: rollingDeploymentProfiles.length, + definitionMetadata: { version: 1, scheduling: "explicit-only", maximumAttemptsPerCell: 1, providerTurns: 2, budgetMonthlyCents: 1000, + lifecycleCoverage: environment.configurationKey ? "warm-reusable" : "per-turn-disposable", + migration: "packaged-offline-entry-verified-export-delete-entire-source-home", authentication: "api-key", stagingQualification: false, + grading: "same-native-and-provider-session-logical-and-remote-paths-reference-bytes-question-and-final-document" }, + })), { id: CURSOR_CLOUD_SUITE, label: "Native Cursor Cloud", manualOnly: true, groups: [], description: "Local Paperclip controller with hosted Cursor work: verified artifact, same-agent follow-up, real active-run Stop and semantic plan revision/approval.", diff --git a/tests/runner-e2e/continuation-flow.ts b/tests/runner-e2e/continuation-flow.ts index 3da0668ebae..f93853b0c9a 100644 --- a/tests/runner-e2e/continuation-flow.ts +++ b/tests/runner-e2e/continuation-flow.ts @@ -1,4 +1,5 @@ import { gradeNecessaryQuestion } from "./necessary-question.js"; +import { gradeLegacyWorkspaceMigration, migrationPrimaryOutputCheckpoints, isLegacyWorkspaceMigration, legacyWorkspaceMigrationPrompt, readMigrationPoint, type MigrationPoint } from "./legacy-workspace-migration.js"; import { necessaryQuestionContract } from "./continuation-cases.js"; import { validResumeQuestionForm, validResumeReferenceForm } from "./question-resume-scoring.js"; import { answerResumeQuestionThroughUi } from "./question-resume-ui.js"; @@ -52,6 +53,8 @@ export async function runContinuationFlow(input: { evidence(name: string, value: unknown): Promise; }) { const { page, api, fixtures, execution } = input; + const migration = isLegacyWorkspaceMigration(execution); + let migrationBefore: MigrationPoint | undefined, migrationAfter: MigrationPoint | undefined; const lifecycleProbe = lifecycleLiveCase(execution.task.id); const scenario = lifecycleProbe ? lifecycleLiveContinuation(execution.task.id, input.nonce) @@ -90,6 +93,7 @@ export async function runContinuationFlow(input: { const idle = continuationAnswerCommitted(state.interactions, answeredInteractionId) && continuationCheckpointReady({ + requireCompletedTurn: migration, issue: { status: state.issue.status, executionRunId: state.issue.executionRunId }, runs: state.runs.map(r => ({ id: r.id, status: r.status, runtimeMode: r.runtimeMode })), interactions: state.interactions, @@ -256,7 +260,7 @@ export async function runContinuationFlow(input: { issuePrefix: fixtures.company.issuePrefix!, agentName: fixtures.agent.name, title: execution.task.buildTitle(input.nonce), - prompt: scenario.prompt, + prompt: migration ? legacyWorkspaceMigrationPrompt(scenario.prompt) : scenario.prompt, workMode: "standard", }); issue = await pollUntil({ @@ -272,6 +276,12 @@ export async function runContinuationFlow(input: { await settle(new Set(), scenario.id !== "revision-preserves-approval"); await snapshot("initial"); assertWaiting(); + if (migration) { + migrationBefore = await readMigrationPoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue.id }); + await input.evidence("legacy-migration-before.json", migrationBefore); + await input.restart(); + await open(); + } if (scenario.id === "untrusted-evidence") { const parentRun = runs.find((r) => r.contextSnapshot?.issueId === issue!.id); await seedContinuationContext({ @@ -300,14 +310,19 @@ export async function runContinuationFlow(input: { assertWaiting(); await reply(scenario.approval); } + if (migration) migrationAfter = await readMigrationPoint({ api, companyId: fixtures.company.id, agentId: fixtures.agent.id, issueId: issue!.id }); await snapshot("final"); } finally { checks = gradeContinuation({ ...scenario, - checkpoints, + checkpoints: migration ? migrationPrimaryOutputCheckpoints(checkpoints, migrationBefore, migrationAfter) : checkpoints, runtimeMode: execution.profile.expectedRuntimeMode, }); checks.push(...gradeLifecycleBaseline(checkpoints)); + if (migration) { + checks.push(...gradeLegacyWorkspaceMigration(migrationBefore, migrationAfter, checkpoints)); + await input.evidence("legacy-migration.json", { before: migrationBefore, after: migrationAfter, checks: gradeLegacyWorkspaceMigration(migrationBefore, migrationAfter, checkpoints) }); + } if (scenario.id === "necessary-free-text") checks.push(...gradeNecessaryQuestion(checkpoints, scenario.answer)); if (lifecycleProbe) checks.push(gradeLifecycleNarrative({ narrative: lifecycleProbe.narrative, diff --git a/tests/runner-e2e/continuation-readiness.test.ts b/tests/runner-e2e/continuation-readiness.test.ts index 6850f9156ab..1945681d5d2 100644 --- a/tests/runner-e2e/continuation-readiness.test.ts +++ b/tests/runner-e2e/continuation-readiness.test.ts @@ -51,6 +51,12 @@ describe("checkpoint finalization boundary", () => { runs: [{ ...run, status: "running" }], interactions: [{ ...pending[0], sourceRunId: "run", payload: { runtimeRequestId: "provider-request" } }] }; expect(continuationCheckpointReady(input)).toBe(true); + expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true })).toBe(false); + expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true, + runs: [run], issue: { status: "in_review", executionRunId: "run" } })).toBe(false); + expect(continuationCheckpointReady({ ...input, requireCompletedTurn: true, + runs: [run], issue: { status: "in_review", executionRunId: null } })).toBe(true); + expect(continuationCheckpointReady({ ...input, interactions: pending })).toBe(false); expect(continuationCheckpointReady({ ...input, issue: { ...input.issue, executionRunId: "other" } })).toBe(false); expect(continuationCheckpointReady({ ...input, runs: [{ ...input.runs[0], runtimeMode: "legacy" }] })).toBe(false); diff --git a/tests/runner-e2e/continuation-readiness.ts b/tests/runner-e2e/continuation-readiness.ts index 16845320935..dc1bd5e8b5e 100644 --- a/tests/runner-e2e/continuation-readiness.ts +++ b/tests/runner-e2e/continuation-readiness.ts @@ -13,6 +13,7 @@ export function continuationAnswerCommitted(interactions: ReadonlyArray<{ id?: u /** Run completion precedes task-lock release. Observe both before taking a * checkpoint, even when the terminal run status is stable across many polls. */ export function continuationCheckpointReady(input: { + requireCompletedTurn?: boolean; issue: { status: string; executionRunId?: string | null }; runs: ReadonlyArray<{ id: string; status: string; runtimeMode?: string }>; interactions: ReadonlyArray<{ id?: unknown; kind?: unknown; status?: unknown; @@ -25,7 +26,7 @@ export function continuationCheckpointReady(input: { const waiting = pending.some(i => ["ask_user_questions", "request_confirmation", "request_approval"].includes(String(i.kind))); return input.issue.executionRunId === null && (!waiting || input.issue.status === "in_review"); } - if (active.length !== 1) return false; + if (input.requireCompletedTurn || active.length !== 1) return false; const run = active[0]; return run.status === "running" && run.runtimeMode === "native" && input.issue.executionRunId === run.id && ["in_progress", "in_review"].includes(input.issue.status) && diff --git a/tests/runner-e2e/installed-cli.test.ts b/tests/runner-e2e/installed-cli.test.ts index a2ef8190188..17fe4787087 100644 --- a/tests/runner-e2e/installed-cli.test.ts +++ b/tests/runner-e2e/installed-cli.test.ts @@ -1,3 +1,4 @@ +import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js"; import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; @@ -8,7 +9,7 @@ import { afterEach, beforeEach, expect, it, vi } from "vitest"; import { runnerMatrix } from "./catalog.js"; import { assertInstalledCliSelection, installedCliKeys, probeInstalledCodexStartup, verifyInstalledCli } from "./installed-cli.js"; import { buildPaperclipServerEnvironment, buildRunnerE2EProcessEnvironment } from "./harness-env.js"; -const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-pi"); +const cells = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-pi"); const roots: string[] = []; // pnpm's source test wrapper sets NODE_PATH; the disposable public-install fixture must start closed. beforeEach(() => { vi.stubEnv("NODE_PATH", undefined); vi.stubEnv("NODE_OPTIONS", undefined); }); @@ -47,7 +48,7 @@ it("admits existing Codex onboarding, useful-file revision and restart journeys }); it("admits only the 20 agreed Copilot cells through normal installed resolution with no qualification override", async () => { const { env, server } = await fixture(); - const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot"); + const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-copilot"); expect(selected).toHaveLength(20); expect(selected.filter(e => e.environment.id === "local")).toHaveLength(10); expect(selected.filter(e => e.environment.id === "daytona")).toHaveLength(10); diff --git a/tests/runner-e2e/installed-daytona-plugin.test.ts b/tests/runner-e2e/installed-daytona-plugin.test.ts index 953a314c905..21e4bbacc33 100644 --- a/tests/runner-e2e/installed-daytona-plugin.test.ts +++ b/tests/runner-e2e/installed-daytona-plugin.test.ts @@ -1,3 +1,4 @@ +import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js"; import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { execFileSync } from "node:child_process"; import { createHash } from "node:crypto"; @@ -37,7 +38,7 @@ it("admits pinned published plugin and resolved SDK graph without claiming a ful }); it("admits all ten Copilot Daytona cells only with the pinned normal plugin graph", async () => { const f = await fixture(); - const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona"); + const selected = runnerMatrix.filter(e => !isRollingWebDeployment(e) && !isLegacyWorkspaceMigration(e) && e.profile.id === "runner-acpx-copilot" && e.environment.id === "daytona"); expect(selected).toHaveLength(10); for (const selectedCell of selected) { expect(await verifyInstalledDaytonaPlugin(f.env, [selectedCell])).toMatchObject({ packageRoot: f.authority.plugin.root, sourceFallback: false, fullGraphVerified: false, completeGraphAuthorityRequired: true }); diff --git a/tests/runner-e2e/legacy-workspace-migration-supervisor.ts b/tests/runner-e2e/legacy-workspace-migration-supervisor.ts new file mode 100644 index 00000000000..24abb54af16 --- /dev/null +++ b/tests/runner-e2e/legacy-workspace-migration-supervisor.ts @@ -0,0 +1,124 @@ +import { spawn, type ChildProcess } from "node:child_process"; +import { randomBytes, randomUUID } from "node:crypto"; +import { createWriteStream } from "node:fs"; +import { mkdir, readFile, rename, rm, writeFile } from "node:fs/promises"; +import path from "node:path"; +import { setTimeout as delay } from "node:timers/promises"; +import { paperclipConfigSchema } from "../../packages/shared/src/config-schema.js"; +import { makeDirectoryTreeRemovable } from "./directory-cleanup.js"; +import { runnerE2EServerControlPaths } from "./harness-env.js"; +import { reserveRunnerE2EServerPort } from "./ports.js"; +import { startRollingDeploymentProcess } from "./rolling-deployment-process.js"; +import { startRollingDeploymentProxy } from "./rolling-deployment-proxy.js"; +import { readRestartRequest } from "./server-restart-request.js"; +import { createRunnerE2EServerStopper } from "./server-stop.js"; +import { runnerE2ETypeScriptProcessArgs } from "./web-server-command.js"; + +type Server = Awaited>; + +export async function superviseLegacyWorkspaceMigration(input: { + temporaryRoot: string; repositoryRoot: string; configPath: string; port: number; + environment: NodeJS.ProcessEnv; executionIds: string[]; cancelled: () => boolean; +}) { + const config = paperclipConfigSchema.parse(JSON.parse(await readFile(input.configPath, "utf8"))); + if (input.executionIds.length !== 1 || config.database.mode !== "postgres" || !config.database.connectionString + || input.environment.PAPERCLIP_STORAGE_PROVIDER !== "s3") throw new Error("Migration qualification requires one owned Postgres/S3 cell"); + const databaseUrl = config.database.connectionString, controls = runnerE2EServerControlPaths(input.temporaryRoot); + const proxy = await startRollingDeploymentProxy(input.port), servers: Server[] = []; + const environment: NodeJS.ProcessEnv = { ...input.environment, + PAPERCLIP_DATABASE_LISTEN_URL: databaseUrl, PAPERCLIP_SECRETS_MASTER_KEY: randomBytes(32).toString("base64"), + HEARTBEAT_SCHEDULER_ENABLED: "true", PAPERCLIP_EPHEMERAL_SERVER: "true", + PAPERCLIP_DURABLE_AGENT_FILES: "true", PAPERCLIP_SHARED_UI_ASSETS: "true", + RUN_LOG_S3_BUCKET: input.environment.PAPERCLIP_STORAGE_S3_BUCKET, + RUN_LOG_S3_ENDPOINT: input.environment.PAPERCLIP_STORAGE_S3_ENDPOINT, + RUN_LOG_S3_REGION: input.environment.PAPERCLIP_STORAGE_S3_REGION, + RUN_LOG_S3_FORCE_PATH_STYLE: input.environment.PAPERCLIP_STORAGE_S3_FORCE_PATH_STYLE }; + const start = async (name: string, env: NodeJS.ProcessEnv) => { + const server = await startRollingDeploymentProcess({ root: input.temporaryRoot, name, repositoryRoot: input.repositoryRoot, + databaseUrl, ingressOrigin: proxy.origin, sharedEnvironment: env, browser: {} }); + servers.push(server); return server; + }; + const json = async (filename: string, value: unknown) => { + await mkdir(path.dirname(filename), { recursive: true }); + await writeFile(`${filename}.tmp`, JSON.stringify(value), { mode: 0o600 }); await rename(`${filename}.tmp`, filename); + }; + const settings = async (enabled: boolean) => { + const response = await fetch(`${proxy.origin}/api/instance/settings/experimental`, { method: "PATCH", + headers: { "content-type": "application/json", origin: proxy.origin }, + body: JSON.stringify({ remoteWorkspaceStorageEnabled: enabled, enableExperimentalFileViewer: true }), signal: AbortSignal.timeout(10_000) }); + if (!response.ok) throw new Error(`Migration fixture settings returned ${response.status}`); + await response.body?.cancel(); + }; + let maintenance: { stop(): Promise } | undefined; + try { + const keyPath = path.resolve(config.secrets.localEncrypted.keyFilePath); + if (!keyPath.startsWith(`${path.resolve(input.temporaryRoot)}${path.sep}`)) throw new Error("Migration fixture key escaped its root"); + await mkdir(path.dirname(keyPath), { recursive: true, mode: 0o700 }); + await writeFile(keyPath, environment.PAPERCLIP_SECRETS_MASTER_KEY!, { mode: 0o600, flag: "wx" }); + const legacyEnvironment: NodeJS.ProcessEnv = { ...environment, PAPERCLIP_EPHEMERAL_SERVER: "false", PAPERCLIP_DURABLE_AGENT_FILES: "false", + PAPERCLIP_SHARED_UI_ASSETS: "false", PAPERCLIP_STORAGE_PROVIDER: "local_disk" }; + for (const name of Object.keys(legacyEnvironment)) if (name.startsWith("RUN_LOG_S3_")) delete legacyEnvironment[name]; + let current = await start("legacy-controller", legacyEnvironment); + await current.waitForReady(); proxy.routeTo(current.origin); await settings(false); + let migrated = false, lastRequest: string | undefined; + while (!input.cancelled()) { + if (current.child.exitCode !== null || current.child.signalCode !== null) throw new Error("Migration controller exited unexpectedly"); + const request = await readRestartRequest(controls.restartRequestPath, input.executionIds); + if (!request || request.requestId === lastRequest) { await delay(100); continue; } + lastRequest = request.requestId; + try { + if (migrated || request.rollingDeployment || request.controllerOutage || request.preserveRunner || request.crashAfterResult) { + throw new Error("Legacy migration permits exactly one offline enrollment"); + } + await settings(true); + const previous = current; await previous.stop({ requireGraceful: true }); + const port = await reserveRunnerE2EServerPort(), token = randomBytes(32).toString("base64url"), id = randomUUID(); + const log = createWriteStream(path.join(input.temporaryRoot, "legacy-maintenance.log"), { mode: 0o600 }); + const errors = new WeakMap(); + const stopper = createRunnerE2EServerStopper({ gracefulTimeoutMs: 15_000, forcedTimeoutMs: 5_000, + hasSpawnError: child => errors.has(child), markExpectedStop() {}, log: message => log.write(message) }); + const child = spawn(process.execPath, runnerE2ETypeScriptProcessArgs(input.repositoryRoot, + path.join(previous.home, "app/server/dist/maintenance/ephemeral-migration.js")), { + cwd: input.repositoryRoot, detached: process.platform !== "win32", stdio: ["ignore", "pipe", "pipe"], + env: { ...environment, HOME: previous.home, PAPERCLIP_HOME: previous.home, PAPERCLIP_CONFIG: previous.configPath, + PAPERCLIP_INSTANCE_ID: "default", DATABASE_URL: databaseUrl, PORT: String(port), + PAPERCLIP_EPHEMERAL_MIGRATION: "true", PAPERCLIP_EPHEMERAL_MIGRATION_ID: id, PAPERCLIP_EPHEMERAL_MIGRATION_TOKEN: token } }); + child.on("error", error => errors.set(child, error)); + child.stdout!.pipe(log, { end: false }); child.stderr!.pipe(log, { end: false }); + await stopper.watch(child); + maintenance = { async stop() { await stopper.stopAll(); await new Promise(resolve => log.end(resolve)); } }; + const endpoint = `http://127.0.0.1:${port}`; + const headers = { authorization: `Bearer ${token}` }; + const deadline = Date.now() + 180_000; + let receipt: Record | undefined; + while (!input.cancelled() && Date.now() < deadline) { + if (errors.has(child) || child.exitCode !== null || child.signalCode !== null) throw new Error("Maintenance exited before a verified export"); + receipt = await fetch(`${endpoint}/status`, { headers, signal: AbortSignal.timeout(2_000) }) + .then(async response => response.ok ? await response.json() as Record : undefined).catch(() => undefined); + if (receipt?.state === "blocked") throw new Error(`Migration blocked: ${receipt.failure}`); + if (receipt?.state === "ready") break; + await delay(250); + } + if (receipt?.state !== "ready" || receipt.id !== id || receipt.eligibility?.eligible !== true) throw new Error("Maintenance did not qualify migration"); + const exported = await fetch(`${endpoint}/export`, { method: "POST", headers, signal: AbortSignal.timeout(10_000) }); + if (!exported.ok) throw new Error(`Migration export returned ${exported.status}`); + const data = await exported.json() as Record; + if (data.id !== id || typeof data.decisionSigningSecret !== "string" || data.decisionSigningSecret.length < 32) throw new Error("Migration key export is incomplete"); + environment.PAPERCLIP_DECISION_SIGNING_SECRET = data.decisionSigningSecret; + await json(path.join(controls.controlDirectory, "legacy-migration-receipt.json"), receipt); + await maintenance.stop(); maintenance = undefined; + current = await start("ephemeral-controller", environment); await current.waitForReady(); proxy.routeTo(current.origin); + await makeDirectoryTreeRemovable(previous.home); await rm(previous.home, { recursive: true }); + migrated = true; + await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "ready", controllerCacheDiscarded: true, + completedAt: new Date().toISOString(), legacyMigration: { ...receipt, oldHomeDeleted: true } }); + } catch (error) { + await json(controls.restartAcknowledgementPath, { requestId: request.requestId, status: "failed", message: String(error) }); throw error; + } + } + } finally { + try { await maintenance?.stop(); } finally { + try { await Promise.all(servers.map(server => server.stop())); } finally { await proxy.close(); } + } + } +} diff --git a/tests/runner-e2e/legacy-workspace-migration.test.ts b/tests/runner-e2e/legacy-workspace-migration.test.ts new file mode 100644 index 00000000000..da0ea95f1f2 --- /dev/null +++ b/tests/runner-e2e/legacy-workspace-migration.test.ts @@ -0,0 +1,48 @@ +import { createHash } from "node:crypto"; +import type { ContinuationCheckpoint } from "./continuation-scoring.js"; +import { describe, expect, it } from "vitest"; +import { runnerMatrix } from "./catalog.js"; +import { gradeLegacyWorkspaceMigration, migrationPrimaryOutputCheckpoints, isLegacyWorkspaceMigration, type MigrationPoint } from "./legacy-workspace-migration.js"; +import { remoteWorkspaceStorageServerEnvironment } from "./remote-workspace-storage.js"; + +const reference = "a".repeat(32) + "\n", referenceSha256 = createHash("sha256").update(reference).digest("hex"); +const checkpoints = [{ phase: "final", attachments: [{ originalFilename: "migration-reference.txt", body: reference, + contentSha256: referenceSha256, contentVerified: true }] }] as ContinuationCheckpoint[]; +const before: MigrationPoint = { workspaceId: "workspace", providerType: "isolated_worktree", migrated: false, + proofSha256: referenceSha256, runId: "first", sessionId: "session", providerSessionSha256: "provider-session", + logicalCwd: "/old-controller/workspace", remoteCwd: "/home/daytona/paperclip-workspace" }; +const after: MigrationPoint = { ...before, runId: "second", migrated: true, providerType: "cloud_sandbox" }; + +describe("legacy workspace migration qualification", () => { + it("requires the preserved session and file, not only a successful new turn", () => { + expect(gradeLegacyWorkspaceMigration(before, after, checkpoints).every(check => check.passed)).toBe(true); + for (const patch of [{ proofSha256: "recreated" }, { sessionId: "fresh" }, { providerSessionSha256: "fresh" }, + { logicalCwd: "/new-home/workspace" }, { remoteCwd: "/new-sandbox/workspace" }, { workspaceId: "other" }, + { runId: before.runId }, { migrated: false }]) { + expect(gradeLegacyWorkspaceMigration(before, { ...after, ...patch }, checkpoints).every(check => check.passed)).toBe(false); + } + expect(gradeLegacyWorkspaceMigration(before).every(check => check.passed)).toBe(false); + }); + + it("excludes only the one verified preserved reference from the primary output count", () => { + expect(migrationPrimaryOutputCheckpoints(checkpoints, before, after)[0].attachments).toEqual([]); + const original = checkpoints[0].attachments[0] as Record; + for (const patch of [{ body: "changed" }, { contentVerified: false }, { originalFilename: "unrequested.txt" }]) { + const changed = [{ ...checkpoints[0], attachments: [{ ...original, ...patch }] }]; + expect(migrationPrimaryOutputCheckpoints(changed, before, after)).toEqual(changed); + expect(gradeLegacyWorkspaceMigration(before, after, changed).every(check => check.passed)).toBe(false); + } + const duplicated = [{ ...checkpoints[0], attachments: [original, { ...original }] }]; + expect(migrationPrimaryOutputCheckpoints(duplicated, before, after)).toEqual(duplicated); + expect(gradeLegacyWorkspaceMigration(before, after, duplicated).every(check => check.passed)).toBe(false); + }); + + it("registers explicit warm and per-turn production profiles with isolated object storage", () => { + const cells = runnerMatrix.filter(isLegacyWorkspaceMigration); + expect(cells.length).toBeGreaterThan(0); + expect(new Set(cells.map(cell => cell.suite.id)).size).toBe(2); + expect(cells.every(cell => cell.suite.manualOnly && cell.task.flow === "continuation" && cell.task.expectedRunCount === 2)).toBe(true); + expect(remoteWorkspaceStorageServerEnvironment({ executions: [cells[0]!], instanceId: "migration-test", + env: { PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET: "owned-migration-bucket" } })).toMatchObject({ PAPERCLIP_STORAGE_PROVIDER: "s3" }); + }); +}); diff --git a/tests/runner-e2e/legacy-workspace-migration.ts b/tests/runner-e2e/legacy-workspace-migration.ts new file mode 100644 index 00000000000..164b0bc6230 --- /dev/null +++ b/tests/runner-e2e/legacy-workspace-migration.ts @@ -0,0 +1,71 @@ +import type { ContinuationCheckpoint } from "./continuation-scoring.js"; +import { createHash } from "node:crypto"; +import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js"; +import type { RunnerApi } from "./api.js"; +import type { StoryCheck } from "./everyday-observations.js"; +import type { MatrixExecution } from "./types.js"; + +export function isLegacyWorkspaceMigration(execution: MatrixExecution) { + return ["legacy-workspace-migration", "legacy-workspace-migration-warm"].includes(execution.suite.id); +} + +export function legacyWorkspaceMigrationPrompt(prompt: string) { + const continuation = prompt.replace("Do not create other tasks or deliverables.", "Do not create other tasks."); + return `Work in this task's own workspace. Before asking the question, create migration-reference.txt containing a fresh random 32-character lowercase hexadecimal reference and a trailing newline. Do not deliver any files or documents before I answer. After I answer, read this same file, keep its original bytes, and attach the original migration-reference.txt as an additional deliverable alongside the confirmation document. Include a clickable Markdown link to the saved confirmation document in your final reply; its title or key alone is not a link. ${continuation}`; +} + +export interface MigrationPoint { + workspaceId: string; providerType: string; migrated: boolean; proofSha256: string; + runId: string; sessionId: string; providerSessionSha256: string; logicalCwd: string; remoteCwd: string; +} + +export async function readMigrationPoint(input: { api: Pick; companyId: string; agentId: string; issueId: string }): Promise { + const { workspace } = await input.api.get<{ workspace: ExecutionWorkspace | null }>(`/api/issues/${input.issueId}/workspace`); + if (!workspace || workspace.companyId !== input.companyId) throw new Error("Migration oracle has no bound workspace"); + const runs = await input.api.get>(`/api/issues/${input.issueId}/runs`); + const latest = runs.filter(run => run.agentId === input.agentId && run.status === "succeeded") + .sort((a, b) => b.finishedAt.localeCompare(a.finishedAt))[0]; + if (!latest || runs.some(run => ["queued", "running"].includes(run.status))) throw new Error("Migration requires a settled retained turn"); + const run = await input.api.get>(`/api/heartbeat-runs/${latest.runId}`); + const execution = run.runnerProfileJson?.nativeExecutionInput; + const reference = run.runnerProfileJson?.nativeWorkspaceSync ?? run.runnerProfileJson?.remoteWorkspace; + if (run.companyId !== input.companyId || run.agentId !== input.agentId || run.nativeIssueId !== input.issueId + || run.runtimeMode !== "native" || !run.nativeSessionId || typeof run.sessionIdAfter !== "string" || !run.sessionIdAfter + || execution?.binding?.executionWorkspaceId !== workspace.id + || !execution?.workspace?.cwd || !reference?.remoteCwd) throw new Error("Migration run identity is incomplete"); + const file = await input.api.get(`/api/issues/${input.issueId}/file-resources/content?workspace=execution&path=migration-reference.txt`); + if (file.resource?.workspaceId !== workspace.id || file.content?.encoding !== "utf8" + || !/^[a-f0-9]{32}\n$/.test(file.content.data)) throw new Error("Migration reference bytes are missing"); + return { workspaceId: workspace.id, providerType: workspace.providerType, + migrated: Boolean(workspace.metadata?.nativeWorkspaceMigration), + proofSha256: createHash("sha256").update(file.content.data).digest("hex"), + runId: run.id, sessionId: run.nativeSessionId, providerSessionSha256: createHash("sha256").update(run.sessionIdAfter).digest("hex"), + logicalCwd: execution.workspace.cwd, remoteCwd: reference.remoteCwd }; +} + +function migrationReferenceAttachments(checkpoints: ContinuationCheckpoint[], before?: MigrationPoint, after?: MigrationPoint) { + if (!before || !after || before.proofSha256 !== after.proofSha256) return []; + return (checkpoints.find(point => point.phase === "final")?.attachments ?? []).filter(value => { + const file = value as Record; + return file.originalFilename === "migration-reference.txt" && file.contentVerified === true + && typeof file.body === "string" && file.contentSha256 === before.proofSha256 + && createHash("sha256").update(file.body).digest("hex") === before.proofSha256; + }); +} + +export function migrationPrimaryOutputCheckpoints(checkpoints: ContinuationCheckpoint[], before?: MigrationPoint, after?: MigrationPoint) { + const references = migrationReferenceAttachments(checkpoints, before, after); + if (references.length !== 1) return checkpoints; + return checkpoints.map(point => point.phase === "final" + ? { ...point, attachments: point.attachments.filter(file => file !== references[0]) } : point); +} + +export function gradeLegacyWorkspaceMigration(before?: MigrationPoint, after?: MigrationPoint, checkpoints: ContinuationCheckpoint[] = []): StoryCheck[] { + return [{ id: "legacy-workspace-migration-continuation", passed: Boolean(before && after && !before.migrated && after.migrated + && before.providerType !== "cloud_sandbox" && after.providerType === "cloud_sandbox" && before.workspaceId === after.workspaceId + && before.runId !== after.runId && before.sessionId === after.sessionId && before.logicalCwd === after.logicalCwd + && before.remoteCwd === after.remoteCwd && before.proofSha256 === after.proofSha256 && before.providerSessionSha256 === after.providerSessionSha256), + detail: "A new turn retains the exact native session, logical and provider paths, workspace, and reference file bytes after offline migration and destruction of the old home." }, + { id: "legacy-workspace-migration-reference-delivered", passed: migrationReferenceAttachments(checkpoints, before, after).length === 1, + detail: "The one requested reference attachment has the exact original workspace bytes, independently downloaded and verified." }]; +} diff --git a/tests/runner-e2e/remote-workspace-storage.ts b/tests/runner-e2e/remote-workspace-storage.ts index 6275706a4e0..3f17cb67ee8 100644 --- a/tests/runner-e2e/remote-workspace-storage.ts +++ b/tests/runner-e2e/remote-workspace-storage.ts @@ -1,4 +1,5 @@ import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; +import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js"; import { createHash } from "node:crypto"; import { remoteWorkspaceAuthoritySchema, remoteWorkspaceRunReferenceSchema } from "../../packages/shared/src/remote-workspace.js"; import type { ExecutionWorkspace, WorkspaceFileContent } from "../../packages/shared/src/index.js"; @@ -134,8 +135,9 @@ export function remoteWorkspaceStorageServerEnvironment(input: { env: NodeJS.ProcessEnv; instanceId: string; }): NodeJS.ProcessEnv | undefined { - if (!input.executions.some(isRemoteWorkspaceStorageExecution)) return undefined; - if (!input.executions.length || !input.executions.every(isRemoteWorkspaceStorageExecution)) { + const requiresStorage = (execution: MatrixExecution) => isRemoteWorkspaceStorageExecution(execution) || isLegacyWorkspaceMigration(execution); + if (!input.executions.some(requiresStorage)) return undefined; + if (!input.executions.length || !input.executions.every(requiresStorage)) { throw new Error("Remote workspace storage qualification must run alone"); } const bucket = input.env.PAPERCLIP_RUNNER_E2E_WORKSPACE_STORAGE_BUCKET?.trim(); diff --git a/tests/runner-e2e/server.ts b/tests/runner-e2e/server.ts index 7698c6a0d8e..213ffffcc35 100644 --- a/tests/runner-e2e/server.ts +++ b/tests/runner-e2e/server.ts @@ -1,5 +1,7 @@ import { isRollingWebDeployment } from "./controller-rolling-deployment.js"; import { superviseRollingDeployment } from "./rolling-deployment-supervisor.js"; +import { isLegacyWorkspaceMigration } from "./legacy-workspace-migration.js"; +import { superviseLegacyWorkspaceMigration } from "./legacy-workspace-migration-supervisor.js"; import { readRestartRequest, type RestartRequest } from "./server-restart-request.js"; import { startCursorCloudCallbackProxy } from "./cursor-cloud-callback-proxy.js"; import { prepareControllerDatabase } from "./controller-database.js"; @@ -449,11 +451,18 @@ async function supervise() { }); // Postgres needs the socket itself; release immediately before child spawn. await databaseReservation?.close(); - if (upgradeArtifacts || executionIds.some(id => id.endsWith(".recover-controller"))) { + const legacyMigration = executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isLegacyWorkspaceMigration(execution); }); + if (upgradeArtifacts || legacyMigration || executionIds.some(id => id.endsWith(".recover-controller"))) { // The wrapper owns this sibling process. Killing the serving child must // leave Postgres available to the stopped-controller observer and successor. controllerDatabase = await prepareControllerDatabase(configPath, temporaryRoot); } + if (legacyMigration) { + if (upgradeArtifacts || installedRelease) throw new Error("Migration source qualification cannot claim installed-artifact evidence"); + await superviseLegacyWorkspaceMigration({ temporaryRoot, repositoryRoot, configPath, port: Number(port), + environment: definedServerEnvironment, executionIds, cancelled: shutdownRequested }); + return; + } if (executionIds.some(id => { const execution = runnerMatrix.find(row => row.id === id); return execution && isRollingWebDeployment(execution); })) { if (upgradeArtifacts || installedRelease) throw new Error("Rolling source qualification cannot claim installed-artifact evidence"); await superviseRollingDeployment({ temporaryRoot, repositoryRoot, configPath, port: Number(port),