mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
Preserve native sessions through legacy workspace enrollment (#18943)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native session retains its workspace and provider session across turns. > - An ephemeral web server must restore those files without its predecessor's disk. > - Legacy enrollment changes the workspace authority and can change both workspace addresses. > - The retained provider backup checks those addresses as part of its identity. > - This pull request preserves the exact addresses in a verified migration receipt. > - The benefit is that the next turn can use the original session after enrollment. ## Linked Issues or Issue Description Refs #18932, merged as `612d9931a6`. This pull request builds on that all-Runner durability change. Related: #18918 fixes concurrent workspace preservation locks. This change covers identity preservation during offline enrollment. ## What Changed - The native workspace migration owner records the selected session bindings in the successful workspace migration operation. - Runtime selection retains the original logical address only for the exact agent, task and session. - Remote workspace preparation restores the original provider address. It refuses conflicting targets, forged receipts and occupied unowned roots. - The environment owner includes the migration operation in the reusable lease fingerprint. Warm policy stays warm and receives a fresh allocation through normal lease cleanup. - Exact migration evidence permits the existing verified provider backup to restore into that fresh warm allocation. - Preserve verified provider allocation files through a normalized, lease-bound digest receipt. Unknown files and altered markers remain refused; interrupted publication is retryable. - Retain legacy managed-instruction session fingerprints using the exact imported content hash and recorded paths, preserving all other configuration and credential checks. - Migrate existing local skill archive/manifest objects through upload claims, readback verification and receipt compare-and-swap; retain original objects and IDs. - Add explicit two-turn browser qualification for every released native Daytona profile, with warm and per-turn variants. The production maintenance entry runs offline. The old home is deleted only after a verified export and successor readiness. The oracle checks provider identity, native session, addresses, reference bytes, question and final document. ## Verification - Full repository typecheck passed at `b12ecd823e`. The production session fix passed 198 session/credential tests. Full repository build passed at `94bb917947`. The final Rust-only provider fix passed 57 ACPX provider tests and the exact Linux release build. - Workspace and transport checks passed: 117 transport/workspace tests, six final real PostgreSQL migration boundary cases, 178 environment tests, 539 Daytona tests (eight skips), and the native warm-backup admission control. - Local skill migration passed eight PostgreSQL/S3 tests (one existing opt-in skip). Failed readback retains the source. Successful readback still works after deletion of the old home. - Latest Product E2E typecheck and 38 focused harness tests pass. Earlier catalog/admission/report checks passed. Twelve unrelated support-test failures reproduce on unchanged base `1a39dea9aa`; these are not reported as passing. - Live Codex per-turn at `2d5bb9a6d1` retained the native and provider sessions, both workspace paths and exact reference bytes after deletion of the old home. It completed the task and delivered its saved document and original reference. Its original failure was an output-count grading defect. The corrected grader passes all 21 checks against retained, hashed evidence; the original result remains unchanged. - Live OpenCode per-turn and Codex warm at harness `94bb917947` pass all 21 product checks. Their launcher cleanup failed because a read-only diagnostic observer kept its disposable database open. The observer now closes before ordinary database cleanup; a no-provider reproduction confirms the fix. All owned sandboxes were independently verified absent, child keys revoked and buckets removed. Original failed campaign dispositions remain unchanged. - Claude at `94bb917947` kept its sessions and reference bytes but omitted the final clickable document link. That user-delivery failure remains recorded. The fixture now explicitly asks for the link and records migration evidence before checking navigation. The fresh Claude attempt at `b12ecd823e` passed both strict migration checks: exact provider/native session, both paths, original file bytes and independently downloaded attachment. Its full browser workflow failed again: it requested unnecessary confirmation, used the answer sentence as the reference, and omitted the clickable document link. This is retained as a failed user-workflow result, not a full pass. - Earlier live failures exposed and reproduced the skill-object, fresh-allocation marker and managed-instruction fingerprint bugs. The fixes retain original identity checks. No provider session reset or accounting bypass was used. - The real Grok continuation at `b12ecd823e` failed before provider execution because its immutable-instruction attach policy rejected relocated registered file copies. A no-provider regression reproduced the exact rejection. `b05ba3728a` selects the existing strict authenticated asset-path policy for Grok; content digests, unknown context fields, session/workspace identity and same-run mutations remain enforced. The regression and all 57 ACPX provider controls pass. The Grok browser migration now passes all 21 checks with the b05 Linux Runner, including provider identity, exact reference bytes and final user output; the launcher also reports successful cleanup. - `24fabf1d81` applies the same policy to released ACPX Codex after a second no-provider reproduction. All 58 ACPX provider controls and the final Linux release build pass. Its live browser migration passes all 21 checks on the final binary, including exact original session/file preservation and completed user output. - The full native-session executor suite passes all 1,121 tests after updating its existing lease database fixture for the migration-receipt join. The prior CI failure was this unsupported test-double method; the production guard still rejects ordinary unproven recovery. CI and Greptile 5/5 passed at `ca1e0d5fe6`. The first CI attempt lost several workers and had one rotated-recovery transport failure; its seven descriptor cases passed locally, and the single unchanged retry passed. The original failure remains recorded. Staging enrollment has not started. Initial offline enrollment and subsequent rolling availability are separate checks. - The final review follow-up adds real runtime-selection PostgreSQL cases for exact, different-session and absent receipts, plus executor-level warm restoration that copies actual backup files and verifies provider identity and the new lease stamp. Both complete affected suites pass (1,172 tests), and full repository typecheck passes. Removing either production migration connection independently makes its new positive case fail; both source files were restored byte-for-byte after these negative controls. - Final commit `0971d373a3` passes all 63 checks (57 successes, five expected skips, one neutral); Greptile is 5/5 and every review thread is resolved. ## Risks - An identity conflict blocks enrollment and keeps the original files. It does not silently reset the session. - A new migration generation retires a reusable lease through its existing owner. The original sandbox must finish normal cleanup before reuse proceeds. - Initial enrollment has an offline step. The separate rolling suites prove uninterrupted deploy behavior after enrollment. - This adds no schema migration and changes no provider credentials or accounting gate. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model build and context window are not exposed in this session. The agent used reasoning, terminal tools, code edits, PostgreSQL/filesystem tests and isolated live browser qualification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
cf51936587
commit
84aa989bf9
40 files changed
+1177
-66
No files matched your search
@@ -17,6 +17,7 @@ import {
|
||||
} from "@paperclipai/adapter-utils/ssh";
|
||||
import {
|
||||
activityLog,
|
||||
workspaceOperations,
|
||||
costEvents,
|
||||
agents,
|
||||
builtInManagedResources,
|
||||
@@ -238,6 +239,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
await stopSshEnvLabFixture(path.join(root, "state.json")).catch(() => undefined);
|
||||
await rm(root, { recursive: true, force: true }).catch(() => undefined);
|
||||
}
|
||||
await db.delete(workspaceOperations);
|
||||
await db.delete(activityLog);
|
||||
await db.delete(environmentLeases);
|
||||
await db.delete(issues);
|
||||
@@ -4441,7 +4443,7 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
});
|
||||
|
||||
it("does not resume released reusable plugin sandbox leases after provider config drift", async () => {
|
||||
it.each(["provider config", "workspace migration"])("does not resume released reusable plugin sandbox leases after %s drift", async drift => {
|
||||
const pluginId = randomUUID();
|
||||
const { companyId, agentId, environment: baseEnvironment, runId } = await seedEnvironment();
|
||||
const providerConfig = {
|
||||
@@ -4488,12 +4490,13 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
await insertReusableProject(projectId, companyId);
|
||||
await insertReusableExecutionWorkspace(executionWorkspaceId, companyId, projectId);
|
||||
|
||||
let acquisitions = 0;
|
||||
const workerManager = {
|
||||
isRunning: vi.fn((id: string) => id === pluginId),
|
||||
call: vi.fn(async (_pluginId: string, method: string, params: any) => {
|
||||
if (method === "environmentAcquireLease") {
|
||||
return {
|
||||
providerLeaseId: `lease-${params.config.image}`,
|
||||
providerLeaseId: `lease-${params.config.image}${++acquisitions > 1 ? "-replacement" : ""}`,
|
||||
metadata: {
|
||||
provider: "fake-plugin",
|
||||
image: params.config.image,
|
||||
@@ -4532,13 +4535,27 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
});
|
||||
await runtimeWithPlugin.releaseRunLeases(runId);
|
||||
|
||||
if (drift === "workspace migration") {
|
||||
const operationId = randomUUID();
|
||||
await db.insert(workspaceOperations).values({ id: operationId, companyId, executionWorkspaceId,
|
||||
phase: "remote_workspace_migration", status: "succeeded", metadata: {
|
||||
schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: {
|
||||
schema: "paperclip.native-workspace-migration.v1", remoteCwd: "/workspace", sessions: [{
|
||||
sourceRunId: runId, agentId, issueId: randomUUID(), sessionId: randomUUID(), logicalCwd: "/old/controller/workspace",
|
||||
}],
|
||||
},
|
||||
} });
|
||||
await db.update(executionWorkspaces).set({ metadata: { nativeWorkspaceMigration: {
|
||||
schema: "paperclip.native-workspace-migration.v1", operationId,
|
||||
} } }).where(eq(executionWorkspaces.id, executionWorkspaceId));
|
||||
}
|
||||
const nextRunId = randomUUID();
|
||||
await insertRunningManualRun(nextRunId, companyId, agentId);
|
||||
const updatedEnvironment: Environment = {
|
||||
...environment,
|
||||
config: {
|
||||
...providerConfig,
|
||||
image: "template-b",
|
||||
image: drift === "provider config" ? "template-b" : "template-a",
|
||||
},
|
||||
};
|
||||
await environmentService(db).update(environment.id, {
|
||||
@@ -4557,7 +4574,8 @@ describeEmbeddedPostgres("environmentRuntimeService", () => {
|
||||
},
|
||||
});
|
||||
|
||||
expect(second.lease.providerLeaseId).toBe("lease-template-b");
|
||||
expect(second.lease.providerLeaseId).toBe(`lease-${updatedEnvironment.config.image}-replacement`);
|
||||
expect(second.lease.metadata?.reusableSandboxLease).not.toEqual(first.lease.metadata?.reusableSandboxLease);
|
||||
expect(workerManager.call).not.toHaveBeenCalledWith(
|
||||
pluginId,
|
||||
"environmentResumeLease",
|
||||
|
||||
@@ -23,6 +23,11 @@ import { verifyAgentFileObjects } from "../services/agent-file-maintenance.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
import { resolveDecisionSigningSecret } from "../services/decision-signing.js";
|
||||
import { getRunLogStore } from "../services/run-log-store.js";
|
||||
import { companySkillService } from "../services/company-skills.js";
|
||||
import { skillPackageStore } from "../services/skill-package-store.js";
|
||||
import { skillInstallationMaintenance } from "../services/skill-installation-maintenance.js";
|
||||
import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js";
|
||||
import { companySkills, companySkillRepositorySnapshots } from "@paperclipai/db";
|
||||
|
||||
// Explicit real-dependency qualification. Never contacts off-host storage in
|
||||
// the default test path; use a disposable S3-compatible service and PostgreSQL.
|
||||
@@ -55,6 +60,8 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, companyId));
|
||||
await db.delete(agents).where(eq(agents.companyId, companyId));
|
||||
await db.delete(companySkills).where(eq(companySkills.companyId, companyId));
|
||||
await db.delete(companySkillRepositorySnapshots).where(eq(companySkillRepositorySnapshots.companyId, companyId));
|
||||
await db.delete(companies).where(eq(companies.id, companyId));
|
||||
let token: string | undefined;
|
||||
do {
|
||||
@@ -100,6 +107,46 @@ describe.skipIf(!process.env.PAPERCLIP_TEST_S3_ENDPOINT)("existing-volume enroll
|
||||
await s3.send(new PutObjectCommand({ Bucket: bucket, Key: file.objectKey, Body: "corrupt" }));
|
||||
await expect(verifyAgentFileObjects(db, getStorageService(), AbortSignal.timeout(30_000))).rejects.toThrow(/agent_file_object_(size|digest)_mismatch/);
|
||||
}, 120_000);
|
||||
it("preserves original skill objects and identity when destination readback fails", async () => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
|
||||
const skills = await companySkillService(db).list(companyId);
|
||||
const local = getStorageService(), store = skillPackageStore(db, local);
|
||||
const before = await Promise.all(skills.map(async skill => {
|
||||
const version = await store.version(companyId, skill.id, skill.currentVersionId!);
|
||||
return { skillId: skill.id, version: version!, files: await store.versionFiles(version!) };
|
||||
}));
|
||||
expect(before.length).toBeGreaterThan(0);
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
|
||||
const remote = getStorageService();
|
||||
await expect(migrateLocalSkillPackageObjects(db, { source: local, destination: { ...remote,
|
||||
getObject: async () => { throw new Error("Synthetic destination readback failed"); } }, writersStopped: true,
|
||||
signal: AbortSignal.timeout(30_000) })).rejects.toThrow("Synthetic destination readback failed");
|
||||
for (const point of before) {
|
||||
expect((await store.descriptor(companyId, point.version.packageSnapshotId!)).storage?.provider).toBe("local_disk");
|
||||
expect(await store.versionFiles(point.version)).toEqual(point.files);
|
||||
}
|
||||
}, 120_000);
|
||||
it("verifies bundled skill migration after removal of the old server home", async () => {
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "local_disk");
|
||||
const skills = await companySkillService(db).list(companyId), local = skillPackageStore(db);
|
||||
const before = await Promise.all(skills.map(async skill => {
|
||||
const version = await local.version(companyId, skill.id, skill.currentVersionId!);
|
||||
return { skillId: skill.id, version: version!, files: await local.versionFiles(version!) };
|
||||
}));
|
||||
vi.stubEnv("PAPERCLIP_STORAGE_PROVIDER", "s3");
|
||||
const result = await migrate();
|
||||
expect(result.eligibility.eligible).toBe(true);
|
||||
expect(result.counts.skillObjects).toBe(before.length);
|
||||
await fs.rm(home, { recursive: true, force: true });
|
||||
const remote = skillPackageStore(db);
|
||||
for (const point of before) {
|
||||
const version = await remote.version(companyId, point.skillId, point.version.id);
|
||||
expect(version?.packageSnapshotId).toBe(point.version.packageSnapshotId);
|
||||
expect(await remote.versionFiles(version!)).toEqual(point.files);
|
||||
}
|
||||
expect((await skillInstallationMaintenance(db).verifyVersions({ companyId, limit: 50 })).results
|
||||
.every(row => ["remote_storage", "database_release_snapshot"].includes(row.status))).toBe(true);
|
||||
}, 120_000);
|
||||
it("migrates overlapping task folders through maintenance and restores both from S3 after volume loss", async () => {
|
||||
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "73".repeat(32));
|
||||
const parent = path.join(home, "project"), child = path.join(parent, ".paperclip/worktrees/task");
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { createHash } from "node:crypto";
|
||||
import { execFile as execFileCallback } from "node:child_process";
|
||||
import fs from "node:fs/promises";
|
||||
import os from "node:os";
|
||||
@@ -2732,6 +2733,28 @@ describe("effective run session config freshness", () => {
|
||||
expect(freshness(await metadata("/replacement-controller", "sha256:entry-v1", "skill-version-2")).changedCategories).toContain("runtimeSkills");
|
||||
});
|
||||
|
||||
it("retains a legacy managed-instruction session after verified import removes the old home", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "legacy-instruction-session-"));
|
||||
const body = "Preserve this exact agent identity and instructions.\n";
|
||||
await fs.writeFile(path.join(root, "AGENTS.md"), body);
|
||||
const config = { model: "gpt-5.4-mini", instructionsBundleMode: "managed", instructionsRootPath: root,
|
||||
instructionsFilePath: path.join(root, "AGENTS.md"), instructionsEntryFile: "AGENTS.md" };
|
||||
const before = await buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: config });
|
||||
await fs.rm(root, { recursive: true });
|
||||
const managedInstructions = { entryFile: "AGENTS.md", contentHash: "sha256:" + createHash("sha256").update(body).digest("hex") };
|
||||
const metadata = (contentHash = managedInstructions.contentHash, model = config.model, retainManagedInstructionPaths = false) =>
|
||||
buildSessionConfigMetadata({ adapterType: "paperclip_runner", effectiveAdapterConfig: { ...config, model },
|
||||
managedInstructions: { ...managedInstructions, contentHash }, retainManagedInstructionPaths });
|
||||
const after = await metadata();
|
||||
const decide = (current: SessionConfigMetadata, compatible: SessionConfigMetadata[] = []) => resolveTaskSessionConfigFreshness({
|
||||
hasTaskSession: true, configuredModel: config.model, taskSessionParams: sessionParamsWithConfigMetadata(before),
|
||||
configMetadata: current, compatibleConfigMetadata: compatible });
|
||||
expect(decide(after)).toMatchObject({ reset: true, changedCategories: ["adapterConfig"] });
|
||||
expect(decide(after, [await metadata(undefined, undefined, true)]).reset).toBe(false);
|
||||
expect(decide(await metadata("sha256:changed"), [await metadata("sha256:changed", undefined, true)]).reset).toBe(true);
|
||||
expect(decide(await metadata(undefined, "different-model"), [await metadata(undefined, "different-model", true)]).reset).toBe(true);
|
||||
});
|
||||
|
||||
it("does not read unbounded legacy instructions paths for config fingerprints", async () => {
|
||||
const root = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-session-fingerprint-"));
|
||||
const instructionsPath = path.join(root, "AGENTS.md");
|
||||
|
||||
@@ -1,3 +1,7 @@
|
||||
import { loadConfig } from "../config.js";
|
||||
import { createStorageService } from "../storage/service.js";
|
||||
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
|
||||
import { migrateLocalSkillPackageObjects } from "../services/skill-package-storage-migration.js";
|
||||
import { heartbeatRuns, type Db } from "@paperclipai/db";
|
||||
import { offlineMigrationExecutionWhere } from "../services/offline-migration-execution.js";
|
||||
import { getStorageService } from "../storage/index.js";
|
||||
@@ -73,6 +77,15 @@ export async function migrateEphemeralServerData(db: Db, input: {
|
||||
|
||||
const packages = skillPackageMaintenance(db);
|
||||
const skills = skillInstallationMaintenance(db);
|
||||
phase("skill_objects");
|
||||
const localSkillStorage = createStorageService(createLocalDiskStorageProvider(loadConfig().storageLocalDiskBaseDir));
|
||||
after = undefined;
|
||||
for (;;) {
|
||||
const page = await migrateLocalSkillPackageObjects(db, { source: localSkillStorage, destination: storage, writersStopped: true, signal, after });
|
||||
counts.skillObjects = (counts.skillObjects ?? 0) + page.migrated;
|
||||
if (page.inspected < 50 || !page.after) break;
|
||||
after = page.after;
|
||||
}
|
||||
for (const [name, migrate] of [["skill_packages", packages.migrateSnapshots], ["skill_versions", packages.migrateVersions]] as const) {
|
||||
phase(name);
|
||||
after = undefined;
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
|
||||
import { HttpError } from "../errors.js";
|
||||
import { captureEnvironmentAcquisitionDiagnostic } from "./environment-acquisition-diagnostics.js";
|
||||
import { createOwnedSandbox, recoverSandboxCreations, recoverableTaskCreation } from "./sandbox-creation-operations.js";
|
||||
@@ -964,6 +965,8 @@ async function buildReusableSandboxLeaseFingerprint(input: {
|
||||
companyId: input.companyId,
|
||||
environment: input.environment,
|
||||
});
|
||||
const workspaceMigration = input.executionWorkspaceId
|
||||
? await readNativeWorkspaceMigration(input.db, { companyId: input.companyId, id: input.executionWorkspaceId }) : null;
|
||||
return createEffectiveRunConfigFingerprints({
|
||||
lease: {
|
||||
companyId: input.companyId,
|
||||
@@ -972,6 +975,7 @@ async function buildReusableSandboxLeaseFingerprint(input: {
|
||||
driver: input.environment.driver,
|
||||
},
|
||||
executionWorkspaceId: input.executionWorkspaceId,
|
||||
...(workspaceMigration ? { workspaceMigrationGeneration: workspaceMigration.operationId } : {}),
|
||||
...(input.executionWorkspaceId ? {} : { projectlessIssueId: input.issueId }),
|
||||
agentId: input.agentId,
|
||||
adapterType: input.adapterType,
|
||||
|
||||
@@ -2,7 +2,7 @@ import { randomUUID } from "node:crypto";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
|
||||
import { activityLog, agents, companies, completionContracts, createDb, applyPendingMigrations, EMBEDDED_POSTGRES_TEST_TIMEOUT_MS, executionWorkspaces, heartbeatRuns, issues, nativeRunFinalizations, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
@@ -82,6 +82,54 @@ describe.skipIf(!support.supported)("heartbeat runtime selection boundary", () =
|
||||
const coordinators = (id: string) => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, id));
|
||||
const sandbox = { kind: "remote" as const, transport: "sandbox" as const, remoteCwd: "/workspace", reusableLeaseConfigured: false };
|
||||
|
||||
it.each(["matching", "different-session", "absent"])("persists the migrated logical cwd only for the receipt's exact session: %s", async receipt => {
|
||||
const sameSession = receipt === "matching";
|
||||
const { input, run, agent, issue } = await fixture();
|
||||
const sessionId = randomUUID(), operationId = randomUUID();
|
||||
const logicalCwd = "/paperclip/legacy/task-workspace";
|
||||
const marker = { schema: "paperclip.native-workspace-migration.v1", operationId };
|
||||
const [workspace] = await db.insert(executionWorkspaces).values({
|
||||
companyId: agent.companyId, sourceIssueId: issue.id, name: "Migrated workspace",
|
||||
mode: "shared_workspace", strategyType: "project_primary", cwd: "/tmp/ephemeral/task-workspace",
|
||||
providerType: "cloud_sandbox", metadata: receipt === "absent" ? {} : { nativeWorkspaceMigration: marker },
|
||||
}).returning();
|
||||
await db.insert(workspaceOperations).values({
|
||||
id: operationId, companyId: agent.companyId, executionWorkspaceId: workspace!.id,
|
||||
phase: "remote_workspace_migration", status: "succeeded", metadata: {
|
||||
schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(),
|
||||
nativeSessionContinuity: { schema: marker.schema, remoteCwd: sandbox.remoteCwd, sessions: [{
|
||||
sourceRunId: randomUUID(), agentId: agent.id, issueId: issue.id,
|
||||
sessionId: sameSession ? sessionId : randomUUID(), logicalCwd,
|
||||
}] },
|
||||
},
|
||||
});
|
||||
input.workspace.persistedExecutionWorkspace = workspace!;
|
||||
input.workspace.executionWorkspace.cwd = logicalCwd;
|
||||
input.workspace.executionTarget = sandbox;
|
||||
const prior = await selectHeartbeatRuntime(db, input);
|
||||
if (!prior.selected || !prior.nativeExecution) throw new Error("Expected prior native selection");
|
||||
const priorInput = { ...prior.nativeExecution, session: { ...prior.nativeExecution.session, normalizedSessionId: sessionId } };
|
||||
const priorProfile = (await read(run.id)).runnerProfileJson;
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", nativeSessionId: sessionId,
|
||||
runnerProfileJson: { ...priorProfile, nativeExecutionInput: priorInput, sessionCheckpoint: {
|
||||
backendKind: "runner", driverKind: "codex_app_server", sessionId: "retained-provider-thread",
|
||||
providerSessionId: "retained-provider-thread", identity: { ...priorInput.binding, sessionId },
|
||||
} },
|
||||
}).where(eq(heartbeatRuns.id, run.id));
|
||||
const [nextRun] = await db.insert(heartbeatRuns).values({ companyId: agent.companyId, agentId: agent.id,
|
||||
issueId: issue.id, scopeKind: "issue", status: "running", nativeSessionId: sessionId,
|
||||
createdAt: new Date(run.createdAt.getTime() + 1000),
|
||||
}).returning();
|
||||
input.run = nextRun!;
|
||||
input.workspace.executionWorkspace.cwd = workspace!.cwd!;
|
||||
const selected = await selectHeartbeatRuntime(db, input);
|
||||
if (!selected.selected || !selected.nativeExecution) throw new Error("Expected migrated native selection");
|
||||
expect(selected.nativeExecution.session.normalizedSessionId).toBe(sessionId);
|
||||
expect(selected.nativeExecution.workspace.cwd).toBe(sameSession ? logicalCwd : workspace!.cwd);
|
||||
expect((await read(nextRun!.id)).runnerProfileJson?.nativeExecutionInput).toEqual(selected.nativeExecution);
|
||||
expect((await read(nextRun!.id)).nativeSessionId).toBe(selected.nativeExecution.session.normalizedSessionId);
|
||||
});
|
||||
|
||||
it.each([false, true])("persists the Cursor provider API placement and completion authority without a Computer lease (ephemeral: %s)", async ephemeral => {
|
||||
vi.stubEnv("PAPERCLIP_EPHEMERAL_SERVER", String(ephemeral));
|
||||
const { input, run, issue } = await fixture();
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { readNativeWorkspaceMigration, migratedNativeExecutionCwd } from "../native-runtime/native-workspace-migration.js";
|
||||
import { resolveProviderApiPlacement } from "../native-runtime/provider-api-placement.js";
|
||||
import { readClaimedNativeStopRecovery } from "../native-runtime/native-cancellation-request.js";
|
||||
import { museRunnerBroker } from "../muse-runner-broker.js";
|
||||
@@ -762,6 +763,8 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
codexCliVersion: codexCliVersion!,
|
||||
});
|
||||
}
|
||||
const workspaceMigration = persistedExecutionWorkspace?.metadata?.nativeWorkspaceMigration
|
||||
? await readNativeWorkspaceMigration(db, { companyId: agent.companyId, id: nativeExecutionWorkspaceId }) : null;
|
||||
const buildExecution = ({ normalizedSessionId, resumedSession }: { normalizedSessionId: string; resumedSession: boolean }) =>
|
||||
buildNativeExecutionInput({
|
||||
agentKeyId: agentIdentity?.keyId,
|
||||
@@ -817,7 +820,7 @@ export async function selectHeartbeatRuntime(db: Db, input: HeartbeatRuntimeSele
|
||||
// transient workspace to the run id so the native input remains durable and replayable
|
||||
// without fabricating a project-scoped execution_workspaces row.
|
||||
id: nativeExecutionWorkspaceId,
|
||||
cwd: executionWorkspace.cwd,
|
||||
cwd: migratedNativeExecutionCwd(workspaceMigration, { agentId: agent.id, issueId: issueRef.id, sessionId: normalizedSessionId }) ?? executionWorkspace.cwd,
|
||||
repoUrl: executionWorkspace.repoUrl,
|
||||
repoRef: executionWorkspace.repoRef,
|
||||
branchName: executionWorkspace.branchName,
|
||||
|
||||
@@ -152,7 +152,14 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
|
||||
: null,
|
||||
};
|
||||
const sessionConfigMetadata = await buildEffectiveRunSessionConfigMetadata(sessionConfigMetadataInput);
|
||||
let compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = [];
|
||||
const compatibleConfigMetadata: EffectiveRunSessionConfigMetadata[] = [];
|
||||
const importedManagedInstructions = Boolean(taskSession && sessionConfigMetadataInput.managedInstructions?.contentHash
|
||||
&& readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint);
|
||||
if (importedManagedInstructions) {
|
||||
compatibleConfigMetadata.push(await buildEffectiveRunSessionConfigMetadata({
|
||||
...sessionConfigMetadataInput, retainManagedInstructionPaths: true,
|
||||
}));
|
||||
}
|
||||
if (managedAiRuntime && aiBinding && taskSession &&
|
||||
readConfigFingerprintFromSessionParams(taskSession.sessionParamsJson)?.fingerprint !== sessionConfigMetadata.fingerprint) {
|
||||
const revisions = requestedAiBinding?.mode === "router"
|
||||
@@ -166,7 +173,12 @@ export async function prepareHeartbeatSession(db: Db, input: HeartbeatSessionPre
|
||||
storedIdentity: taskSession.sessionParamsJson?.paperclipAiCredentialIdentity,
|
||||
sessionIdentity: managedAiRuntime.sessionIdentity, credentialIdentity: managedAiRuntime.identity, revisions,
|
||||
});
|
||||
compatibleConfigMetadata = await Promise.all(candidates.map(candidate => buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate })));
|
||||
compatibleConfigMetadata.push(...await Promise.all(candidates.flatMap(candidate => [
|
||||
buildEffectiveRunSessionConfigMetadata({ ...sessionConfigMetadataInput, ...candidate }),
|
||||
...(importedManagedInstructions ? [buildEffectiveRunSessionConfigMetadata({
|
||||
...sessionConfigMetadataInput, ...candidate, retainManagedInstructionPaths: true,
|
||||
})] : []),
|
||||
])));
|
||||
}
|
||||
const configuredModel =
|
||||
readConfiguredModelFromAdapterConfig(runtimeConfig);
|
||||
|
||||
@@ -2839,6 +2839,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: {
|
||||
effectiveAdapterConfig: Record<string, unknown>;
|
||||
managedAiHome?: string;
|
||||
managedInstructions?: { entryFile: string; contentHash: string | null } | null;
|
||||
retainManagedInstructionPaths?: boolean;
|
||||
agentRuntimeConfig: unknown;
|
||||
issueOverrides: unknown;
|
||||
workspaceConfig: unknown;
|
||||
@@ -2858,7 +2859,7 @@ export async function buildEffectiveRunSessionConfigMetadata(input: {
|
||||
...(input.managedInstructions.contentHash !== null ? { contentHash: input.managedInstructions.contentHash } : {}),
|
||||
} : await resolveInstructionsConfigFingerprintMetadata(input.effectiveAdapterConfig);
|
||||
const effectiveAdapterConfig = { ...managedAiSessionFingerprintConfig(input.effectiveAdapterConfig, input.managedAiHome) };
|
||||
if (input.managedInstructions) {
|
||||
if (input.managedInstructions && !input.retainManagedInstructionPaths) {
|
||||
delete effectiveAdapterConfig.instructionsRootPath;
|
||||
delete effectiveAdapterConfig.instructionsFilePath;
|
||||
}
|
||||
|
||||
@@ -44,6 +44,7 @@ import {
|
||||
budgetReservations,
|
||||
heartbeatRuns,
|
||||
environmentLeases,
|
||||
executionWorkspaces,
|
||||
heartbeatRunEvents,
|
||||
issueThreadInteractions,
|
||||
issues,
|
||||
@@ -2677,6 +2678,14 @@ describe("verified native harness backups", () => {
|
||||
).toBe(false);
|
||||
});
|
||||
|
||||
it("requires verified migration continuity before restoring a retained warm session into a fresh sandbox", () => {
|
||||
const acquisition = { acquisitionOutcome: "created" as const, reusableLeaseConfigured: true, backupAvailable: true };
|
||||
expect(shouldRestoreNativeHarnessBackupIntoSandbox(acquisition)).toBe(false);
|
||||
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true })).toBe(true);
|
||||
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, backupAvailable: false })).toBe(false);
|
||||
expect(shouldRestoreNativeHarnessBackupIntoSandbox({ ...acquisition, migratedSessionContinuation: true, acquisitionOutcome: "resumed" })).toBe(false);
|
||||
});
|
||||
|
||||
it("accepts a complete digest-matched backup and rejects corruption", async () => {
|
||||
const root = await mkdtemp(join(tmpdir(), "paperclip-harness-backup-"));
|
||||
try {
|
||||
@@ -6660,6 +6669,7 @@ function leaseDb(
|
||||
: [];
|
||||
const query = {
|
||||
then: Promise.resolve(rows).then.bind(Promise.resolve(rows)),
|
||||
leftJoin: () => query,
|
||||
where: () => query,
|
||||
orderBy: () => query,
|
||||
for: () => query,
|
||||
@@ -16811,6 +16821,100 @@ describe("runnerd provider runtime wiring", () => {
|
||||
expect(runnerEnvironment).toMatchObject(paths);
|
||||
});
|
||||
|
||||
it.each([true, false])("restores a migrated warm backup into a created lease only for its exact session (matching: %s)", async matching => {
|
||||
const remoteCwd = join(isolatedStateDirectory, "remote");
|
||||
const current: NativeExecutionInputV1 = { ...execution,
|
||||
binding: { ...execution.binding, companyId: randomUUID(), agentId: randomUUID(), issueId: randomUUID(),
|
||||
runId: randomUUID(), executionWorkspaceId: randomUUID() },
|
||||
workspace: { ...execution.workspace, cwd: "/paperclip/legacy/task-workspace" },
|
||||
session: { ...execution.session, normalizedSessionId: randomUUID(), lifecyclePolicy: { mode: "warm", idleTimeoutMs: 300_000 } },
|
||||
};
|
||||
const operationId = randomUUID(), runnerInstanceId = "migrated-warm-runner";
|
||||
const receipt = { operationId,
|
||||
workspaceMetadata: { nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId } },
|
||||
metadata: { schema: "paperclip.remote-workspace-migration.v1", snapshotId: randomUUID(), nativeSessionContinuity: {
|
||||
schema: "paperclip.native-workspace-migration.v1", remoteCwd, sessions: [{ sourceRunId: randomUUID(),
|
||||
agentId: current.binding.agentId, issueId: current.binding.issueId,
|
||||
sessionId: matching ? current.session.normalizedSessionId : randomUUID(), logicalCwd: current.workspace.cwd,
|
||||
}],
|
||||
} },
|
||||
};
|
||||
const updates: Array<{ table: unknown; values: Record<string, unknown> }> = [];
|
||||
const db = leaseDb(current, {}, {}, updates), ordinarySelect = db.select.bind(db);
|
||||
db.select = ((fields?: Record<string, unknown>) => ({ from: (table: unknown) => {
|
||||
if (table !== executionWorkspaces && table !== environmentLeases) return ordinarySelect(fields as never).from(table as never);
|
||||
const rows = table === executionWorkspaces ? [receipt] : [{ metadata: {}, providerLeaseId: "sandbox-created",
|
||||
companyId: current.binding.companyId, heartbeatRunId: current.binding.runId }];
|
||||
const query = { then: Promise.resolve(rows).then.bind(Promise.resolve(rows)),
|
||||
leftJoin: () => query, where: () => query, limit: () => query };
|
||||
return query;
|
||||
} })) as unknown as Db["select"];
|
||||
const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
|
||||
if (command.args?.[0] === "--build-metadata") return { exitCode: 0, timedOut: false, stderr: "", stdout: JSON.stringify({
|
||||
schema: "paperclip-runner/runnerd-build-metadata/v1", binaryName: "paperclip-runnerd",
|
||||
packageName: "@paperclipai/paperclip-runner", binaryContractVersion: 2,
|
||||
durableSessionCapabilities: ["unlimited_runtime", "connection_lease_renewal"], prpTransportModes: ["listen_ws"],
|
||||
}) };
|
||||
if (command.args?.[0] === "--version") return { exitCode: 0, timedOut: false, stdout: "codex-cli 0.160.0", stderr: "" };
|
||||
if (command.args?.[1]?.includes("base64")) {
|
||||
try { return { exitCode: 0, timedOut: false, stdout: execFileSync("sh", command.args, { encoding: "utf8" }), stderr: "" }; }
|
||||
catch { return { exitCode: 1, timedOut: false, stdout: "", stderr: "" }; }
|
||||
}
|
||||
return { exitCode: 0, timedOut: false, stdout: "", stderr: "" };
|
||||
});
|
||||
// The sandbox boundary copies actual backup bytes; migration admission and
|
||||
// the executor's manifest, restored identity, and lease stamp checks stay real.
|
||||
const syncIn = vi.fn(async (operations: Array<{ files: Array<{ sourcePath: string; targetPath: string }> }>) => {
|
||||
for (const operation of operations) for (const file of operation.files) {
|
||||
expect(file.targetPath.startsWith(`${remoteCwd}/`)).toBe(true);
|
||||
await mkdir(dirname(file.targetPath), { recursive: true });
|
||||
await cp(file.sourcePath, file.targetPath, { recursive: true });
|
||||
}
|
||||
});
|
||||
await createRunnerdBackend({ db, execution: current, runnerInstanceId, runnerIngressAuthorized: true,
|
||||
runnerExecutionTarget: { kind: "remote", transport: "sandbox", remoteCwd, environmentId: "environment",
|
||||
leaseId: "lease-created", providerKey: "daytona", reusableLeaseConfigured: true,
|
||||
effectiveCapabilities: { runnerWebSocketIngress: true },
|
||||
sandboxLeaseAcquisition: { outcome: "created", providerLeaseId: "sandbox-created" },
|
||||
runner: { execute: remoteExecute, syncIn },
|
||||
} as never,
|
||||
});
|
||||
state.createBackend.mock.calls.at(-1)![1].codexTransportFactory!();
|
||||
const options = state.createTransport.mock.calls.at(-1)![0] as RunnerTransportOptions & { prepareExternalRunnerState: () => Promise<void> };
|
||||
const backupRoot = join(options.stateDirectory!, "failover-backups/current");
|
||||
for (const directory of resolveNativeHarnessPersistenceProfile(current).directories)
|
||||
await mkdir(join(backupRoot, directory.name), { recursive: true });
|
||||
const runnerState = durableRunnerState({ runId: current.binding.runId, normalizedSessionId: current.session.normalizedSessionId,
|
||||
runnerInstanceId, environmentLeaseId: "lease-created" }, "suspended");
|
||||
await writeFile(join(backupRoot, "runner/runner-state.json"), JSON.stringify(runnerState));
|
||||
await writeFile(join(backupRoot, "runner/codex-provider-state.json"), JSON.stringify({
|
||||
schema: "paperclip.runner.codex-provider-state.v1", lifecycle: "prepared", threadId: "retained-provider-thread",
|
||||
providerSessionId: null, activeProviderTurnId: null, config: { provider: "codex", driver: "codex_app_server" },
|
||||
}));
|
||||
await writeFile(join(backupRoot, "codex-home/sentinel.txt"), "retained provider home");
|
||||
await writeFile(join(backupRoot, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({
|
||||
backupRoot, execution: current, runnerInstanceId, sourceProviderLeaseId: "prior-sandbox",
|
||||
providerSessionIdentity: { providerSessionId: "retained-provider-thread", providerBackendSessionId: null, providerSessionIdentity: null },
|
||||
})));
|
||||
if (matching) {
|
||||
await expect(options.prepareExternalRunnerState()).resolves.toBeUndefined();
|
||||
const remoteSession = join(remoteCwd, ".paperclip-runtime/paperclip-runner/sessions",
|
||||
createHash("sha256").update(current.session.normalizedSessionId!).digest("hex"));
|
||||
expect(JSON.parse(await readFile(join(remoteSession, "runner/runner-state.json"), "utf8"))).toEqual(runnerState);
|
||||
expect(await readFile(join(remoteSession, "filesystem/codex-home/sentinel.txt"), "utf8")).toBe("retained provider home");
|
||||
expect(updates.filter(update => update.table === environmentLeases)).toEqual([expect.objectContaining({
|
||||
values: expect.objectContaining({ metadata: { nativeHarnessBackup: expect.objectContaining({
|
||||
normalizedSessionId: current.session.normalizedSessionId, runnerInstanceId,
|
||||
sourceProviderLeaseId: "prior-sandbox", authorizedProviderLeaseId: "sandbox-created",
|
||||
}) } }),
|
||||
})]);
|
||||
} else {
|
||||
await expect(options.prepareExternalRunnerState()).rejects.toThrow("backup_without_reusable_lease");
|
||||
expect(syncIn).not.toHaveBeenCalled();
|
||||
expect(updates.filter(update => update.table === environmentLeases)).toEqual([]);
|
||||
}
|
||||
});
|
||||
|
||||
it("archives failover evidence with an explicitly replaced provider session", async () => {
|
||||
const remoteCwd = join(isolatedStateDirectory, "remote");
|
||||
const remoteExecute = vi.fn(async (command: { command: string; args?: string[] }) => {
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { canRestoreMigratedNativeSession } from "./native-workspace-migration.js";
|
||||
import { prepareCursorCloudRuntime } from "./cursor-cloud-runtime.js";
|
||||
import { resolveLocalRuntimeApiUrl } from "../../runtime-api.js";
|
||||
import { providerFailureFromError } from "../../vendor/paperclip-runner/index.js";
|
||||
@@ -6638,10 +6639,11 @@ export function shouldRestoreNativeHarnessBackupIntoSandbox(input: {
|
||||
acquisitionOutcome: "created" | "resumed" | "replacement" | null;
|
||||
reusableLeaseConfigured: boolean | null | undefined;
|
||||
backupAvailable: boolean;
|
||||
migratedSessionContinuation?: boolean;
|
||||
}): boolean {
|
||||
return (
|
||||
input.acquisitionOutcome === "created" &&
|
||||
input.reusableLeaseConfigured === false &&
|
||||
(input.reusableLeaseConfigured === false || input.migratedSessionContinuation === true) &&
|
||||
input.backupAvailable
|
||||
);
|
||||
}
|
||||
@@ -13880,6 +13882,11 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
reusableLeaseConfigured:
|
||||
remoteTarget.reusableLeaseConfigured,
|
||||
backupAvailable,
|
||||
migratedSessionContinuation: sandboxLeaseAcquisition?.outcome === "created" && backupAvailable
|
||||
&& remoteTarget.reusableLeaseConfigured === true && await canRestoreMigratedNativeSession(input.db, {
|
||||
...input.execution.binding, workspaceId: input.execution.binding.executionWorkspaceId,
|
||||
sessionId: nativeSessionKey(input.execution), logicalCwd: input.execution.workspace.cwd, remoteCwd: remoteTarget.remoteCwd,
|
||||
}),
|
||||
});
|
||||
await input.trace?.record({
|
||||
name: "sandbox.lease.acquisition",
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
import path from "node:path";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { executionWorkspaces, heartbeatRuns, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
||||
import { findNativeSessionResumeRun } from "./native-session-resume.js";
|
||||
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
||||
|
||||
const SCHEMA = "paperclip.native-workspace-migration.v1";
|
||||
const object = (value: unknown): Record<string, unknown> => value && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
||||
const absolute = (value: unknown): value is string => typeof value === "string" && value !== "/"
|
||||
&& !value.includes("\0") && path.posix.isAbsolute(value) && path.posix.normalize(value) === value;
|
||||
const uuid = (value: unknown): value is string => typeof value === "string" && /^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$/.test(value);
|
||||
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "companyId" | "id">;
|
||||
type SessionBinding = { sourceRunId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string };
|
||||
export type NativeWorkspaceMigration = { schema: typeof SCHEMA; remoteCwd: string; sessions: SessionBinding[] };
|
||||
|
||||
export async function inspectNativeWorkspaceMigration(db: Db, workspace: Workspace): Promise<NativeWorkspaceMigration | null> {
|
||||
const rows = await db.execute<{ agentId: string; issueId: string; sessionId: string; createdAt: Date }>(sql`
|
||||
select s.agent_id as "agentId", r.native_issue_id as "issueId", r.native_session_id as "sessionId", r.created_at as "createdAt"
|
||||
from agent_task_sessions s join heartbeat_runs r on r.id = s.last_run_id
|
||||
and r.company_id = s.company_id and r.agent_id = s.agent_id
|
||||
where s.company_id = ${workspace.companyId} and s.adapter_type = 'paperclip_runner'
|
||||
and r.native_issue_id is not null and r.native_session_id is not null
|
||||
and r.runner_profile_json#>>'{nativeExecutionInput,binding,executionWorkspaceId}' = ${workspace.id}
|
||||
order by s.id limit 1001`);
|
||||
if (rows.length > 1000) throw new Error("native_workspace_migration_session_limit");
|
||||
const sessions: SessionBinding[] = [];
|
||||
let remoteCwd: string | null = null;
|
||||
for (const row of rows) {
|
||||
const source = await findNativeSessionResumeRun(db, { companyId: workspace.companyId, agentId: row.agentId,
|
||||
issueId: row.issueId, normalizedSessionId: row.sessionId, beforeCreatedAt: new Date(row.createdAt), includeProgressBarrier: true });
|
||||
if (!source) continue;
|
||||
const execution = parseNativeExecutionInput(source.runnerProfileJson.nativeExecutionInput);
|
||||
const [saved] = await db.select({ sync: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'` })
|
||||
.from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, workspace.companyId), eq(heartbeatRuns.id, source.id)));
|
||||
const sync = readNativeWorkspaceSyncReference(saved?.sync);
|
||||
if (execution.binding.executionWorkspaceId !== workspace.id || execution.binding.companyId !== workspace.companyId
|
||||
|| execution.binding.agentId !== row.agentId || execution.binding.issueId !== row.issueId
|
||||
|| execution.session.normalizedSessionId !== row.sessionId || !absolute(execution.workspace.cwd)
|
||||
|| !sync || sync.workspaceId !== workspace.id || !absolute(sync.remoteCwd)) throw new Error("native_workspace_migration_identity_unproven");
|
||||
if (remoteCwd && remoteCwd !== sync.remoteCwd) throw new Error("native_workspace_migration_remote_path_conflict");
|
||||
remoteCwd = sync.remoteCwd;
|
||||
sessions.push({ sourceRunId: source.id, agentId: row.agentId, issueId: row.issueId, sessionId: row.sessionId, logicalCwd: execution.workspace.cwd });
|
||||
}
|
||||
return remoteCwd ? { schema: SCHEMA, remoteCwd, sessions } : null;
|
||||
}
|
||||
|
||||
export async function readNativeWorkspaceMigration(db: Pick<Db, "select">, workspace: Workspace) {
|
||||
const [row] = await db.select({ workspaceMetadata: executionWorkspaces.metadata, metadata: workspaceOperations.metadata, operationId: workspaceOperations.id })
|
||||
.from(executionWorkspaces).leftJoin(workspaceOperations, and(
|
||||
eq(workspaceOperations.companyId, executionWorkspaces.companyId), eq(workspaceOperations.executionWorkspaceId, executionWorkspaces.id),
|
||||
sql`${workspaceOperations.id} = case when ${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId' ~ '^[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}$'
|
||||
then (${executionWorkspaces.metadata}->'nativeWorkspaceMigration'->>'operationId')::uuid end`,
|
||||
eq(workspaceOperations.phase, "remote_workspace_migration"), eq(workspaceOperations.status, "succeeded"),
|
||||
)).where(and(eq(executionWorkspaces.companyId, workspace.companyId), eq(executionWorkspaces.id, workspace.id)));
|
||||
if (!row?.workspaceMetadata?.nativeWorkspaceMigration) return null;
|
||||
const migration = object(row.metadata?.nativeSessionContinuity);
|
||||
if (object(row.workspaceMetadata.nativeWorkspaceMigration).schema !== SCHEMA || !row.operationId || row.metadata?.schema !== "paperclip.remote-workspace-migration.v1" || !uuid(row.metadata.snapshotId)
|
||||
|| migration.schema !== SCHEMA || !absolute(migration.remoteCwd) || !Array.isArray(migration.sessions)
|
||||
|| !migration.sessions.length || migration.sessions.length > 1000
|
||||
|| !migration.sessions.every(value => {
|
||||
const session = object(value);
|
||||
return [session.sourceRunId, session.agentId, session.issueId, session.sessionId].every(uuid) && absolute(session.logicalCwd);
|
||||
})) throw new Error("native_workspace_migration_receipt_invalid");
|
||||
return { ...(migration as NativeWorkspaceMigration), operationId: row.operationId };
|
||||
}
|
||||
|
||||
export function nativeWorkspaceMigrationMarker(operationId: string) {
|
||||
return { schema: SCHEMA, operationId };
|
||||
}
|
||||
|
||||
export function migratedNativeExecutionCwd(migration: NativeWorkspaceMigration | null, input: { agentId: string; issueId: string; sessionId: string }) {
|
||||
return migration?.sessions.find(session => session.agentId === input.agentId && session.issueId === input.issueId
|
||||
&& session.sessionId === input.sessionId)?.logicalCwd ?? null;
|
||||
}
|
||||
|
||||
export async function canRestoreMigratedNativeSession(db: Db, input: {
|
||||
companyId: string; workspaceId: string; agentId: string; issueId: string; sessionId: string; logicalCwd: string | null; remoteCwd: string;
|
||||
}) {
|
||||
if (input.logicalCwd === null) return false;
|
||||
const migration = await readNativeWorkspaceMigration(db, { companyId: input.companyId, id: input.workspaceId });
|
||||
return migration?.remoteCwd === input.remoteCwd && migratedNativeExecutionCwd(migration, input) === input.logicalCwd;
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import type { PluginEnvironmentWorkspaceAllocationFiles } from "@paperclipai/plugin-sdk";
|
||||
import type { EnvironmentLease } from "@paperclipai/shared";
|
||||
|
||||
export function workspaceAllocationFiles(lease: EnvironmentLease, remoteCwd: string) {
|
||||
const raw = lease.metadata?.workspaceAllocationFiles;
|
||||
if (raw === undefined) return [];
|
||||
const receipt = raw as PluginEnvironmentWorkspaceAllocationFiles | null;
|
||||
if (!receipt || receipt.providerLeaseId !== lease.providerLeaseId || receipt.remoteCwd !== remoteCwd
|
||||
|| !Array.isArray(receipt.files) || receipt.files.length > 8
|
||||
|| receipt.files.some(file => !file || typeof file.relativePath !== "string"
|
||||
|| !/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
|
||||
|| file.relativePath === ".paperclip-runtime/workspace-restored.json"
|
||||
|| typeof file.sha256 !== "string" || !/^[a-f0-9]{64}$/.test(file.sha256))
|
||||
|| new Set(receipt.files.map(file => file.relativePath)).size !== receipt.files.length) {
|
||||
throw new Error("remote_workspace_allocation_receipt_invalid");
|
||||
}
|
||||
return receipt.files;
|
||||
}
|
||||
@@ -22,6 +22,8 @@ import { readNativeWorkspaceSyncReference } from "./native-runtime/native-worksp
|
||||
import { isLegacyWorkspaceTaskRelated, readLegacyWorkspaceRecoveryAllocations, LEGACY_WORKSPACE_RECOVERY_MIGRATION, type LegacyWorkspaceRecoverySource } from "./legacy-workspace-recovery-migration.js";
|
||||
import { offlineWorkspaceMigrationBlockers, prepareNestedWorkspaceImports, validateImportedWorkspaceChildren, workspaceMigrationLayoutBlockers } from "./remote-workspace-migration-tree.js";
|
||||
|
||||
import { inspectNativeWorkspaceMigration, nativeWorkspaceMigrationMarker } from "./native-runtime/native-workspace-migration.js";
|
||||
|
||||
const SCHEMA = "paperclip.remote-workspace-migration.v1";
|
||||
type Workspace = typeof executionWorkspaces.$inferSelect;
|
||||
type ReadDb = Pick<Db, "select">;
|
||||
@@ -233,6 +235,7 @@ export async function migrateRemoteWorkspace(input: {
|
||||
const current = await inspect(tx as unknown as Db, input);
|
||||
if (current.authority) return { workspace, authority: current.authority, operation: null, source: current.source, sourceFingerprint: "", issueBinding: current.issueBinding, retainedDirectory: null, archivedGitSource: null, legacyRecoverySources: [] as LegacyWorkspaceRecoverySource[] };
|
||||
if (current.reasons.length) throw new Error(`remote_workspace_migration_blocked:${current.reasons.join(",")}`);
|
||||
const nativeSessionContinuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
|
||||
const source = remoteWorkspaceSourceSchema.parse(input.source ?? current.source);
|
||||
const sourceFingerprint = fingerprint(workspace);
|
||||
const marker = current.marker;
|
||||
@@ -241,6 +244,7 @@ export async function migrateRemoteWorkspace(input: {
|
||||
const [operation] = await tx.select().from(workspaceOperations).where(and(eq(workspaceOperations.companyId, workspace.companyId),
|
||||
eq(workspaceOperations.executionWorkspaceId, workspace.id), eq(workspaceOperations.id, marker.operationId), eq(workspaceOperations.phase, "remote_workspace_migration")));
|
||||
if (!operation) throw new Error("remote_workspace_migration_operation_missing");
|
||||
if (canonical(operation.metadata?.nativeSessionContinuity ?? null) !== canonical(nativeSessionContinuity)) throw new Error("native_workspace_migration_source_changed");
|
||||
// Older operations only admitted the task's current workspace. Preserve
|
||||
// that binding when resuming an import journaled before this field existed.
|
||||
const issueBinding = Object.hasOwn(operation.metadata ?? {}, "sourceIssueBinding")
|
||||
@@ -254,6 +258,7 @@ export async function migrateRemoteWorkspace(input: {
|
||||
const [operation] = await tx.insert(workspaceOperations).values({ companyId: workspace.companyId, executionWorkspaceId: workspace.id,
|
||||
issueId: workspace.sourceIssueId, cwd: workspace.cwd ?? workspace.providerRef, phase: "remote_workspace_migration", status: "running",
|
||||
metadata: { schema: SCHEMA, sourceFingerprint, source, sourceIssueBinding: current.issueBinding,
|
||||
...(nativeSessionContinuity ? { nativeSessionContinuity } : {}),
|
||||
nestedWorkspaceScope: { version: 1, state: "pending" },
|
||||
...(current.legacyRecoverySources.length ? { legacyRecoverySources: current.legacyRecoverySources } : {}),
|
||||
...(current.archivedGitSource ? { archivedGitSource: gitSourceProof(current.archivedGitSource) } : {}),
|
||||
@@ -347,9 +352,12 @@ export async function migrateRemoteWorkspace(input: {
|
||||
|| saved.sourceOperationId !== child.operationId) throw new Error("nested_workspace_snapshot_changed");
|
||||
}
|
||||
if (canonical(validation.legacyRecoverySources) !== canonical(reserved.legacyRecoverySources)) throw new Error("remote_workspace_migration_source_changed");
|
||||
const continuity = await inspectNativeWorkspaceMigration(tx as unknown as Db, workspace);
|
||||
if (canonical(continuity) !== canonical(operation.metadata?.nativeSessionContinuity ?? null)) throw new Error("native_workspace_migration_source_changed");
|
||||
const authority: RemoteWorkspaceAuthority = { schema: "paperclip.remote-workspace.v1", epoch: 1, snapshotId: ref.id,
|
||||
source: reserved.source, location: null, writerRunId: null, dirty: false };
|
||||
const metadata: Record<string, unknown> = { ...workspace.metadata, remoteWorkspace: authority }; delete metadata.remoteWorkspaceMigration;
|
||||
if (continuity) metadata.nativeWorkspaceMigration = nativeWorkspaceMigrationMarker(operation.id);
|
||||
if (reserved.legacyRecoverySources.length) {
|
||||
metadata.legacyWorkspaceRecoveryMigration = { schema: LEGACY_WORKSPACE_RECOVERY_MIGRATION, operationId: operation.id };
|
||||
for (const source of reserved.legacyRecoverySources) {
|
||||
|
||||
@@ -161,6 +161,53 @@ describe("remote workspace transport safety with real sandbox commands", () => {
|
||||
} finally { await captured.cleanup(); }
|
||||
});
|
||||
|
||||
it.each(["owned", "changed", "unrelated", "symlink", "interrupted"])("preserves only verified allocation files during a migrated restore: %s", async scenario => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "retained user work");
|
||||
const { remote, execute } = target();
|
||||
const captured = await captureRemoteWorkspaceArchive({ target: remote, remoteCwd: workspace, operationId: randomUUID() });
|
||||
try {
|
||||
const root = path.join(scratch, "allocated"), relativePath = ".paperclip-runtime/reusable-sandbox-lease.json";
|
||||
const bytes = JSON.stringify({ version: 1, token: "exact-allocation-token", provider: "daytona" });
|
||||
await fs.mkdir(path.join(root, ".paperclip-runtime"), { recursive: true });
|
||||
const sentinel = path.join(root, relativePath);
|
||||
await fs.writeFile(sentinel, bytes);
|
||||
const input = { target: remote, remoteCwd: root, archivePath: captured.archivePath, manifestDigest: captured.manifestDigest,
|
||||
restorationIdentity: "d".repeat(64), allowEmptyRoot: true,
|
||||
allocationFiles: [{ relativePath, sha256: createHash("sha256").update(bytes).digest("hex") }] };
|
||||
if (scenario === "changed") await fs.writeFile(sentinel, "foreign marker");
|
||||
if (scenario === "unrelated") await fs.writeFile(path.join(root, ".paperclip-runtime", "unrelated.txt"), "keep");
|
||||
if (scenario === "symlink") {
|
||||
const outside = path.join(scratch, "outside-marker"); await fs.writeFile(outside, bytes);
|
||||
await fs.unlink(sentinel); await fs.symlink(outside, sentinel);
|
||||
}
|
||||
if (["changed", "unrelated", "symlink"].includes(scenario)) {
|
||||
await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow();
|
||||
expect(await fs.readFile(sentinel, "utf8")).toBe(scenario === "changed" ? "foreign marker" : bytes);
|
||||
expect(await fs.lstat(path.join(root, "saved.txt")).catch(() => null)).toBeNull();
|
||||
return;
|
||||
}
|
||||
if (scenario === "interrupted") {
|
||||
const run = execute.getMockImplementation()!; let interrupt = true;
|
||||
execute.mockImplementation(async command => {
|
||||
if (interrupt && command.args?.[1]?.includes("remote_workspace_restore_failed")) {
|
||||
interrupt = false;
|
||||
return run({ ...command, args: command.args.map((arg, index) => index === 1
|
||||
? arg.replace("try{await fs.rename(staged,root);}", "throw Error('interrupted');try{await fs.rename(staged,root);}") : arg) });
|
||||
}
|
||||
return run(command);
|
||||
});
|
||||
await expect(restoreRemoteWorkspaceDirectory(input)).rejects.toThrow();
|
||||
expect(await fs.readFile(path.join(root + ".allocation-" + input.restorationIdentity, relativePath), "utf8")).toBe(bytes);
|
||||
}
|
||||
await restoreRemoteWorkspaceDirectory(input);
|
||||
expect(await fs.readFile(sentinel, "utf8")).toBe(bytes);
|
||||
expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("retained user work");
|
||||
await fs.writeFile(path.join(root, "saved.txt"), "later user edits");
|
||||
await restoreRemoteWorkspaceDirectory(input);
|
||||
expect(await fs.readFile(path.join(root, "saved.txt"), "utf8")).toBe("later user edits");
|
||||
} finally { await captured.cleanup(); }
|
||||
});
|
||||
|
||||
it("keeps a completed restore available when the staging cleanup reply is lost", async () => {
|
||||
await fs.writeFile(path.join(workspace, "saved.txt"), "durable synthetic files\n");
|
||||
const { remote, execute } = target();
|
||||
|
||||
@@ -279,7 +279,8 @@ export async function captureRemoteWorkspaceArchive(input: {
|
||||
}
|
||||
|
||||
export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string;
|
||||
target: RemoteWorkspaceTarget; remoteCwd: string; archivePath: string; manifestDigest: string; restorationIdentity: string; allowEmptyRoot?: boolean;
|
||||
allocationFiles?: Array<{ relativePath: string; sha256: string }>;
|
||||
}) {
|
||||
if (!/^[a-f0-9]{64}$/.test(input.restorationIdentity)) throw new Error("remote_workspace_restore_identity_invalid");
|
||||
const id = input.restorationIdentity;
|
||||
@@ -292,25 +293,54 @@ export async function restoreRemoteWorkspaceDirectory(input: {
|
||||
}]);
|
||||
// Publish the restored folder and its identity together. A lost successful
|
||||
// reply can then adopt the same files without repeating a destructive restore.
|
||||
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process');
|
||||
(async()=>{const [helper,archive,root,digest,identity]=process.argv.slice(1);
|
||||
if(!/^[a-f0-9]{64}$/.test(identity))throw Error('invalid_identity');
|
||||
const prior=await fs.lstat(root).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
if(prior){
|
||||
if(!prior.isDirectory()||prior.isSymbolicLink()||await fs.realpath(root)!==root)throw Error('unsafe_root');
|
||||
const runtime=path.join(root,'.paperclip-runtime'),marker=path.join(runtime,'workspace-restored.json');
|
||||
const dir=await fs.lstat(runtime);if(!dir.isDirectory()||dir.isSymbolicLink())throw Error('unsafe_receipt');
|
||||
const file=await fs.lstat(marker);if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
|
||||
const restore = String.raw`const fs=require('node:fs/promises'),path=require('node:path'),cp=require('node:child_process'),crypto=require('node:crypto');
|
||||
(async()=>{const [helper,archive,root,digest,identity,allowEmpty,rawFiles]=process.argv.slice(1),files=JSON.parse(rawFiles);
|
||||
if(!/^[a-f0-9]{64}$/.test(identity)||!Array.isArray(files)||files.length>8)throw Error('invalid_identity');
|
||||
for(const file of files)if(!/^\.paperclip-runtime\/[a-zA-Z0-9_-]+\.json$/.test(file.relativePath)
|
||||
||file.relativePath==='.paperclip-runtime/workspace-restored.json'||!/^[a-f0-9]{64}$/.test(file.sha256))throw Error('invalid_allocation');
|
||||
if(new Set(files.map(file=>file.relativePath)).size!==files.length)throw Error('invalid_allocation');
|
||||
const stat=async p=>fs.lstat(p).catch(e=>{if(e.code==='ENOENT')return null;throw e});
|
||||
const directory=async p=>{const s=await fs.lstat(p);if(!s.isDirectory()||s.isSymbolicLink()||await fs.realpath(p)!==p)throw Error('unsafe_root')};
|
||||
const allocation=async p=>{
|
||||
await directory(p);const names=await fs.readdir(p);
|
||||
if(names.length===0&&files.length===0)return [];
|
||||
if(names.length!==1||names[0]!=='.paperclip-runtime'||!files.length)throw Error('occupied_root');
|
||||
const runtime=path.join(p,'.paperclip-runtime');await directory(runtime);
|
||||
const contents=await fs.readdir(runtime),expected=files.map(file=>path.basename(file.relativePath));
|
||||
if(contents.length!==expected.length||contents.some(name=>!expected.includes(name)))throw Error('occupied_root');
|
||||
return Promise.all(files.map(async file=>{const name=path.join(p,file.relativePath),s=await fs.lstat(name);
|
||||
if(!s.isFile()||s.isSymbolicLink()||s.nlink!==1||s.size>16384)throw Error('unsafe_allocation');
|
||||
const content=await fs.readFile(name);if(crypto.createHash('sha256').update(content).digest('hex')!==file.sha256)throw Error('allocation_changed');
|
||||
return {name:path.basename(name),content};}));
|
||||
};
|
||||
const parked=root+'.allocation-'+identity;
|
||||
if(await stat(parked)){if(allowEmpty!=='true')throw Error('unexpected_allocation');await allocation(parked);if(!await stat(root))await fs.rename(parked,root);}
|
||||
const prior=await stat(root);if(prior)await directory(root);
|
||||
const marker=path.join(root,'.paperclip-runtime','workspace-restored.json');
|
||||
if(prior&&await stat(marker)){
|
||||
await directory(path.dirname(marker));const file=await fs.lstat(marker);
|
||||
if(!file.isFile()||file.isSymbolicLink()||file.nlink!==1||file.size>16384)throw Error('unsafe_receipt');
|
||||
const receipt=JSON.parse(await fs.readFile(marker,'utf8'));
|
||||
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');return;
|
||||
if(receipt.identity!==identity||receipt.manifestDigest!==digest)throw Error('workspace_owner_changed');
|
||||
if(await stat(parked)){await allocation(parked);await fs.rm(parked,{recursive:true});}return;
|
||||
}
|
||||
if(prior&&allowEmpty!=='true')throw Error('occupied_root');
|
||||
const retained=prior?await allocation(root):[];
|
||||
if(!prior&&files.length)throw Error('allocation_missing');
|
||||
const staged=await fs.mkdtemp(root+'.restoring-');
|
||||
await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576});
|
||||
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
|
||||
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
|
||||
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');await fs.rename(staged,root);
|
||||
try{
|
||||
await require('node:util').promisify(cp.execFile)(process.execPath,[helper,'restore',archive,staged,digest],{maxBuffer:1048576});
|
||||
const runtime=path.join(staged,'.paperclip-runtime');await fs.mkdir(runtime,{mode:0o700});
|
||||
for(const file of retained)await fs.writeFile(path.join(runtime,file.name),file.content,{flag:'wx',mode:0o600});
|
||||
await fs.writeFile(path.join(runtime,'workspace-restored.json'),JSON.stringify({identity,manifestDigest:digest}),{flag:'wx',mode:0o600});
|
||||
if(await fs.realpath(path.dirname(root))!==path.dirname(root))throw Error('unsafe_root');
|
||||
if(prior){await allocation(root);if(await stat(parked))throw Error('occupied_parking');await fs.rename(root,parked);}
|
||||
try{await fs.rename(staged,root);}catch(e){if(prior&&!await stat(root))await fs.rename(parked,root);throw e;}
|
||||
if(prior){await allocation(parked);await fs.rm(parked,{recursive:true});}
|
||||
}finally{await fs.rm(staged,{recursive:true,force:true});}
|
||||
})().catch(()=>{process.stderr.write('remote_workspace_restore_failed');process.exitCode=1});`;
|
||||
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd, input.manifestDigest, input.restorationIdentity]);
|
||||
await command(input.target, "restore", ["-e", restore, staged.helper, staged.archive, input.remoteCwd,
|
||||
input.manifestDigest, input.restorationIdentity, String(input.allowEmptyRoot === true), JSON.stringify(input.allocationFiles ?? [])]);
|
||||
// Publish the restored location before cleanup can fail, stall, or lose its
|
||||
// reply. The owner journals this receipt and then reclaims disposable staging.
|
||||
return { operationId, stagingRoot: parent, cleanupPending: true };
|
||||
|
||||
@@ -1,4 +1,7 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { parseNativeExecutionInput } from "../vendor/paperclip-runner/index.js";
|
||||
import { readNativeWorkspaceMigration, migratedNativeExecutionCwd, canRestoreMigratedNativeSession } from "./native-runtime/native-workspace-migration.js";
|
||||
import { buildNativeHarnessBackupManifest, verifyNativeHarnessBackup } from "./native-runtime/native-session-executor.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { execFile, spawn } from "node:child_process";
|
||||
import net from "node:net";
|
||||
import { promisify } from "node:util";
|
||||
@@ -8,7 +11,7 @@ import path from "node:path";
|
||||
import { pathToFileURL } from "node:url";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { and, eq, sql } from "drizzle-orm";
|
||||
import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments,
|
||||
import { activityLog, agents, agentTaskSessions, agentWakeupRequests, applyPendingMigrations, closeRegisteredClients, companies, createDb, environmentLeases, environments,
|
||||
executionWorkspaces, executionWorkspaceSnapshots, heartbeatRunEvents, heartbeatRuns, issueRecoveryActions, issues, livenessDb, nativeRunFinalizations, projects, projectWorkspaces, workspaceOperations } from "@paperclipai/db";
|
||||
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
||||
import type { CommandManagedRuntimeRunner } from "@paperclipai/adapter-utils/command-managed-runtime";
|
||||
@@ -90,6 +93,7 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
await db.delete(nativeRunFinalizations).where(eq(nativeRunFinalizations.companyId, ids.company));
|
||||
await db.delete(issueRecoveryActions).where(eq(issueRecoveryActions.companyId, ids.company));
|
||||
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, ids.company));
|
||||
await db.delete(agentTaskSessions).where(eq(agentTaskSessions.companyId, ids.company));
|
||||
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, ids.company));
|
||||
await db.delete(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, ids.company));
|
||||
await db.update(issues).set({ executionWorkspaceId: null }).where(eq(issues.companyId, ids.company));
|
||||
@@ -437,6 +441,101 @@ suite("remote workspace lifecycle with real Postgres and independent filesystem
|
||||
try { expect(await fs.readFile(path.join(browse.root, "report.md"), "utf8")).toContain("useful saved result"); } finally { await browse.cleanup(); }
|
||||
}, 60_000);
|
||||
|
||||
it.each(["per_turn", "warm", "occupied", "foreign-target", "foreign-allocation", "forged-receipt"])("preserves retained native workspace identity through migration: %s", async scenario => {
|
||||
const original = path.join(root, "old-controller", "workspace"); await fs.mkdir(original, { recursive: true });
|
||||
await fs.writeFile(path.join(original, "saved.txt"), "retained work");
|
||||
const remote = await target("legacy-provider-root");
|
||||
const [local] = await db.insert(executionWorkspaces).values({ companyId: ids.company, sourceIssueId: ids.issue,
|
||||
mode: "isolated_workspace", strategyType: "project_primary", providerType: "local_fs", name: "Retained session",
|
||||
cwd: original, providerRef: original }).returning();
|
||||
const sessionId = randomUUID(), runnerInstanceId = randomUUID();
|
||||
const execution = parseNativeExecutionInput({ schema: "paperclip.native-execution-input.v1",
|
||||
provider: { kind: "codex", model: null }, binding: { companyId: ids.company, agentId: ids.agent,
|
||||
issueId: ids.issue, runId: ids.run, executionWorkspaceId: local.id },
|
||||
task: { identifier: "RWP-1", title: "Retained work", description: null, prompt: "Continue.", workMode: "standard" },
|
||||
workspace: { cwd: original, repoUrl: null, repoRef: null, branchName: null },
|
||||
session: { normalizedSessionId: sessionId, driverKind: "codex_app_server", protocolVersion: 1,
|
||||
lifecyclePolicy: { mode: scenario === "warm" ? "warm" : "per_turn", idleTimeoutMs: scenario === "warm" ? 120000 : null } },
|
||||
completionContract: { id: randomUUID(), sha256: "fixture", schemaVersion: "paperclip.completion-contract.v1",
|
||||
contract: { revision: "1", objective: "Continue", criteria: [{ id: "objective", requirement: "Save work." }] } },
|
||||
credentialBindings: [], interactionResponses: [],
|
||||
});
|
||||
if (execution.schema !== "paperclip.native-execution-input.v1") throw new Error("Unexpected fixture schema");
|
||||
await db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date(), nativeSessionId: sessionId,
|
||||
runnerInstanceId, runnerProfileJson: { nativeExecutionInput: execution, sessionCheckpoint: { sessionId }, nativeWorkspaceSync: {
|
||||
schema: "paperclip.native-workspace-sync/v3", state: "finalized", descriptorSha256: "a".repeat(64), baselineSha256: "b".repeat(64),
|
||||
finalHostSha256: "c".repeat(64), workspaceId: local.id, leaseId: ids.lease, providerLeaseId: `allocation-${ids.lease}`,
|
||||
remoteCwd: remote.remoteCwd, resourceDisposition: "destroy",
|
||||
} } }).where(eq(heartbeatRuns.id, ids.run));
|
||||
await db.insert(agentTaskSessions).values({ companyId: ids.company, agentId: ids.agent, adapterType: "paperclip_runner",
|
||||
taskKey: ids.issue, lastRunId: ids.run });
|
||||
await db.update(issues).set({ executionWorkspaceId: local.id }).where(eq(issues.id, ids.issue));
|
||||
const backupRoot = path.join(root, "private-state"), current = path.join(backupRoot, "failover-backups", "current");
|
||||
await fs.mkdir(path.join(current, "runner"), { recursive: true }); await fs.mkdir(path.join(current, "codex-home"));
|
||||
await fs.writeFile(path.join(current, "runner", "state.json"), "unchanged opaque provider state");
|
||||
await fs.writeFile(path.join(current, "manifest.json"), JSON.stringify(buildNativeHarnessBackupManifest({ backupRoot: current,
|
||||
execution, runnerInstanceId, sourceProviderLeaseId: "retained-allocation", providerSessionIdentity: {
|
||||
providerSessionId: "retained-thread", providerBackendSessionId: "retained-backend", providerSessionIdentity: null } })));
|
||||
expect(verifyNativeHarnessBackup({ root: backupRoot, execution, runnerInstanceId })).not.toBeNull();
|
||||
await migrateRemoteWorkspace({ db, storage, companyId: ids.company, workspaceId: local.id, source: { kind: "empty" }, apply: true });
|
||||
const migrated = await workspace(local.id);
|
||||
const receipt = await readNativeWorkspaceMigration(db, migrated);
|
||||
expect(receipt?.remoteCwd).toBe(remote.remoteCwd);
|
||||
const continuation = { companyId: ids.company, workspaceId: local.id, agentId: ids.agent, issueId: ids.issue, sessionId, logicalCwd: original, remoteCwd: remote.remoteCwd };
|
||||
expect(await canRestoreMigratedNativeSession(db, continuation)).toBe(true);
|
||||
expect(await canRestoreMigratedNativeSession(db, { ...continuation, sessionId: randomUUID() })).toBe(false);
|
||||
expect(await canRestoreMigratedNativeSession(db, { ...continuation, remoteCwd: "/foreign" })).toBe(false);
|
||||
expect(migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId })).toBe(original);
|
||||
expect(migratedNativeExecutionCwd(receipt, { agentId: randomUUID(), issueId: ids.issue, sessionId })).toBeNull();
|
||||
await fs.rm(path.join(root, "old-controller"), { recursive: true });
|
||||
ids.run = randomUUID();
|
||||
await db.insert(heartbeatRuns).values({ id: ids.run, companyId: ids.company, agentId: ids.agent, nativeIssueId: ids.issue, runtimeMode: "native", status: "running" });
|
||||
await db.update(environmentLeases).set({ heartbeatRunId: ids.run }).where(eq(environmentLeases.id, ids.lease));
|
||||
const reserved = await reserveRemoteWorkspace({ db, storage, companyId: ids.company, issueId: ids.issue, runId: ids.run,
|
||||
agentId: ids.agent, environmentId: ids.environment, existingWorkspaceId: local.id, projectId: null, projectWorkspaceId: null, source: { kind: "empty" } });
|
||||
const preparation = { db, storage, workspace: reserved, runId: ids.run, lease: await lease(), target: remote };
|
||||
const allocationPath = ".paperclip-runtime/reusable-sandbox-lease.json";
|
||||
const allocationBytes = JSON.stringify({ version: 1, token: "owned-allocation", companyId: ids.company,
|
||||
environmentId: ids.environment, agentId: ids.agent, executionWorkspaceId: local.id, provider: "daytona" });
|
||||
await fs.mkdir(path.join(remote.remoteCwd, ".paperclip-runtime"));
|
||||
await fs.writeFile(path.join(remote.remoteCwd, allocationPath), allocationBytes);
|
||||
preparation.lease.metadata = { ...preparation.lease.metadata, workspaceAllocationFiles: {
|
||||
providerLeaseId: preparation.lease.providerLeaseId, remoteCwd: remote.remoteCwd,
|
||||
files: [{ relativePath: allocationPath, sha256: createHash("sha256").update(allocationBytes).digest("hex") }],
|
||||
} };
|
||||
|
||||
if (scenario === "forged-receipt") {
|
||||
await db.update(executionWorkspaces).set({ metadata: { ...reserved.metadata,
|
||||
nativeWorkspaceMigration: { schema: "paperclip.native-workspace-migration.v1", operationId: randomUUID() } } }).where(eq(executionWorkspaces.id, local.id));
|
||||
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_receipt_invalid"); return;
|
||||
}
|
||||
if (scenario === "occupied") {
|
||||
await fs.writeFile(path.join(remote.remoteCwd, "unrelated.txt"), "do not overwrite");
|
||||
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow();
|
||||
expect(await fs.readFile(path.join(remote.remoteCwd, "unrelated.txt"), "utf8")).toBe("do not overwrite"); return;
|
||||
}
|
||||
if (scenario === "foreign-allocation") {
|
||||
const receipt = preparation.lease.metadata.workspaceAllocationFiles as { providerLeaseId: string };
|
||||
receipt.providerLeaseId = "another-allocation";
|
||||
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("remote_workspace_allocation_receipt_invalid");
|
||||
expect(await fs.readFile(path.join(remote.remoteCwd, allocationPath), "utf8")).toBe(allocationBytes); return;
|
||||
}
|
||||
if (scenario === "foreign-target") {
|
||||
remote.remoteCwd = path.join(root, "different-provider-root");
|
||||
await expect(prepareRemoteWorkspace(preparation)).rejects.toThrow("native_workspace_migration_target_changed"); return;
|
||||
}
|
||||
const prepared = await prepareRemoteWorkspace(preparation);
|
||||
expect(prepared.remoteRoot).toBe(remote.remoteCwd);
|
||||
expect(await fs.readFile(path.join(prepared.remoteRoot, allocationPath), "utf8")).toBe(allocationBytes);
|
||||
expect(await fs.readFile(path.join(prepared.remoteRoot, "saved.txt"), "utf8")).toBe("retained work");
|
||||
const successor = { ...execution, workspace: { ...execution.workspace,
|
||||
cwd: migratedNativeExecutionCwd(receipt, { agentId: ids.agent, issueId: ids.issue, sessionId }) ?? prepared.remoteRoot } };
|
||||
expect(verifyNativeHarnessBackup({ root: backupRoot, execution: successor, runnerInstanceId })).not.toBeNull();
|
||||
expect(verifyNativeHarnessBackup({ root: backupRoot, execution: { ...successor, workspace: { ...successor.workspace,
|
||||
cwd: prepared.remoteRoot } }, runnerInstanceId })).toBeNull();
|
||||
expect(await fs.readFile(path.join(current, "runner", "state.json"), "utf8")).toBe("unchanged opaque provider state");
|
||||
}, 60_000);
|
||||
|
||||
it.each(["path", "file-url"] as const)("cold restores an imported secondary %s repository and acquires only a new descriptor", async sourceKind => {
|
||||
const origin = path.join(root, "import-origin"), original = path.join(root, "old-checkout");
|
||||
await fs.mkdir(origin); await fs.mkdir(original);
|
||||
|
||||
@@ -1,3 +1,5 @@
|
||||
import { workspaceAllocationFiles } from "./remote-workspace-allocation.js";
|
||||
import { readNativeWorkspaceMigration } from "./native-runtime/native-workspace-migration.js";
|
||||
import { createHash, randomUUID } from "node:crypto";
|
||||
import { existsSync } from "node:fs";
|
||||
import { mkdtemp, rm, statfs } from "node:fs/promises";
|
||||
@@ -352,7 +354,10 @@ export async function prepareRemoteWorkspace(input: {
|
||||
const sameAllocation = initial.location?.providerLeaseId === input.lease.providerLeaseId
|
||||
&& initial.location?.environmentId === input.lease.environmentId;
|
||||
if (initial.location && !sameAllocation && initial.dirty) throw conflict("The prior workspace has unsaved work; recover it before replacing its allocation.");
|
||||
const remoteRoot = sameAllocation ? initial.location!.remoteCwd : path.posix.join(target.remoteCwd, ".paperclip-workspaces",
|
||||
const migration = input.workspace.metadata?.nativeWorkspaceMigration
|
||||
? await readNativeWorkspaceMigration(input.db, input.workspace) : null;
|
||||
if (migration && migration.remoteCwd !== target.remoteCwd) throw new Error("native_workspace_migration_target_changed");
|
||||
const remoteRoot = sameAllocation ? initial.location!.remoteCwd : migration?.remoteCwd ?? path.posix.join(target.remoteCwd, ".paperclip-workspaces",
|
||||
publishedWorkspaceDirectory(input.workspace, initial) ?? `${input.workspace.id}-${initial.epoch}-${input.lease.id}`);
|
||||
const preparationIdentity = createHash("sha256").update(JSON.stringify([input.workspace.companyId, input.workspace.id,
|
||||
input.runId, initial.epoch, input.lease.id, input.lease.providerLeaseId, remoteRoot, initial.source])).digest("hex");
|
||||
@@ -369,7 +374,8 @@ export async function prepareRemoteWorkspace(input: {
|
||||
const archivePath = path.join(scratch, "workspace.pcws");
|
||||
await loadWorkspaceSnapshotArchive({ storage: input.storage ?? getStorageService(), reference: snapshot, targetPath: archivePath });
|
||||
restoredExport = await restoreRemoteWorkspaceDirectory({ target, remoteCwd: remoteRoot, archivePath, manifestDigest: snapshot.manifestDigest,
|
||||
restorationIdentity: preparationIdentity });
|
||||
restorationIdentity: preparationIdentity, allowEmptyRoot: migration !== null,
|
||||
allocationFiles: migration ? workspaceAllocationFiles(input.lease, remoteRoot) : [] });
|
||||
} finally { await rm(scratch, { recursive: true, force: true }); }
|
||||
} else {
|
||||
try {
|
||||
|
||||
@@ -0,0 +1,66 @@
|
||||
import { createReadStream, createWriteStream } from "node:fs";
|
||||
import { promises as fs } from "node:fs";
|
||||
import { randomUUID } from "node:crypto";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { pipeline } from "node:stream/promises";
|
||||
import { Writable } from "node:stream";
|
||||
import { and, asc, eq, gt, sql } from "drizzle-orm";
|
||||
import { companySkillRepositorySnapshots as snapshots, type Db, type SkillPackageStorageReceipt } from "@paperclipai/db";
|
||||
import type { StorageService } from "../storage/index.js";
|
||||
import { boundedHashStream } from "./skill-package-files.js";
|
||||
import { skillPackageStore } from "./skill-package-store.js";
|
||||
|
||||
export async function migrateLocalSkillPackageObjects(db: Db, input: {
|
||||
source: StorageService; destination: StorageService; writersStopped: boolean; signal: AbortSignal; after?: string;
|
||||
}) {
|
||||
if (!input.writersStopped || process.env.PAPERCLIP_EPHEMERAL_MIGRATION !== "true"
|
||||
|| input.source.provider !== "local_disk" || input.destination.provider !== "s3") throw new Error("offline_skill_object_migration_required");
|
||||
const rows = await db.select({ id: snapshots.id, companyId: snapshots.companyId, storage: snapshots.storage })
|
||||
.from(snapshots).where(and(sql`${snapshots.storage}->>'provider' = 'local_disk'`,
|
||||
input.after ? gt(snapshots.id, input.after) : undefined)).orderBy(asc(snapshots.id)).limit(50);
|
||||
const source = skillPackageStore(db, input.source);
|
||||
for (const row of rows) {
|
||||
input.signal.throwIfAborted();
|
||||
const original = row.storage!;
|
||||
if (!await source.verifyDurable(row.companyId, row.id, input.signal)) throw new Error("local_skill_object_verification_failed");
|
||||
const token = randomUUID(), prefix = `${row.companyId}/skill-packages/${token}`;
|
||||
const upload = { token, legacy: true, provider: "s3" as const, archiveKey: `${prefix}/files.tar.gz`,
|
||||
manifestKey: `${prefix}/manifest.json`, expiresAt: new Date(Date.now() + 10 * 60_000).toISOString() };
|
||||
const unchanged = sql`${snapshots.storage} = ${JSON.stringify(original)}::jsonb`;
|
||||
const [claimed] = await db.update(snapshots).set({ upload }).where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
|
||||
sql`(${snapshots.upload} is null or (${snapshots.upload}->>'expiresAt')::timestamptz <= now())`)).returning({ id: snapshots.id });
|
||||
if (!claimed) throw new Error("skill_object_migration_owner_busy");
|
||||
const directory = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-skill-object-migration-"));
|
||||
try {
|
||||
for (const object of [
|
||||
{ source: original.archiveKey, target: upload.archiveKey, bytes: original.archiveBytes, sha256: original.archiveSha256,
|
||||
filename: "files.tar.gz", contentType: "application/gzip" },
|
||||
{ source: original.manifestKey, target: upload.manifestKey, bytes: original.manifestBytes, sha256: original.manifestSha256,
|
||||
filename: "manifest.json", contentType: "application/json" },
|
||||
]) {
|
||||
const filename = path.join(directory, object.filename);
|
||||
const read = await input.source.getObject(row.companyId, object.source, { signal: input.signal });
|
||||
const originalDigest = boundedHashStream(object.bytes);
|
||||
await pipeline(read.stream, originalDigest.stream, createWriteStream(filename, { flags: "wx", mode: 0o600 }), { signal: input.signal });
|
||||
const actual = originalDigest.result();
|
||||
if (actual.bytes !== object.bytes || actual.sha256 !== object.sha256) throw new Error("local_skill_object_changed");
|
||||
await input.destination.putFile({ companyId: row.companyId, namespace: "skill-packages", objectKey: object.target,
|
||||
originalFilename: object.filename, contentType: object.contentType, body: createReadStream(filename),
|
||||
byteSize: object.bytes, sha256: object.sha256, signal: input.signal });
|
||||
const saved = await input.destination.getObject(row.companyId, object.target, { signal: input.signal });
|
||||
const remoteDigest = boundedHashStream(object.bytes);
|
||||
await pipeline(saved.stream, remoteDigest.stream, new Writable({ write(_chunk, _encoding, done) { done(); } }), { signal: input.signal });
|
||||
const verified = remoteDigest.result();
|
||||
if (verified.bytes !== object.bytes || verified.sha256 !== object.sha256) throw new Error("remote_skill_object_verification_failed");
|
||||
}
|
||||
input.signal.throwIfAborted();
|
||||
const receipt: SkillPackageStorageReceipt = { ...original, provider: "s3", archiveKey: upload.archiveKey, manifestKey: upload.manifestKey };
|
||||
const [published] = await db.update(snapshots).set({ storage: receipt, upload: null, migrationAt: null })
|
||||
.where(and(eq(snapshots.companyId, row.companyId), eq(snapshots.id, row.id), unchanged,
|
||||
sql`${snapshots.upload}->>'token' = ${token}`)).returning({ id: snapshots.id });
|
||||
if (!published) throw new Error("skill_object_migration_owner_changed");
|
||||
} finally { await fs.rm(directory, { recursive: true, force: true }); }
|
||||
}
|
||||
return { inspected: rows.length, migrated: rows.length, after: rows.at(-1)?.id ?? null };
|
||||
}
|
||||
Reference in new issue
Block a user