mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 22:13:59 +01:00
## Thinking Path > - Paperclip manages durable tasks and agent work through the control plane. > - An ephemeral web server needs existing workspace files in shared storage. > - Failed historical runs can still own recoverable sandbox files or unfinished coordinator state. > - Moving local files must preserve those obligations without replaying the old agent turn. > - Migration journals the exact source and fences old copyback; the recovery owner saves verified sandbox files separately. > - New user input can continue only after matching preservation receipts and independent execution-stop checks. > - Accepted results retain their finalization owner and must have its durable workspace receipt before storage moves. ## Linked Issues or Issue Description Builds on merged #18826 and #18835. **What happened?** Existing workspaces with failed native copyback could not migrate to remote storage. Simply changing their storage authority would let later recovery target a retired local directory, lose child-task attachments, or leave a recovered task unable to accept a new turn. **Expected behavior** Preserve local files, exact sandbox provenance, task bindings, checkpoints, and unresolved history. Recover surviving sandbox files into immutable task-scoped attachments after migration, and let a new authorized user turn use those files without replaying the old execution. **Steps to reproduce** Create a terminal native run with a prepared copyback descriptor and a retained sandbox. Migrate its local workspace, remove the old host filesystem, recover the sandbox, and submit a new user turn. Also exercise a child task reusing its ancestor's workspace and an expired observed coordinator. ## What Changed - Journal exact legacy run, task, descriptor, and unfinished coordinator bindings during offline migration. Descriptor-free runs retain the complete declared lease/allocation/directory identities; publication locks and revalidates them. - Publish a run-scoped fence that prevents automatic restart or copyback into the retired local authority. - Release only the exact journaled expired coordinator lease under the offline deployment fence and locked revalidation, recording the prior owner in recovery history. Preserve phase, checkpoint, result, and run status. - Save recovered sandbox files through the existing immutable snapshot owner, retaining allocation and provider-stop checks. - Attach recovered files using the current task's identity when a child reuses an ancestor workspace. - Accept matching migration and snapshot receipts for workspace-repair admission and explicit new-user continuation. Changed coordinator bindings and invalid receipts remain blocked. - Require the finalizer's successful workspace_finalize operation before moving a workspace whose accepted result has finalized copyback; the existing owner can then finish assessment/status effects. - Document unresolved workspace-policy, task-ownership, and missing-provenance cases as separate owner actions that block detachment. ## Verification - 569 tests across migration, retained recovery, and explicit continuation passed onffcde13f57. - The final saved-result follow-up passes all 75 migration tests. Four receipt cases cover prepared copyback, a successful finalization receipt, a missing receipt, and a failed receipt. The successful case invokes the actual finalizer after migration and completes the existing saved operation. - Five real-file journeys cover a retained host, removed host, child-task reuse, an expired observed coordinator, and a descriptor-free run with two separate sandbox allocations. They recover sandbox files, clear the repair blocker, admit an explicit /new user comment, reserve a new remote workspace, and verify recovered attachment contents. - Negative cases reject changed coordinator phase/owner, invalid migration fences, changed source sets and allocations, mismatched task/allocation/action receipts, malformed archives, and unfinished snapshot operations. Descriptor-free continuation requires every journaled source; one missing archive keeps the hold. - Twelve PostgreSQL/S3 and maintenance-service tests pass on the exact final rebuilt runtime, including a plain-Node bundled process, authenticated original-key export, overlapping task-folder restore, and complete predecessor filesystem removal. The final server and runtime bundle builds also pass. - Head9ab6a38fa4passed all 573 migration/recovery/continuation tests, server typecheck, server build, runtime bundle, and 12 PostgreSQL/S3/bundled-maintenance tests. - Verification caught an expired-owner continuation rejection (160 recovery cases passed, one failed); the admission fix and subsequent 569-test run passed. Earlier fixture failures and the parent PR's unrelated full-suite failure remain documented in their verification history. - Greptile and Paperclip Review both scored final headfa58d4a5ce5/5, with no remaining findings. - CI exposed 15 blocker-read projection regressions: descriptor-free checks performed an extra full run read for ordinary unmigrated tasks. The follow-up gates the lookup on the migration marker, includes that marker in the bounded blocker projection, and selects only recovery fields. All 31 blocker-projection and real-file journey cases plus server typecheck pass. Final-head CI passed all 52 successful checks, including repository typecheck, build, server tests, and browser suites. - Preserved CI failure: the unchanged Runner package had one Cursor Cloud Stop/checkpoint timing failure (4,903 cases passed). All four focused local cases passed. Retrying only the failed job and its verification gate passed without source changes. - This PR does not claim tenant enrollment or a hosted rolling-deploy result. ## Risks Cloud must stop every predecessor deployment before enabling the offline migration contract. Migration preserves uncertain work; it does not certify provider termination, invent a completed result, or authorize sandbox deletion. Missing or ambiguous sources remain held. Preserved attachments are separate files rather than an automatic merge into newer task edits. Parallel shared-write policy, stale task ownership, missing task provenance, and accepted results without a successful finalization receipt remain enrollment blockers. Original folders and the original volume remain retained. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools, and code execution. This session does not expose a more specific serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used with available version and capability details - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have described the issue in-PR using the bug template - [x] I have not referenced internal or instance-local issues or links - [x] My branch name describes the change and contains no internal task ID - [x] I have run the focused tests locally and they pass - [x] I have added tests for the changed behavior - [x] I have updated relevant documentation - [x] I have considered and documented risks - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address reviewer findings before merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1268 lines
50 KiB
TypeScript
1268 lines
50 KiB
TypeScript
import { hasMigratedLegacyWorkspaceSource } from "../legacy-workspace-recovery-migration.js";
|
|
import { notifyCancelledWorkspacePreservation } from "../cancelled-workspace-preservation-notifications.js";
|
|
import { notifyEnvironmentCleanup } from "../environment-cleanup-notifications.js";
|
|
import { measureRunPhase } from "../execution-preparation-timing.js";
|
|
import { emitStartupReuse, type StartupReuseReason } from "@paperclipai/adapter-utils/startup-reuse";
|
|
import type { AdapterRuntimeEvent } from "@paperclipai/adapter-utils";
|
|
import type { WorkspaceCheckpointMetrics } from "@paperclipai/adapter-utils/workspace-checkpoint";
|
|
import type { RuntimeSpanRunner } from "@paperclipai/adapter-utils/acpx-engine/startup-timing";
|
|
import fs from "node:fs/promises";
|
|
import { createReadStream } from "node:fs";
|
|
import path from "node:path";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { eq } from "drizzle-orm";
|
|
import type { Db } from "@paperclipai/db";
|
|
import { environmentLeases, heartbeatRuns } from "@paperclipai/db";
|
|
import type { EnvironmentLease } from "@paperclipai/shared";
|
|
import {
|
|
prepareAdapterExecutionTargetRuntime,
|
|
type AdapterExecutionTarget,
|
|
type PreparedAdapterExecutionTargetRuntime,
|
|
} from "@paperclipai/adapter-utils/execution-target";
|
|
import { disposeGitWorkspaceSnapshot, type GitWorkspaceSnapshot } from "@paperclipai/adapter-utils/git-workspace-sync";
|
|
import {
|
|
directorySnapshotSha256,
|
|
disposeDirectorySnapshot,
|
|
openDirectorySnapshot,
|
|
parseDirectorySnapshot,
|
|
serializeDirectorySnapshot,
|
|
type DirectorySnapshot,
|
|
type SerializedDirectorySnapshot,
|
|
} from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
import type {
|
|
WorkspaceDurableSeedPaths,
|
|
WorkspaceInboundMode,
|
|
} from "@paperclipai/adapter-utils/sandbox-managed-runtime";
|
|
import { assertWorkspaceManifestDiskSpace, isPathManifest, manifestFileSha256, readManifestRecords, type PathManifest, type WorkspacePaths } from "@paperclipai/adapter-utils/workspace-manifest";
|
|
import { resolvePaperclipInstanceRoot } from "../../home-paths.js";
|
|
import { parseObject } from "../../adapters/utils.js";
|
|
import type { NativeRestartRecoveryClaim } from "./native-restart-recovery.js";
|
|
|
|
const LEGACY_DESCRIPTOR_SCHEMA = "paperclip.native-workspace-sync/v1";
|
|
const PREVIOUS_DESCRIPTOR_SCHEMA = "paperclip.native-workspace-sync/v2";
|
|
const DESCRIPTOR_SCHEMA = "paperclip.native-workspace-sync/v3";
|
|
type DescriptorSchema = typeof DESCRIPTOR_SCHEMA | typeof PREVIOUS_DESCRIPTOR_SCHEMA | typeof LEGACY_DESCRIPTOR_SCHEMA;
|
|
const STAMP_SCHEMA = "paperclip.native-workspace-stamp/v1";
|
|
const STATE_ROOT_NAME = "native-workspace-sync";
|
|
const DESCRIPTOR_NAME = "descriptor";
|
|
const WORKSPACE_SEED_NAME = "workspace-seed.tar";
|
|
const GIT_SEED_NAME = "git-seed.tar";
|
|
const REMOTE_STAMP_NAME = "workspace-sync-v1.json";
|
|
const MAX_DESCRIPTOR_BYTES = 64 * 1024 * 1024;
|
|
const SAFE_SEGMENT_RE = /^[A-Za-z0-9_-]+$/;
|
|
|
|
type NativeWorkspaceSyncState = "prepared" | "finalized";
|
|
export type NativeWorkspaceResourceDisposition =
|
|
"keep_running" | "stop_and_retain" | "destroy";
|
|
|
|
interface NativeWorkspaceSyncDescriptor {
|
|
schema: DescriptorSchema;
|
|
binding: {
|
|
runId: string;
|
|
companyId: string;
|
|
workspaceId: string;
|
|
leaseId: string;
|
|
providerLeaseId: string;
|
|
localCwd: string;
|
|
remoteCwd: string;
|
|
};
|
|
state: NativeWorkspaceSyncState;
|
|
baselineSha256: string;
|
|
baseline: SerializedDirectorySnapshot;
|
|
gitSnapshot: GitWorkspaceSnapshot | null;
|
|
repositories?: NonNullable<GitWorkspaceSnapshot["repositories"]>;
|
|
seed: {
|
|
workspaceArchiveSha256: string;
|
|
gitArchiveSha256: string | null;
|
|
} | null;
|
|
createdAt: string;
|
|
finalizedAt: string | null;
|
|
finalHostSha256: string | null;
|
|
resourceDisposition: NativeWorkspaceResourceDisposition | null;
|
|
}
|
|
|
|
export interface NativeWorkspaceSyncReference {
|
|
schema: DescriptorSchema;
|
|
state: NativeWorkspaceSyncState;
|
|
descriptorSha256: string;
|
|
baselineSha256: string;
|
|
finalHostSha256: string | null;
|
|
workspaceId: string;
|
|
leaseId: string;
|
|
providerLeaseId: string;
|
|
remoteCwd: string;
|
|
resourceDisposition: NativeWorkspaceResourceDisposition | null;
|
|
}
|
|
|
|
export interface PreparedNativeWorkspaceSync {
|
|
mode: WorkspaceInboundMode;
|
|
reference: NativeWorkspaceSyncReference;
|
|
restoreWorkspace(assertOwnership?: () => Promise<void>): Promise<void>;
|
|
cleanup(): Promise<void>;
|
|
}
|
|
|
|
export type NativeWorkspaceInboundEvidence =
|
|
| {
|
|
kind: "existing_run";
|
|
restartRecovery: boolean;
|
|
sameRunRecovery?: boolean;
|
|
sameProviderLease: boolean;
|
|
}
|
|
| {
|
|
kind: "new_run";
|
|
acquisition: "created" | "resumed" | "replacement" | null;
|
|
hasPriorStamp: boolean;
|
|
};
|
|
|
|
export function classifyNativeWorkspaceInbound(
|
|
evidence: NativeWorkspaceInboundEvidence,
|
|
): WorkspaceInboundMode {
|
|
if (evidence.kind === "existing_run") {
|
|
if (!evidence.restartRecovery && !evidence.sameRunRecovery) {
|
|
throw new Error("native_workspace_sync_unexpected_existing_descriptor");
|
|
}
|
|
return evidence.sameProviderLease ? "adopt_remote" : "durable_seed";
|
|
}
|
|
return evidence.acquisition === "resumed" && evidence.hasPriorStamp
|
|
? "adopt_remote"
|
|
: "host_current";
|
|
}
|
|
|
|
function canonicalJson(value: unknown): string {
|
|
if (Array.isArray(value)) {
|
|
return `[${value.map((entry) => canonicalJson(entry)).join(",")}]`;
|
|
}
|
|
if (value && typeof value === "object") {
|
|
const record = value as Record<string, unknown>;
|
|
return `{${Object.keys(record)
|
|
.sort()
|
|
.map((key) => `${JSON.stringify(key)}:${canonicalJson(record[key])}`)
|
|
.join(",")}}`;
|
|
}
|
|
return JSON.stringify(value);
|
|
}
|
|
|
|
function sha256(value: unknown): string {
|
|
return createHash("sha256").update(canonicalJson(value)).digest("hex");
|
|
}
|
|
|
|
function requireSafeSegment(value: string, label: string): string {
|
|
if (!SAFE_SEGMENT_RE.test(value)) {
|
|
throw new Error(`native_workspace_sync_invalid_${label}`);
|
|
}
|
|
return value;
|
|
}
|
|
|
|
export function nativeWorkspaceSyncStateRoot(): string {
|
|
return path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME);
|
|
}
|
|
|
|
function descriptorDirectory(runId: string): string {
|
|
return path.join(nativeWorkspaceSyncStateRoot(), requireSafeSegment(runId, "run_id"));
|
|
}
|
|
|
|
function descriptorPath(runId: string, descriptorSha256: string): string {
|
|
if (!/^[0-9a-f]{64}$/.test(descriptorSha256)) {
|
|
throw new Error("native_workspace_sync_descriptor_digest_invalid");
|
|
}
|
|
return path.join(
|
|
descriptorDirectory(runId),
|
|
`${DESCRIPTOR_NAME}.${descriptorSha256}.json`,
|
|
);
|
|
}
|
|
|
|
function legacyDescriptorPath(runId: string): string {
|
|
return path.join(descriptorDirectory(runId), `${DESCRIPTOR_NAME}.json`);
|
|
}
|
|
|
|
function durableSeedPaths(runId: string): {
|
|
workspaceArchivePath: string;
|
|
gitArchivePath: string;
|
|
} {
|
|
const directory = descriptorDirectory(runId);
|
|
return {
|
|
workspaceArchivePath: path.join(directory, WORKSPACE_SEED_NAME),
|
|
gitArchivePath: path.join(directory, GIT_SEED_NAME),
|
|
};
|
|
}
|
|
|
|
async function sha256File(filePath: string): Promise<string> {
|
|
return measureRunPhase("workspace_sync.archive_hash", () => new Promise((resolveDigest, rejectDigest) => {
|
|
const digest = createHash("sha256");
|
|
const stream = createReadStream(filePath);
|
|
stream.on("data", (chunk) => digest.update(chunk));
|
|
stream.on("error", rejectDigest);
|
|
stream.on("end", () => resolveDigest(digest.digest("hex")));
|
|
}));
|
|
}
|
|
|
|
async function verifiedDurableSeed(input: {
|
|
runId: string;
|
|
seed: NonNullable<NativeWorkspaceSyncDescriptor["seed"]>;
|
|
gitSnapshot: GitWorkspaceSnapshot | null;
|
|
repositories?: NonNullable<GitWorkspaceSnapshot["repositories"]>;
|
|
}): Promise<WorkspaceDurableSeedPaths> {
|
|
try {
|
|
const paths = durableSeedPaths(input.runId);
|
|
const workspaceStat = await fs.lstat(paths.workspaceArchivePath);
|
|
if (workspaceStat.isSymbolicLink() || !workspaceStat.isFile()) {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
if (
|
|
(await sha256File(paths.workspaceArchivePath)) !==
|
|
input.seed.workspaceArchiveSha256
|
|
) {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
if (input.gitSnapshot || input.repositories?.length) {
|
|
const gitStat = await fs.lstat(paths.gitArchivePath);
|
|
if (
|
|
gitStat.isSymbolicLink() ||
|
|
!gitStat.isFile() ||
|
|
!input.seed.gitArchiveSha256 ||
|
|
(await sha256File(paths.gitArchivePath)) !== input.seed.gitArchiveSha256
|
|
) {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
} else if (input.seed.gitArchiveSha256 !== null) {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
return {
|
|
workspaceArchivePath: paths.workspaceArchivePath,
|
|
workspaceArchiveSha256: input.seed.workspaceArchiveSha256,
|
|
gitArchivePath: input.gitSnapshot || input.repositories?.length ? paths.gitArchivePath : null,
|
|
gitArchiveSha256: input.seed.gitArchiveSha256,
|
|
};
|
|
} catch (error) {
|
|
if (
|
|
error instanceof Error &&
|
|
error.message === "workspace_sync_out_unrecoverable"
|
|
) {
|
|
throw error;
|
|
}
|
|
throw new Error("workspace_sync_out_unrecoverable", { cause: error });
|
|
}
|
|
}
|
|
|
|
async function ensurePrivateDirectory(dir: string): Promise<void> {
|
|
await fs.mkdir(dir, { recursive: true, mode: 0o700 });
|
|
const stat = await fs.lstat(dir);
|
|
if (stat.isSymbolicLink() || !stat.isDirectory()) {
|
|
throw new Error("native_workspace_sync_state_root_invalid");
|
|
}
|
|
}
|
|
|
|
const runDirectoryListeners = new Set<() => void>();
|
|
|
|
export function onNativeWorkspaceSyncRunDirectory(listener: () => void): () => void {
|
|
runDirectoryListeners.add(listener);
|
|
return () => {
|
|
runDirectoryListeners.delete(listener);
|
|
};
|
|
}
|
|
|
|
async function ensureRunDirectory(runId: string): Promise<string> {
|
|
const dir = descriptorDirectory(runId);
|
|
await ensurePrivateDirectory(dir);
|
|
for (const listener of runDirectoryListeners) listener();
|
|
return dir;
|
|
}
|
|
|
|
async function writeDescriptor(
|
|
descriptor: NativeWorkspaceSyncDescriptor,
|
|
): Promise<NativeWorkspaceSyncReference> {
|
|
const dir = await ensureRunDirectory(descriptor.binding.runId);
|
|
const body = `${canonicalJson(descriptor)}\n`;
|
|
const descriptorSha256 = createHash("sha256").update(body).digest("hex");
|
|
const finalPath = descriptorPath(descriptor.binding.runId, descriptorSha256);
|
|
const existingBody = await fs.readFile(finalPath, "utf8").catch(() => null);
|
|
if (existingBody !== null && existingBody !== body) {
|
|
throw new Error("native_workspace_sync_descriptor_digest_collision");
|
|
}
|
|
const tempPath = path.join(dir, `${DESCRIPTOR_NAME}.${randomUUID()}.tmp`);
|
|
try {
|
|
if (existingBody === null) {
|
|
await fs.writeFile(tempPath, body, { encoding: "utf8", mode: 0o600 });
|
|
await fs.rename(tempPath, finalPath);
|
|
}
|
|
} finally {
|
|
await fs.rm(tempPath, { force: true }).catch(() => undefined);
|
|
}
|
|
return {
|
|
schema: descriptor.schema,
|
|
state: descriptor.state,
|
|
descriptorSha256,
|
|
baselineSha256: descriptor.baselineSha256,
|
|
finalHostSha256: descriptor.finalHostSha256,
|
|
workspaceId: descriptor.binding.workspaceId,
|
|
leaseId: descriptor.binding.leaseId,
|
|
providerLeaseId: descriptor.binding.providerLeaseId,
|
|
remoteCwd: descriptor.binding.remoteCwd,
|
|
resourceDisposition: descriptor.resourceDisposition,
|
|
};
|
|
}
|
|
|
|
export function readNativeWorkspaceSyncReference(
|
|
value: unknown,
|
|
): NativeWorkspaceSyncReference | null {
|
|
const candidate = parseObject(value);
|
|
if (
|
|
(candidate.schema !== DESCRIPTOR_SCHEMA && candidate.schema !== PREVIOUS_DESCRIPTOR_SCHEMA && candidate.schema !== LEGACY_DESCRIPTOR_SCHEMA) ||
|
|
(candidate.state !== "prepared" && candidate.state !== "finalized") ||
|
|
typeof candidate.descriptorSha256 !== "string" ||
|
|
!/^[0-9a-f]{64}$/.test(candidate.descriptorSha256) ||
|
|
typeof candidate.baselineSha256 !== "string" ||
|
|
!/^[0-9a-f]{64}$/.test(candidate.baselineSha256) ||
|
|
(candidate.finalHostSha256 !== null &&
|
|
candidate.finalHostSha256 !== undefined &&
|
|
(typeof candidate.finalHostSha256 !== "string" ||
|
|
!/^[0-9a-f]{64}$/.test(candidate.finalHostSha256))) ||
|
|
typeof candidate.workspaceId !== "string" ||
|
|
typeof candidate.leaseId !== "string" ||
|
|
typeof candidate.providerLeaseId !== "string" ||
|
|
typeof candidate.remoteCwd !== "string" ||
|
|
(candidate.resourceDisposition !== null &&
|
|
candidate.resourceDisposition !== undefined &&
|
|
candidate.resourceDisposition !== "keep_running" &&
|
|
candidate.resourceDisposition !== "stop_and_retain" &&
|
|
candidate.resourceDisposition !== "destroy")
|
|
) {
|
|
return null;
|
|
}
|
|
return {
|
|
schema: candidate.schema,
|
|
state: candidate.state,
|
|
descriptorSha256: candidate.descriptorSha256,
|
|
baselineSha256: candidate.baselineSha256,
|
|
finalHostSha256:
|
|
typeof candidate.finalHostSha256 === "string"
|
|
? candidate.finalHostSha256
|
|
: null,
|
|
workspaceId: candidate.workspaceId,
|
|
leaseId: candidate.leaseId,
|
|
providerLeaseId: candidate.providerLeaseId,
|
|
remoteCwd: candidate.remoteCwd,
|
|
resourceDisposition:
|
|
candidate.resourceDisposition === "keep_running" ||
|
|
candidate.resourceDisposition === "stop_and_retain" ||
|
|
candidate.resourceDisposition === "destroy"
|
|
? candidate.resourceDisposition
|
|
: null,
|
|
};
|
|
}
|
|
|
|
function validManifestShape(value: unknown): value is PathManifest {
|
|
const record = parseObject(value);
|
|
return record.kind === "path_manifest" && record.version === 1 && typeof record.filePath === "string"
|
|
&& typeof record.category === "string" && ["overlay", "deleted", "ignored", "baseline"].includes(record.category)
|
|
&& typeof record.count === "number" && Number.isSafeInteger(record.count) && record.count >= 0;
|
|
}
|
|
|
|
function* gitManifests(snapshot: GitWorkspaceSnapshot | null): Generator<PathManifest> {
|
|
if (!snapshot) return;
|
|
for (const paths of [snapshot.overlayPaths, snapshot.deletedPaths, snapshot.ignoredPaths]) if (isPathManifest(paths)) yield paths;
|
|
for (const repo of snapshot.repositories ?? []) yield* gitManifests(repo.snapshot);
|
|
}
|
|
|
|
async function verifyManifest(runId: string, manifest: PathManifest): Promise<void> {
|
|
const directory = descriptorDirectory(runId);
|
|
const name = path.basename(manifest.filePath);
|
|
const match = /^manifest\.([0-9a-f]{64})\.sqlite$/.exec(name);
|
|
if (!match || manifest.filePath !== path.join(directory, name)) throw new Error("native_workspace_sync_manifest_path_invalid");
|
|
for (const dir of [path.dirname(directory), directory]) {
|
|
const stat = await fs.lstat(dir);
|
|
if (!stat.isDirectory() || stat.isSymbolicLink()) throw new Error("native_workspace_sync_manifest_root_invalid");
|
|
}
|
|
if (await manifestFileSha256(manifest.filePath) !== match[1]) throw new Error("native_workspace_sync_manifest_digest_mismatch");
|
|
// Verify every record before consumers can mutate a workspace on recovery.
|
|
for (const _record of readManifestRecords(manifest)) { /* bounded validation */ }
|
|
}
|
|
|
|
async function persistSnapshotManifests(runId: string, snapshot: NonNullable<PreparedAdapterExecutionTargetRuntime["workspaceSyncSnapshot"]>): Promise<void> {
|
|
const baseline = serializeDirectorySnapshot(snapshot.baseline);
|
|
const manifests = function* () {
|
|
if (baseline.version === 2) {
|
|
yield baseline.entries;
|
|
if (baseline.ignoredPaths && isPathManifest(baseline.ignoredPaths)) yield baseline.ignoredPaths;
|
|
}
|
|
yield* gitManifests(snapshot.gitSnapshot);
|
|
for (const repo of snapshot.repositories ?? []) yield* gitManifests(repo.snapshot);
|
|
};
|
|
const copies = new Map<string, string>();
|
|
for (const manifest of manifests()) {
|
|
const oldPath = manifest.filePath;
|
|
const previous = copies.get(oldPath);
|
|
if (previous) { manifest.filePath = previous; continue; }
|
|
// A shared reference may already have been updated by the baseline.
|
|
if (path.dirname(oldPath) === descriptorDirectory(runId)) continue;
|
|
const temporary = path.join(descriptorDirectory(runId), `manifest.${randomUUID()}.tmp`);
|
|
try {
|
|
const sourceStat = await fs.lstat(oldPath);
|
|
if (!sourceStat.isFile() || sourceStat.isSymbolicLink()) throw new Error("native_workspace_sync_manifest_invalid");
|
|
assertWorkspaceManifestDiskSpace(descriptorDirectory(runId), sourceStat.size);
|
|
await fs.copyFile(oldPath, temporary);
|
|
await fs.chmod(temporary, 0o600);
|
|
const digest = await manifestFileSha256(temporary);
|
|
const target = path.join(descriptorDirectory(runId), `manifest.${digest}.sqlite`);
|
|
await fs.rename(temporary, target);
|
|
copies.set(oldPath, target);
|
|
manifest.filePath = target;
|
|
} finally { await fs.rm(temporary, { force: true }); }
|
|
}
|
|
// Ownership tracks original scratch directories, never the new durable path.
|
|
await disposeDirectorySnapshot(snapshot.baseline);
|
|
await disposeGitWorkspaceSnapshot(snapshot.gitSnapshot);
|
|
for (const repo of snapshot.repositories ?? []) await disposeGitWorkspaceSnapshot(repo.snapshot);
|
|
}
|
|
|
|
function parseGitSnapshot(
|
|
value: unknown,
|
|
nested = false,
|
|
): GitWorkspaceSnapshot | null | undefined {
|
|
if (value === null) return null;
|
|
const candidate = parseObject(value);
|
|
const paths = [candidate.overlayPaths, candidate.deletedPaths, candidate.ignoredPaths];
|
|
if ((candidate.originUrl !== undefined && candidate.originUrl !== null && typeof candidate.originUrl !== "string") || typeof candidate.headCommit !== "string" || (candidate.branchName !== null && typeof candidate.branchName !== "string")
|
|
|| !paths.every((entries) => validManifestShape(entries) || (Array.isArray(entries) && entries.every((entry) =>
|
|
typeof entry === "string" && entry.length > 0 && !entry.includes("\0") && !path.posix.isAbsolute(entry) && !path.win32.isAbsolute(entry)
|
|
&& !entry.split(/[\\/]/).some((segment) => segment === ".." || segment === "."))))) return undefined;
|
|
const repositories: NonNullable<GitWorkspaceSnapshot["repositories"]> = [];
|
|
if (candidate.repositories !== undefined) {
|
|
if (nested || !Array.isArray(candidate.repositories)) return undefined;
|
|
const seen = new Set<string>();
|
|
for (const raw of candidate.repositories) {
|
|
const repo = parseObject(raw);
|
|
if (typeof repo.path !== "string" || !/^\.paperclip-repositories\/[a-zA-Z0-9_-]+$/.test(repo.path) || seen.has(repo.path)) return undefined;
|
|
const snapshot = parseGitSnapshot(repo.snapshot, true);
|
|
if (!snapshot) return undefined;
|
|
seen.add(repo.path);
|
|
repositories.push({ path: repo.path, snapshot });
|
|
}
|
|
}
|
|
return {
|
|
headCommit: candidate.headCommit,
|
|
branchName: candidate.branchName as string | null,
|
|
...(candidate.originUrl !== undefined ? { originUrl: candidate.originUrl as string | null } : {}),
|
|
overlayPaths: candidate.overlayPaths as WorkspacePaths,
|
|
deletedPaths: candidate.deletedPaths as WorkspacePaths,
|
|
ignoredPaths: candidate.ignoredPaths as WorkspacePaths,
|
|
...(repositories.length ? { repositories } : {}),
|
|
};
|
|
}
|
|
|
|
async function readDescriptor(input: {
|
|
runId: string;
|
|
reference: NativeWorkspaceSyncReference;
|
|
}): Promise<{
|
|
descriptor: NativeWorkspaceSyncDescriptor;
|
|
baseline: DirectorySnapshot;
|
|
}> {
|
|
const contentAddressedPath = descriptorPath(
|
|
input.runId,
|
|
input.reference.descriptorSha256,
|
|
);
|
|
// Early v1 writers used descriptor.json. Prefer the immutable,
|
|
// content-addressed name, but keep the digest-verified legacy path readable
|
|
// so an interrupted upgrade can still finish its already-proposed result.
|
|
const filePath = await fs
|
|
.lstat(contentAddressedPath)
|
|
.then(() => contentAddressedPath)
|
|
.catch(async (error: NodeJS.ErrnoException) => {
|
|
if (error.code !== "ENOENT") throw error;
|
|
const legacyPath = legacyDescriptorPath(input.runId);
|
|
await fs.lstat(legacyPath);
|
|
return legacyPath;
|
|
});
|
|
const stat = await fs.lstat(filePath);
|
|
if (
|
|
stat.isSymbolicLink() ||
|
|
!stat.isFile() ||
|
|
stat.size > MAX_DESCRIPTOR_BYTES
|
|
) {
|
|
throw new Error("native_workspace_sync_descriptor_invalid");
|
|
}
|
|
const body = await fs.readFile(filePath, "utf8");
|
|
if (
|
|
createHash("sha256").update(body).digest("hex") !==
|
|
input.reference.descriptorSha256
|
|
) {
|
|
throw new Error("native_workspace_sync_descriptor_digest_mismatch");
|
|
}
|
|
const parsed = JSON.parse(body) as unknown;
|
|
const candidate = parseObject(parsed);
|
|
const binding = parseObject(candidate.binding);
|
|
const rawBaseline = parseObject(candidate.baseline);
|
|
let baseline: DirectorySnapshot | null;
|
|
if ((candidate.schema === DESCRIPTOR_SCHEMA || candidate.schema === PREVIOUS_DESCRIPTOR_SCHEMA) && rawBaseline.version === 2
|
|
&& Array.isArray(rawBaseline.exclude) && rawBaseline.exclude.every((entry) => typeof entry === "string")
|
|
&& validManifestShape(rawBaseline.entries) && rawBaseline.entries.category === "baseline"
|
|
&& (rawBaseline.ignoredPaths === undefined || validManifestShape(rawBaseline.ignoredPaths))) {
|
|
await verifyManifest(input.runId, rawBaseline.entries);
|
|
if (validManifestShape(rawBaseline.ignoredPaths)) await verifyManifest(input.runId, rawBaseline.ignoredPaths);
|
|
baseline = openDirectorySnapshot(rawBaseline as unknown as Extract<SerializedDirectorySnapshot, { version: 2 }>);
|
|
} else {
|
|
baseline = parseDirectorySnapshot(candidate.baseline);
|
|
}
|
|
const gitSnapshot = parseGitSnapshot(candidate.gitSnapshot);
|
|
const repositoryContainer = candidate.repositories === undefined ? null : parseGitSnapshot({
|
|
headCommit: "", branchName: null, overlayPaths: [], deletedPaths: [], ignoredPaths: [], repositories: candidate.repositories,
|
|
});
|
|
if (candidate.repositories !== undefined && (!repositoryContainer || candidate.schema !== DESCRIPTOR_SCHEMA || gitSnapshot)) throw new Error("native_workspace_sync_repositories_invalid");
|
|
const repositories = repositoryContainer?.repositories;
|
|
for (const repo of repositories ?? []) for (const manifest of gitManifests(repo.snapshot)) await verifyManifest(input.runId, manifest);
|
|
for (const manifest of gitManifests(gitSnapshot ?? null)) await verifyManifest(input.runId, manifest);
|
|
const rawSeed =
|
|
candidate.seed === null || candidate.seed === undefined
|
|
? null
|
|
: parseObject(candidate.seed);
|
|
const seed =
|
|
rawSeed === null
|
|
? null
|
|
: typeof rawSeed.workspaceArchiveSha256 === "string" &&
|
|
/^[0-9a-f]{64}$/.test(rawSeed.workspaceArchiveSha256) &&
|
|
(rawSeed.gitArchiveSha256 === null ||
|
|
(typeof rawSeed.gitArchiveSha256 === "string" &&
|
|
/^[0-9a-f]{64}$/.test(rawSeed.gitArchiveSha256)))
|
|
? {
|
|
workspaceArchiveSha256: rawSeed.workspaceArchiveSha256,
|
|
gitArchiveSha256: rawSeed.gitArchiveSha256 as string | null,
|
|
}
|
|
: undefined;
|
|
const finalHostSha256 =
|
|
typeof candidate.finalHostSha256 === "string" &&
|
|
/^[0-9a-f]{64}$/.test(candidate.finalHostSha256)
|
|
? candidate.finalHostSha256
|
|
: candidate.finalHostSha256 === null
|
|
? null
|
|
: undefined;
|
|
const resourceDisposition =
|
|
candidate.resourceDisposition === "keep_running" ||
|
|
candidate.resourceDisposition === "stop_and_retain" ||
|
|
candidate.resourceDisposition === "destroy"
|
|
? candidate.resourceDisposition
|
|
: candidate.resourceDisposition === null
|
|
? null
|
|
: undefined;
|
|
if (
|
|
(candidate.schema !== DESCRIPTOR_SCHEMA && candidate.schema !== PREVIOUS_DESCRIPTOR_SCHEMA && candidate.schema !== LEGACY_DESCRIPTOR_SCHEMA) ||
|
|
(candidate.state !== "prepared" && candidate.state !== "finalized") ||
|
|
!baseline ||
|
|
gitSnapshot === undefined ||
|
|
seed === undefined ||
|
|
binding.runId !== input.runId ||
|
|
binding.workspaceId !== input.reference.workspaceId ||
|
|
binding.leaseId !== input.reference.leaseId ||
|
|
binding.providerLeaseId !== input.reference.providerLeaseId ||
|
|
binding.remoteCwd !== input.reference.remoteCwd ||
|
|
typeof binding.companyId !== "string" ||
|
|
typeof binding.localCwd !== "string" ||
|
|
!path.isAbsolute(binding.localCwd) ||
|
|
typeof binding.remoteCwd !== "string" ||
|
|
!path.posix.isAbsolute(binding.remoteCwd) ||
|
|
typeof candidate.baselineSha256 !== "string" ||
|
|
candidate.baselineSha256 !== input.reference.baselineSha256 ||
|
|
directorySnapshotSha256(baseline) !== candidate.baselineSha256 ||
|
|
typeof candidate.createdAt !== "string" ||
|
|
!Number.isFinite(Date.parse(candidate.createdAt)) ||
|
|
candidate.schema !== input.reference.schema ||
|
|
candidate.state !== input.reference.state ||
|
|
finalHostSha256 === undefined ||
|
|
finalHostSha256 !== input.reference.finalHostSha256 ||
|
|
resourceDisposition === undefined ||
|
|
resourceDisposition !== input.reference.resourceDisposition ||
|
|
(candidate.state === "prepared" &&
|
|
(candidate.finalizedAt !== null || finalHostSha256 !== null)) ||
|
|
(candidate.state === "finalized" &&
|
|
(typeof candidate.finalizedAt !== "string" ||
|
|
!Number.isFinite(Date.parse(candidate.finalizedAt)) ||
|
|
finalHostSha256 === null))
|
|
) {
|
|
throw new Error("native_workspace_sync_descriptor_binding_mismatch");
|
|
}
|
|
return {
|
|
descriptor: {
|
|
schema: candidate.schema,
|
|
binding: {
|
|
runId: binding.runId as string,
|
|
companyId: binding.companyId,
|
|
workspaceId: binding.workspaceId as string,
|
|
leaseId: binding.leaseId as string,
|
|
providerLeaseId: binding.providerLeaseId as string,
|
|
localCwd: binding.localCwd,
|
|
remoteCwd: binding.remoteCwd as string,
|
|
},
|
|
state: candidate.state,
|
|
baselineSha256: candidate.baselineSha256,
|
|
baseline: candidate.baseline as SerializedDirectorySnapshot,
|
|
gitSnapshot,
|
|
...(repositories?.length ? { repositories } : {}),
|
|
seed,
|
|
createdAt: candidate.createdAt,
|
|
finalizedAt:
|
|
typeof candidate.finalizedAt === "string"
|
|
? candidate.finalizedAt
|
|
: null,
|
|
finalHostSha256,
|
|
resourceDisposition,
|
|
},
|
|
baseline,
|
|
};
|
|
}
|
|
|
|
async function persistRunReference(
|
|
db: Db,
|
|
runId: string,
|
|
reference: NativeWorkspaceSyncReference,
|
|
): Promise<void> {
|
|
await db.transaction(async (tx) => {
|
|
const run = await tx
|
|
.select({ companyId: heartbeatRuns.companyId, runnerProfileJson: heartbeatRuns.runnerProfileJson })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, runId))
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!run) throw new Error("native_workspace_sync_run_missing");
|
|
await notifyCancelledWorkspacePreservation(tx, { companyId: run.companyId, runId });
|
|
await tx
|
|
.update(heartbeatRuns)
|
|
.set({
|
|
runnerProfileJson: {
|
|
...parseObject(run.runnerProfileJson),
|
|
nativeWorkspaceSync: reference,
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(heartbeatRuns.id, runId));
|
|
});
|
|
}
|
|
|
|
async function persistLeaseStamp(input: {
|
|
db: Db;
|
|
leaseId: string;
|
|
stamp: Record<string, unknown>;
|
|
}): Promise<void> {
|
|
await input.db.transaction(async (tx) => {
|
|
const lease = await tx
|
|
.select({ metadata: environmentLeases.metadata, companyId: environmentLeases.companyId, heartbeatRunId: environmentLeases.heartbeatRunId })
|
|
.from(environmentLeases)
|
|
.where(eq(environmentLeases.id, input.leaseId))
|
|
.for("update")
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
if (!lease) throw new Error("native_workspace_sync_lease_missing");
|
|
await notifyEnvironmentCleanup(tx, { companyId: lease.companyId, runId: lease.heartbeatRunId ?? undefined });
|
|
await tx
|
|
.update(environmentLeases)
|
|
.set({
|
|
metadata: {
|
|
...parseObject(lease.metadata),
|
|
nativeWorkspaceSync: input.stamp,
|
|
},
|
|
updatedAt: new Date(),
|
|
})
|
|
.where(eq(environmentLeases.id, input.leaseId));
|
|
});
|
|
}
|
|
|
|
function providerLeaseIdFor(input: {
|
|
target: Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
|
lease: EnvironmentLease;
|
|
}): string {
|
|
const providerLeaseId =
|
|
input.target.sandboxLeaseAcquisition?.providerLeaseId ??
|
|
input.lease.providerLeaseId;
|
|
if (!providerLeaseId) {
|
|
throw new Error("native_workspace_sync_provider_lease_missing");
|
|
}
|
|
return providerLeaseId;
|
|
}
|
|
|
|
function shellQuote(value: string): string {
|
|
return `'${value.replaceAll("'", "'\\''")}'`;
|
|
}
|
|
|
|
async function writeRemoteStamp(input: {
|
|
target: Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
|
stamp: Record<string, unknown>;
|
|
}): Promise<void> {
|
|
if (!input.target.runner) {
|
|
throw new Error("native_workspace_sync_runner_missing");
|
|
}
|
|
const runtimeDir = path.posix.join(
|
|
input.target.remoteCwd,
|
|
".paperclip-runtime",
|
|
"paperclip-runner",
|
|
);
|
|
const stampPath = path.posix.join(runtimeDir, REMOTE_STAMP_NAME);
|
|
const tempPath = `${stampPath}.tmp`;
|
|
const encoded = Buffer.from(
|
|
`${canonicalJson(input.stamp)}\n`,
|
|
"utf8",
|
|
).toString("base64");
|
|
const result = await input.target.runner.execute({
|
|
command: input.target.shellCommand ?? "sh",
|
|
args: [
|
|
"-c",
|
|
`umask 077; mkdir -p ${shellQuote(runtimeDir)} && printf %s ${shellQuote(encoded)} | base64 -d > ${shellQuote(tempPath)} && mv ${shellQuote(tempPath)} ${shellQuote(stampPath)}`,
|
|
],
|
|
cwd: "/",
|
|
timeoutMs: 15_000,
|
|
bypassSession: true,
|
|
});
|
|
if (result.timedOut || result.exitCode !== 0) {
|
|
throw new Error("native_workspace_sync_remote_stamp_failed");
|
|
}
|
|
}
|
|
|
|
async function remoteStampMatches(input: {
|
|
target: Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
|
expected: Record<string, unknown>;
|
|
gitSnapshot?: GitWorkspaceSnapshot | null;
|
|
repositories?: NonNullable<GitWorkspaceSnapshot["repositories"]>;
|
|
}): Promise<boolean> {
|
|
if (!input.target.runner) return false;
|
|
const stampPath = path.posix.join(
|
|
input.target.remoteCwd,
|
|
".paperclip-runtime",
|
|
"paperclip-runner",
|
|
REMOTE_STAMP_NAME,
|
|
);
|
|
// File hashes do not detect empty commits or branch changes. A fresh host
|
|
// snapshot may authorize replacement only if the retained sandbox actually
|
|
// starts from that same Git identity, including each managed repository.
|
|
const gitChecks: string[] = [];
|
|
const checkGit = (remoteDir: string, snapshot: GitWorkspaceSnapshot) => {
|
|
const git = `git -C ${shellQuote(remoteDir)}`;
|
|
gitChecks.push(`test "$(${git} rev-parse HEAD)" = ${shellQuote(snapshot.headCommit)}`);
|
|
gitChecks.push(snapshot.branchName === null
|
|
? `(${git} symbolic-ref --quiet HEAD >/dev/null 2>&1; test $? -eq 1)`
|
|
: `test "$(${git} symbolic-ref --quiet --short HEAD)" = ${shellQuote(snapshot.branchName)}`);
|
|
if (snapshot.originUrl !== undefined) gitChecks.push(snapshot.originUrl === null
|
|
? `! ${git} config --get remote.origin.url >/dev/null 2>&1`
|
|
: `test "$(${git} config --get remote.origin.url)" = ${shellQuote(snapshot.originUrl)}`);
|
|
for (const repository of snapshot.repositories ?? []) {
|
|
checkGit(path.posix.join(remoteDir, repository.path), repository.snapshot);
|
|
}
|
|
};
|
|
if (input.gitSnapshot) checkGit(input.target.remoteCwd, input.gitSnapshot);
|
|
for (const repo of input.repositories ?? []) checkGit(path.posix.join(input.target.remoteCwd, repo.path), repo.snapshot);
|
|
const result = await input.target.runner.execute({
|
|
command: input.target.shellCommand ?? "sh",
|
|
args: [
|
|
"-c",
|
|
[...gitChecks, `test -f ${shellQuote(stampPath)}`, `cat ${shellQuote(stampPath)}`].join(" && "),
|
|
],
|
|
cwd: "/",
|
|
timeoutMs: 15_000,
|
|
bypassSession: true,
|
|
});
|
|
if (result.timedOut || result.exitCode !== 0) return false;
|
|
try {
|
|
return (
|
|
canonicalJson(JSON.parse(result.stdout)) === canonicalJson(input.expected)
|
|
);
|
|
} catch {
|
|
return false;
|
|
}
|
|
}
|
|
|
|
function leaseStamp(input: {
|
|
lease: EnvironmentLease;
|
|
workspaceId: string;
|
|
providerLeaseId: string;
|
|
remoteCwd: string;
|
|
}): Record<string, unknown> | null {
|
|
const stamp = parseObject(
|
|
parseObject(input.lease.metadata).nativeWorkspaceSync,
|
|
);
|
|
if (
|
|
stamp.schema !== STAMP_SCHEMA ||
|
|
stamp.workspaceId !== input.workspaceId ||
|
|
stamp.providerLeaseId !== input.providerLeaseId ||
|
|
stamp.remoteCwd !== input.remoteCwd ||
|
|
typeof stamp.hostSha256 !== "string" ||
|
|
!/^[0-9a-f]{64}$/.test(stamp.hostSha256)
|
|
) {
|
|
return null;
|
|
}
|
|
return stamp;
|
|
}
|
|
|
|
function finalizedWorkspaceStamp(input: {
|
|
descriptor: NativeWorkspaceSyncDescriptor;
|
|
hostSha256: string;
|
|
}): Record<string, unknown> {
|
|
return {
|
|
schema: STAMP_SCHEMA,
|
|
workspaceId: input.descriptor.binding.workspaceId,
|
|
providerLeaseId: input.descriptor.binding.providerLeaseId,
|
|
remoteCwd: input.descriptor.binding.remoteCwd,
|
|
hostSha256: input.hostSha256,
|
|
finalizedRunId: input.descriptor.binding.runId,
|
|
};
|
|
}
|
|
|
|
async function prepareRuntime(input: {
|
|
runId: string;
|
|
target: Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
|
workspaceLocalDir: string;
|
|
mode: WorkspaceInboundMode;
|
|
baseline?: DirectorySnapshot;
|
|
gitSnapshot?: GitWorkspaceSnapshot | null;
|
|
repositories?: NonNullable<GitWorkspaceSnapshot["repositories"]>;
|
|
durableSeed?: WorkspaceDurableSeedPaths;
|
|
seedCacheDirectory?: string;
|
|
seedCacheCompanyDirectory?: string;
|
|
workspaceGitIndexRefresh?: boolean;
|
|
onPhase?: RuntimeSpanRunner;
|
|
onCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise<void>;
|
|
}): Promise<PreparedAdapterExecutionTargetRuntime> {
|
|
const phase = {
|
|
host_current: "workspace_sync.prepare_host_current",
|
|
durable_seed: "workspace_sync.prepare_durable_seed",
|
|
adopt_remote: "workspace_sync.prepare_adopt_remote",
|
|
} as const;
|
|
return measureRunPhase(phase[input.mode], () => prepareAdapterExecutionTargetRuntime({
|
|
runId: input.runId,
|
|
target: input.target,
|
|
adapterKey: "paperclip-runner",
|
|
workspaceLocalDir: input.workspaceLocalDir,
|
|
workspaceRemoteDir: input.target.remoteCwd,
|
|
workspaceInboundMode: input.mode,
|
|
workspaceSnapshotIsolation: "worker",
|
|
workspaceGitIndexRefresh: input.workspaceGitIndexRefresh,
|
|
workspaceDurableSeed: input.durableSeed,
|
|
workspaceBaseline: input.baseline,
|
|
workspaceGitSnapshot: input.gitSnapshot,
|
|
workspaceRepositories: input.repositories ?? (input.baseline ? [] : undefined),
|
|
workspaceCheckpoint: true,
|
|
onWorkspaceCheckpoint: input.onCheckpoint,
|
|
workspaceSeedCacheDirectory: input.seedCacheDirectory,
|
|
workspaceSeedCacheCompanyDirectory: input.seedCacheCompanyDirectory,
|
|
workspaceSeedCacheScope: "company",
|
|
runtimeSpan: input.onPhase,
|
|
}));
|
|
}
|
|
|
|
async function finalizePreparedRuntime(input: {
|
|
db: Db;
|
|
runId: string;
|
|
target: Extract<AdapterExecutionTarget, { transport: "sandbox" }>;
|
|
runtime: PreparedAdapterExecutionTargetRuntime;
|
|
descriptor: NativeWorkspaceSyncDescriptor;
|
|
assertOwnership?: () => Promise<void>;
|
|
onPhase?: RuntimeSpanRunner;
|
|
onCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise<void>;
|
|
}): Promise<NativeWorkspaceSyncReference> {
|
|
if (input.onPhase) await input.onPhase("workspace.restore", () => input.runtime.restoreWorkspace());
|
|
else await input.runtime.restoreWorkspace();
|
|
return measureRunPhase("execution.workspace_commit", async () => {
|
|
await input.assertOwnership?.();
|
|
const finalSnapshot =
|
|
await import("@paperclipai/adapter-utils/workspace-restore-merge").then(
|
|
({ captureDirectorySnapshotOffThread: captureDirectorySnapshot }) =>
|
|
captureDirectorySnapshot(input.descriptor.binding.localCwd, {
|
|
exclude: input.runtime.workspaceSyncSnapshot?.baseline.exclude ?? [],
|
|
ignoredPaths: input.runtime.workspaceSyncSnapshot?.baseline.ignoredPaths,
|
|
diskBacked: true,
|
|
}),
|
|
);
|
|
const finalHostSha256 = directorySnapshotSha256(finalSnapshot);
|
|
await disposeDirectorySnapshot(finalSnapshot);
|
|
const stamp = finalizedWorkspaceStamp({
|
|
descriptor: input.descriptor,
|
|
hostSha256: finalHostSha256,
|
|
});
|
|
await writeRemoteStamp({ target: input.target, stamp });
|
|
const finalizedDescriptor: NativeWorkspaceSyncDescriptor = {
|
|
...input.descriptor,
|
|
state: "finalized",
|
|
finalizedAt: new Date().toISOString(),
|
|
finalHostSha256,
|
|
};
|
|
await input.assertOwnership?.();
|
|
const reference = await writeDescriptor(finalizedDescriptor);
|
|
await persistRunReference(input.db, input.runId, reference);
|
|
await persistLeaseStamp({
|
|
db: input.db,
|
|
leaseId: input.descriptor.binding.leaseId,
|
|
stamp,
|
|
});
|
|
return reference;
|
|
});
|
|
}
|
|
|
|
export async function prepareNativeWorkspaceSync(input: {
|
|
db: Db;
|
|
runId: string;
|
|
companyId: string;
|
|
workspaceId: string;
|
|
workspaceLocalDir: string;
|
|
target: AdapterExecutionTarget | null;
|
|
lease: EnvironmentLease;
|
|
restartRecovery?: NativeRestartRecoveryClaim;
|
|
sameRunRecovery?: boolean;
|
|
resourceDisposition?: NativeWorkspaceResourceDisposition;
|
|
workspaceGitIndexRefresh?: boolean;
|
|
onPhase?: RuntimeSpanRunner;
|
|
onCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise<void>;
|
|
onEvent?: (event: AdapterRuntimeEvent) => Promise<void>;
|
|
}): Promise<PreparedNativeWorkspaceSync | null> {
|
|
if (input.target?.kind !== "remote" || input.target.transport !== "sandbox" || input.target.runnerTask) {
|
|
return null;
|
|
}
|
|
const target = input.target;
|
|
const providerLeaseId = providerLeaseIdFor({ target, lease: input.lease });
|
|
// Reattachment may never seed a replacement sandbox over a still-live run.
|
|
if (input.restartRecovery?.kind === "reattach_remote_runner" &&
|
|
(input.restartRecovery.remote.kind === "plugin_task" || input.restartRecovery.runId !== input.runId ||
|
|
input.restartRecovery.remote.providerLeaseId !== providerLeaseId ||
|
|
input.restartRecovery.remote.remoteCwd !== target.remoteCwd)) {
|
|
throw new Error("native_remote_recovery_lease_mismatch");
|
|
}
|
|
const run = await measureRunPhase("workspace_sync.run_read", () => input.db
|
|
.select({ companyId: heartbeatRuns.companyId, runnerProfileJson: heartbeatRuns.runnerProfileJson })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, input.runId))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null));
|
|
if (!run) throw new Error("native_workspace_sync_run_missing");
|
|
const existingReference = readNativeWorkspaceSyncReference(
|
|
parseObject(run.runnerProfileJson).nativeWorkspaceSync,
|
|
);
|
|
|
|
let runtime: PreparedAdapterExecutionTargetRuntime | undefined;
|
|
let descriptor: NativeWorkspaceSyncDescriptor;
|
|
let mode: WorkspaceInboundMode = "host_current";
|
|
let reuseReason: StartupReuseReason = "created";
|
|
const seedPaths = durableSeedPaths(input.runId);
|
|
await measureRunPhase("workspace_sync.descriptor_directory", () => ensureRunDirectory(input.runId));
|
|
|
|
try {
|
|
if (existingReference) {
|
|
const existing = await measureRunPhase("workspace_sync.descriptor_read", () => readDescriptor({
|
|
runId: input.runId,
|
|
reference: existingReference,
|
|
}));
|
|
if (
|
|
existing.descriptor.binding.companyId !== input.companyId ||
|
|
existing.descriptor.binding.workspaceId !== input.workspaceId ||
|
|
path.resolve(existing.descriptor.binding.localCwd) !==
|
|
path.resolve(input.workspaceLocalDir) ||
|
|
existing.descriptor.binding.remoteCwd !== target.remoteCwd
|
|
) {
|
|
throw new Error("native_workspace_sync_descriptor_binding_mismatch");
|
|
}
|
|
const sameProviderLease =
|
|
existing.descriptor.binding.providerLeaseId === providerLeaseId;
|
|
mode = classifyNativeWorkspaceInbound({
|
|
kind: "existing_run",
|
|
restartRecovery: Boolean(input.restartRecovery),
|
|
sameRunRecovery: input.sameRunRecovery,
|
|
sameProviderLease,
|
|
});
|
|
reuseReason = sameProviderLease ? "verified_reuse" : "replacement";
|
|
const durableSeed =
|
|
mode === "durable_seed"
|
|
? existing.descriptor.seed
|
|
? await measureRunPhase("workspace_sync.durable_seed_verify", () => verifiedDurableSeed({
|
|
runId: input.runId,
|
|
seed: existing.descriptor.seed!,
|
|
gitSnapshot: existing.descriptor.gitSnapshot,
|
|
repositories: existing.descriptor.repositories,
|
|
}))
|
|
: (() => {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
})()
|
|
: undefined;
|
|
runtime = await prepareRuntime({
|
|
runId: input.runId,
|
|
target,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
workspaceLocalDir: input.workspaceLocalDir,
|
|
mode,
|
|
baseline: existing.baseline,
|
|
gitSnapshot: existing.descriptor.gitSnapshot,
|
|
repositories: existing.descriptor.repositories,
|
|
durableSeed,
|
|
});
|
|
descriptor = {
|
|
...existing.descriptor,
|
|
binding: {
|
|
...existing.descriptor.binding,
|
|
leaseId: target.leaseId ?? input.lease.id,
|
|
providerLeaseId,
|
|
},
|
|
};
|
|
} else {
|
|
const acquisition = target.sandboxLeaseAcquisition?.outcome ?? null;
|
|
reuseReason = acquisition === "resumed" ? "cache_miss" : acquisition === "replacement" ? "replacement" : "created";
|
|
const priorStamp = leaseStamp({
|
|
lease: input.lease,
|
|
workspaceId: input.workspaceId,
|
|
providerLeaseId,
|
|
remoteCwd: target.remoteCwd,
|
|
});
|
|
mode = classifyNativeWorkspaceInbound({
|
|
kind: "new_run",
|
|
acquisition,
|
|
hasPriorStamp: priorStamp !== null,
|
|
});
|
|
const seedCacheCompanyDirectory = path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME, "generations", requireSafeSegment(input.companyId, "company_id"));
|
|
const seedCacheDirectory = path.join(seedCacheCompanyDirectory, "shared-v1");
|
|
runtime = await prepareRuntime({
|
|
runId: input.runId,
|
|
target,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
workspaceLocalDir: input.workspaceLocalDir,
|
|
mode,
|
|
seedCacheCompanyDirectory,
|
|
seedCacheDirectory,
|
|
workspaceGitIndexRefresh: input.workspaceGitIndexRefresh === true
|
|
&& acquisition === "created" && mode === "host_current"
|
|
&& !input.restartRecovery && !input.sameRunRecovery,
|
|
durableSeed: {
|
|
workspaceArchivePath: seedPaths.workspaceArchivePath,
|
|
gitArchivePath: seedPaths.gitArchivePath,
|
|
},
|
|
});
|
|
const currentSnapshot = runtime.workspaceSyncSnapshot;
|
|
if (!currentSnapshot) {
|
|
throw new Error("native_workspace_sync_snapshot_missing");
|
|
}
|
|
let workspaceArchiveSha256 = await sha256File(seedPaths.workspaceArchivePath);
|
|
let gitArchiveSha256 = currentSnapshot.gitSnapshot || currentSnapshot.repositories?.length
|
|
? await sha256File(seedPaths.gitArchivePath)
|
|
: null;
|
|
if (acquisition === "resumed" && priorStamp) {
|
|
const currentHostSha256 = directorySnapshotSha256(
|
|
currentSnapshot.baseline,
|
|
);
|
|
const verifiedWarmAdoption =
|
|
priorStamp.hostSha256 === currentHostSha256 &&
|
|
(await measureRunPhase("workspace_sync.warm_stamp", () => remoteStampMatches({ target, expected: priorStamp, gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories })));
|
|
if (verifiedWarmAdoption) {
|
|
mode = "adopt_remote";
|
|
reuseReason = "verified_reuse";
|
|
} else {
|
|
mode = "host_current";
|
|
reuseReason = "workspace_changed";
|
|
let preparedSeed: WorkspaceDurableSeedPaths | undefined;
|
|
try {
|
|
preparedSeed = await measureRunPhase("workspace_sync.durable_seed_verify", () => verifiedDurableSeed({ runId: input.runId,
|
|
seed: { workspaceArchiveSha256, gitArchiveSha256 },
|
|
gitSnapshot: currentSnapshot.gitSnapshot, repositories: currentSnapshot.repositories }));
|
|
} catch (error) {
|
|
if (!(error instanceof Error) || error.message !== "workspace_sync_out_unrecoverable") throw error;
|
|
}
|
|
runtime = await prepareRuntime({
|
|
runId: input.runId,
|
|
target,
|
|
workspaceLocalDir: input.workspaceLocalDir,
|
|
mode: preparedSeed ? "durable_seed" : mode,
|
|
seedCacheCompanyDirectory,
|
|
seedCacheDirectory,
|
|
baseline: currentSnapshot.baseline,
|
|
gitSnapshot: currentSnapshot.gitSnapshot,
|
|
repositories: currentSnapshot.repositories,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
durableSeed: preparedSeed ?? seedPaths,
|
|
});
|
|
if (!preparedSeed) {
|
|
workspaceArchiveSha256 = await sha256File(seedPaths.workspaceArchivePath);
|
|
gitArchiveSha256 = currentSnapshot.gitSnapshot || currentSnapshot.repositories?.length
|
|
? await sha256File(seedPaths.gitArchivePath) : null;
|
|
}
|
|
}
|
|
}
|
|
const snapshot = runtime.workspaceSyncSnapshot;
|
|
if (!snapshot) throw new Error("native_workspace_sync_snapshot_missing");
|
|
await measureRunPhase("workspace_sync.manifests_persist", () => persistSnapshotManifests(input.runId, snapshot));
|
|
const now = new Date().toISOString();
|
|
descriptor = {
|
|
schema: DESCRIPTOR_SCHEMA,
|
|
binding: {
|
|
runId: input.runId,
|
|
companyId: input.companyId,
|
|
workspaceId: input.workspaceId,
|
|
leaseId: target.leaseId ?? input.lease.id,
|
|
providerLeaseId,
|
|
localCwd: path.resolve(input.workspaceLocalDir),
|
|
remoteCwd: target.remoteCwd,
|
|
},
|
|
state: "prepared",
|
|
baselineSha256: directorySnapshotSha256(snapshot.baseline),
|
|
baseline: serializeDirectorySnapshot(snapshot.baseline),
|
|
gitSnapshot: snapshot.gitSnapshot,
|
|
...(snapshot.repositories?.length ? { repositories: snapshot.repositories } : {}),
|
|
seed: { workspaceArchiveSha256, gitArchiveSha256 },
|
|
createdAt: now,
|
|
finalizedAt: null,
|
|
finalHostSha256: null,
|
|
resourceDisposition: input.resourceDisposition ?? null,
|
|
};
|
|
}
|
|
|
|
const preparedRuntime = runtime;
|
|
let reference = await measureRunPhase("workspace_sync.descriptor_write", () => writeDescriptor(descriptor));
|
|
await measureRunPhase("workspace_sync.reference_write", () => persistRunReference(input.db, input.runId, reference));
|
|
await emitStartupReuse(input.onEvent, "workspace", {
|
|
state: mode === "adopt_remote" ? "reused" : "fresh",
|
|
reason: reuseReason,
|
|
});
|
|
let restorePromise: Promise<void> | null = null;
|
|
return {
|
|
mode,
|
|
get reference() {
|
|
return reference;
|
|
},
|
|
restoreWorkspace: async (assertOwnership) => {
|
|
if (!restorePromise) {
|
|
restorePromise = finalizePreparedRuntime({
|
|
db: input.db,
|
|
runId: input.runId,
|
|
target,
|
|
runtime: preparedRuntime,
|
|
descriptor,
|
|
assertOwnership,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
})
|
|
.then((finalizedReference) => {
|
|
reference = finalizedReference;
|
|
})
|
|
.catch((error) => {
|
|
restorePromise = null;
|
|
throw error;
|
|
});
|
|
}
|
|
await restorePromise;
|
|
},
|
|
cleanup: async () => { await preparedRuntime.cleanupWorkspaceSnapshot?.(); await cleanupNativeWorkspaceSync(input.runId); },
|
|
};
|
|
} catch (error) {
|
|
await runtime?.cleanupWorkspaceSnapshot?.();
|
|
if (!existingReference) await cleanupNativeWorkspaceSync(input.runId);
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
export async function readNativeWorkspaceSyncLocalCwd(input: {
|
|
companyId: string;
|
|
runId: string;
|
|
workspaceId: string;
|
|
reference: NativeWorkspaceSyncReference;
|
|
}): Promise<string> {
|
|
const existing = await readDescriptor(input);
|
|
try {
|
|
if (existing.descriptor.binding.companyId !== input.companyId ||
|
|
existing.descriptor.binding.workspaceId !== input.workspaceId) {
|
|
throw new Error("native_workspace_sync_descriptor_binding_mismatch");
|
|
}
|
|
return existing.descriptor.binding.localCwd;
|
|
} finally {
|
|
await disposeDirectorySnapshot(existing.baseline);
|
|
await disposeGitWorkspaceSnapshot(existing.descriptor.gitSnapshot);
|
|
for (const repository of existing.descriptor.repositories ?? []) {
|
|
await disposeGitWorkspaceSnapshot(repository.snapshot);
|
|
}
|
|
}
|
|
}
|
|
|
|
export async function resumeNativeWorkspaceSync(input: {
|
|
db: Db;
|
|
runId: string;
|
|
target: AdapterExecutionTarget;
|
|
assertOwnership?: () => Promise<void>;
|
|
onPhase?: RuntimeSpanRunner;
|
|
onCheckpoint?: (metrics: WorkspaceCheckpointMetrics) => Promise<void>;
|
|
}): Promise<boolean> {
|
|
if (input.target.kind !== "remote" || input.target.transport !== "sandbox") {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
const run = await input.db
|
|
.select({ companyId: heartbeatRuns.companyId, runnerProfileJson: heartbeatRuns.runnerProfileJson })
|
|
.from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.id, input.runId))
|
|
.limit(1)
|
|
.then((rows) => rows[0] ?? null);
|
|
const reference = readNativeWorkspaceSyncReference(
|
|
parseObject(run?.runnerProfileJson).nativeWorkspaceSync,
|
|
);
|
|
if (!reference) return false;
|
|
if (run && await hasMigratedLegacyWorkspaceSource(input.db, { companyId: run.companyId, runId: input.runId, reference }))
|
|
throw new Error("native_workspace_sync_source_migrated");
|
|
const existing = await readDescriptor({ runId: input.runId, reference });
|
|
const providerLeaseId =
|
|
input.target.sandboxLeaseAcquisition?.providerLeaseId ??
|
|
reference.providerLeaseId;
|
|
if (
|
|
providerLeaseId !== reference.providerLeaseId ||
|
|
input.target.remoteCwd !== reference.remoteCwd
|
|
) {
|
|
throw new Error("workspace_sync_out_unrecoverable");
|
|
}
|
|
await input.assertOwnership?.();
|
|
if (
|
|
existing.descriptor.state === "finalized" &&
|
|
existing.descriptor.finalHostSha256
|
|
) {
|
|
const stamp = finalizedWorkspaceStamp({
|
|
descriptor: existing.descriptor,
|
|
hostSha256: existing.descriptor.finalHostSha256,
|
|
});
|
|
if (
|
|
!(await remoteStampMatches({ target: input.target, expected: stamp }))
|
|
) {
|
|
await writeRemoteStamp({ target: input.target, stamp });
|
|
}
|
|
await persistLeaseStamp({
|
|
db: input.db,
|
|
leaseId: existing.descriptor.binding.leaseId,
|
|
stamp,
|
|
});
|
|
return true;
|
|
}
|
|
const runtime = await prepareRuntime({
|
|
runId: input.runId,
|
|
target: input.target,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
workspaceLocalDir: existing.descriptor.binding.localCwd,
|
|
mode: "adopt_remote",
|
|
baseline: existing.baseline,
|
|
gitSnapshot: existing.descriptor.gitSnapshot,
|
|
repositories: existing.descriptor.repositories,
|
|
});
|
|
await finalizePreparedRuntime({
|
|
db: input.db,
|
|
runId: input.runId,
|
|
target: input.target,
|
|
runtime,
|
|
descriptor: existing.descriptor,
|
|
assertOwnership: input.assertOwnership,
|
|
onPhase: input.onPhase,
|
|
onCheckpoint: input.onCheckpoint,
|
|
});
|
|
return true;
|
|
}
|
|
|
|
export async function cleanupNativeWorkspaceSync(runId: string): Promise<void> {
|
|
await fs.rm(descriptorDirectory(runId), { recursive: true, force: true });
|
|
}
|
|
|
|
export const nativeWorkspaceSyncInternals = {
|
|
descriptorPath,
|
|
legacyDescriptorPath,
|
|
durableSeedPaths,
|
|
persistSnapshotManifests,
|
|
writeDescriptor,
|
|
readDescriptor,
|
|
readReference: readNativeWorkspaceSyncReference,
|
|
sha256,
|
|
};
|