mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-26 18:31:34 +01:00
## Thinking Path > - Daytona preserves a stopped sandbox filesystem, but deleting or replacing a sandbox removes its only remote copy. > - Warm reuse therefore improves latency but cannot be Paperclip's durability boundary. > - The host execution workspace must remain authoritative after every successful turn, while same-run recovery must avoid overwriting unexported remote work. > - Result proposal, workspace export/merge, and terminal completion need a durable, replayable ordering so a crash never starts a duplicate provider turn. > - A paid browser acceptance suite must exercise both legacy Codex and Runner Codex for three real turns on one continuously warm Daytona sandbox. ## Linked Issues or Issue Description Refs #12901. Runner Codex did not previously export successful Daytona workspace changes back to the authoritative host workspace. That made warm reuse depend on Daytona's remote filesystem and left deleted/replacement sandboxes without a reliable reconstruction path. The existing paid fixture also lacked a focused three-turn continuity case for both Codex adapters. ## What Changed - Persist versioned, atomic native workspace-sync descriptors and durable seeds in `PAPERCLIP_HOME`, without credentials or a database migration. - Classify fresh, warm, replacement, and same-run-recovery workspace preparation explicitly; ambiguous lease/root/digest evidence fails closed. - Finalize native workspace export/merge after semantic result proposal and before run completion, with idempotent replay that never submits a second provider turn. - Surface legacy Codex workspace restoration failures instead of masking them, while preserving an earlier provider error when both fail. - Keep healthy reusable Daytona leases warm for legacy and native adapters, stamp finalized workspace generations, and retain existing cleanup behavior for per-turn or unhealthy leases. - Preserve Runner Codex's provider process/session across warm turns, including bounded post-terminal tail draining and exact authority rotation. - Add the exact paid `daytona-warm-continuity` matrix: - `legacy-codex × daytona × warm-three-turn` - `runner-codex × daytona × warm-three-turn` - Drive all three turns through the browser, verify ordered file continuity and stable lease/workspace/runtime identities, capture per-turn timings, and delete the sandbox immediately after assertions. - Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no package script was added. ## Verification - `pnpm typecheck` — passed, including migration safety (no migration added) - Focused server/runner Vitest coverage — 144 passed - `pnpm test:e2e:runner:unit` — 114 passed - `pnpm test:e2e:runner:typecheck` — passed - `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed, 1 helper ignored - `native-session-executor.test.ts` — 139 passed, including safe fail-closed cleanup after remote runner identity capture failure - Paid local browser acceptance, exact post-rebase Linux/amd64 runner binary: - Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created, resumed, resumed`; 10/10 matchers; cleanup passed - Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created, resumed, resumed`; 10/10 matchers; cleanup passed - [Protected paid GitHub Actions campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033) against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2: - Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`; evidence and cleanup passed - Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`; evidence and cleanup passed - Merge/enforcement, S3 history, and Pages publication jobs passed - Paid result artifacts were scanned for both provider credentials; neither secret was present. - Current PR checks — 31 passed, 1 expected Storybook skip; Greptile 5/5; Superagent security scan passed - `git diff --check origin/master...HEAD` — passed - Confirmed no `package.json`, lockfile, migration, or SQL changes. ## Risks - Workspace synchronization now sits on the terminal-success path, so a remote export failure deliberately prevents false success. Retryable state retains its lease/seed; loss of the only unexported remote copy fails closed. - Warm provider reuse has strict identity and quiescence checks. Mismatched or ambiguous evidence blocks reuse rather than risking concurrent provider work. - The paid suite incurs Daytona and Codex cost only in the existing protected scheduled/manual workflow and explicitly destroys its sandbox after each cell. ## Model Used OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real browser E2E execution, Rust/TypeScript test execution, and GitHub Actions diagnostics. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either linked an existing issue or described the issue in-PR - [x] I have not referenced internal/instance-local Paperclip issues or links - [x] My branch name contains no internal ticket id - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have documented the dedicated suite invocation without adding a package script - [x] I have considered and documented risks above - [x] All Paperclip CI gates are green on the current revision - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups on the current revision - [x] I will address all reviewer comments before requesting merge
306 lines
9.6 KiB
TypeScript
306 lines
9.6 KiB
TypeScript
import { mkdtemp, readdir, rm } from "node:fs/promises";
|
|
import os from "node:os";
|
|
import path from "node:path";
|
|
import { afterEach, describe, expect, it, vi } from "vitest";
|
|
import {
|
|
directorySnapshotSha256,
|
|
serializeDirectorySnapshot,
|
|
} from "@paperclipai/adapter-utils/workspace-restore-merge";
|
|
|
|
import {
|
|
classifyNativeWorkspaceInbound,
|
|
nativeWorkspaceSyncInternals,
|
|
readNativeWorkspaceSyncReference,
|
|
resumeNativeWorkspaceSync,
|
|
} from "../services/native-runtime/native-workspace-sync.js";
|
|
|
|
const digest = "a".repeat(64);
|
|
|
|
describe("native workspace sync durable metadata", () => {
|
|
const originalPaperclipHome = process.env.PAPERCLIP_HOME;
|
|
const originalPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
|
|
const cleanupDirs: string[] = [];
|
|
|
|
afterEach(async () => {
|
|
if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
|
|
else process.env.PAPERCLIP_HOME = originalPaperclipHome;
|
|
if (originalPaperclipInstanceId === undefined)
|
|
delete process.env.PAPERCLIP_INSTANCE_ID;
|
|
else process.env.PAPERCLIP_INSTANCE_ID = originalPaperclipInstanceId;
|
|
await Promise.all(
|
|
cleanupDirs
|
|
.splice(0)
|
|
.map((directory) => rm(directory, { recursive: true, force: true })),
|
|
);
|
|
});
|
|
|
|
it("classifies fresh, warm, replacement, and same-run recovery inputs", () => {
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "new_run",
|
|
acquisition: "created",
|
|
hasPriorStamp: false,
|
|
}),
|
|
).toBe("host_current");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "new_run",
|
|
acquisition: "resumed",
|
|
hasPriorStamp: true,
|
|
}),
|
|
).toBe("adopt_remote");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "new_run",
|
|
acquisition: "resumed",
|
|
hasPriorStamp: false,
|
|
}),
|
|
).toBe("host_current");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "new_run",
|
|
acquisition: "replacement",
|
|
hasPriorStamp: true,
|
|
}),
|
|
).toBe("host_current");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "existing_run",
|
|
restartRecovery: true,
|
|
sameProviderLease: true,
|
|
}),
|
|
).toBe("adopt_remote");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "existing_run",
|
|
restartRecovery: true,
|
|
sameProviderLease: false,
|
|
}),
|
|
).toBe("durable_seed");
|
|
expect(
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "existing_run",
|
|
restartRecovery: false,
|
|
sameRunRecovery: true,
|
|
sameProviderLease: true,
|
|
}),
|
|
).toBe("adopt_remote");
|
|
expect(() =>
|
|
classifyNativeWorkspaceInbound({
|
|
kind: "existing_run",
|
|
restartRecovery: false,
|
|
sameProviderLease: true,
|
|
}),
|
|
).toThrow("native_workspace_sync_unexpected_existing_descriptor");
|
|
});
|
|
|
|
it("reads backward-compatible references and the resource disposition", () => {
|
|
const base = {
|
|
schema: "paperclip.native-workspace-sync/v1",
|
|
state: "prepared",
|
|
descriptorSha256: digest,
|
|
baselineSha256: digest,
|
|
finalHostSha256: null,
|
|
workspaceId: "workspace-1",
|
|
leaseId: "lease-1",
|
|
providerLeaseId: "sandbox-1",
|
|
remoteCwd: "/workspace",
|
|
};
|
|
|
|
expect(readNativeWorkspaceSyncReference(base)).toEqual({
|
|
...base,
|
|
resourceDisposition: null,
|
|
});
|
|
expect(
|
|
readNativeWorkspaceSyncReference({
|
|
...base,
|
|
resourceDisposition: "keep_running",
|
|
}),
|
|
).toEqual({ ...base, resourceDisposition: "keep_running" });
|
|
expect(
|
|
readNativeWorkspaceSyncReference({
|
|
...base,
|
|
resourceDisposition: "delete_everything",
|
|
}),
|
|
).toBeNull();
|
|
});
|
|
|
|
it("rejects traversal before constructing a durable state path", () => {
|
|
expect(() =>
|
|
nativeWorkspaceSyncInternals.descriptorPath("../run", digest),
|
|
).toThrow("native_workspace_sync_invalid_run_id");
|
|
expect(() =>
|
|
nativeWorkspaceSyncInternals.descriptorPath("run-1", "../descriptor"),
|
|
).toThrow("native_workspace_sync_descriptor_digest_invalid");
|
|
});
|
|
|
|
it("writes one immutable descriptor when the same state is replayed", async () => {
|
|
const paperclipHome = await mkdtemp(
|
|
path.join(os.tmpdir(), "paperclip-native-workspace-sync-"),
|
|
);
|
|
cleanupDirs.push(paperclipHome);
|
|
process.env.PAPERCLIP_HOME = paperclipHome;
|
|
process.env.PAPERCLIP_INSTANCE_ID = "descriptor-test";
|
|
const baseline = {
|
|
exclude: [".paperclip-runtime"],
|
|
entries: new Map([
|
|
[
|
|
"continuity.txt",
|
|
{ kind: "file" as const, mode: 0o644, hash: digest },
|
|
],
|
|
]),
|
|
};
|
|
const descriptor = {
|
|
schema: "paperclip.native-workspace-sync/v1" as const,
|
|
binding: {
|
|
runId: "run-idempotent",
|
|
companyId: "company-1",
|
|
workspaceId: "workspace-1",
|
|
leaseId: "lease-1",
|
|
providerLeaseId: "sandbox-1",
|
|
localCwd: path.join(paperclipHome, "workspace"),
|
|
remoteCwd: "/workspace",
|
|
},
|
|
state: "prepared" as const,
|
|
baselineSha256: directorySnapshotSha256(baseline),
|
|
baseline: serializeDirectorySnapshot(baseline),
|
|
gitSnapshot: null,
|
|
seed: null,
|
|
createdAt: "2026-01-01T00:00:00.000Z",
|
|
finalizedAt: null,
|
|
finalHostSha256: null,
|
|
resourceDisposition: "keep_running" as const,
|
|
};
|
|
|
|
const first =
|
|
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
|
|
const second =
|
|
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
|
|
|
|
expect(second).toEqual(first);
|
|
const files = await readdir(
|
|
path.dirname(
|
|
nativeWorkspaceSyncInternals.descriptorPath(
|
|
descriptor.binding.runId,
|
|
first.descriptorSha256,
|
|
),
|
|
),
|
|
);
|
|
expect(files.filter((file) => file.endsWith(".json"))).toEqual([
|
|
`descriptor.${first.descriptorSha256}.json`,
|
|
]);
|
|
await expect(
|
|
nativeWorkspaceSyncInternals.readDescriptor({
|
|
runId: descriptor.binding.runId,
|
|
reference: first,
|
|
}),
|
|
).resolves.toMatchObject({ descriptor });
|
|
});
|
|
|
|
it("repairs finalized remote and lease stamps after an interrupted commit", async () => {
|
|
const paperclipHome = await mkdtemp(
|
|
path.join(os.tmpdir(), "paperclip-native-workspace-sync-repair-"),
|
|
);
|
|
cleanupDirs.push(paperclipHome);
|
|
process.env.PAPERCLIP_HOME = paperclipHome;
|
|
process.env.PAPERCLIP_INSTANCE_ID = "descriptor-repair-test";
|
|
const baseline = {
|
|
exclude: [".paperclip-runtime"],
|
|
entries: new Map([
|
|
[
|
|
"continuity.txt",
|
|
{ kind: "file" as const, mode: 0o644, hash: digest },
|
|
],
|
|
]),
|
|
};
|
|
const finalHostSha256 = "b".repeat(64);
|
|
const descriptor = {
|
|
schema: "paperclip.native-workspace-sync/v1" as const,
|
|
binding: {
|
|
runId: "run-finalized-repair",
|
|
companyId: "company-1",
|
|
workspaceId: "workspace-1",
|
|
leaseId: "lease-1",
|
|
providerLeaseId: "sandbox-1",
|
|
localCwd: path.join(paperclipHome, "workspace"),
|
|
remoteCwd: "/workspace",
|
|
},
|
|
state: "finalized" as const,
|
|
baselineSha256: directorySnapshotSha256(baseline),
|
|
baseline: serializeDirectorySnapshot(baseline),
|
|
gitSnapshot: null,
|
|
seed: null,
|
|
createdAt: "2026-01-01T00:00:00.000Z",
|
|
finalizedAt: "2026-01-01T00:01:00.000Z",
|
|
finalHostSha256,
|
|
resourceDisposition: "keep_running" as const,
|
|
};
|
|
const reference =
|
|
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
|
|
const rows = (values: unknown[]) => {
|
|
const query = {
|
|
from: () => query,
|
|
where: () => query,
|
|
for: () => query,
|
|
limit: () => query,
|
|
then: <TResult1 = unknown, TResult2 = never>(
|
|
onfulfilled?:
|
|
((value: unknown[]) => TResult1 | PromiseLike<TResult1>) | null,
|
|
onrejected?:
|
|
((reason: unknown) => TResult2 | PromiseLike<TResult2>) | null,
|
|
) => Promise.resolve(values).then(onfulfilled, onrejected),
|
|
};
|
|
return query;
|
|
};
|
|
let persistedLeaseMetadata: Record<string, unknown> | null = null;
|
|
const db = {
|
|
select: () =>
|
|
rows([{ runnerProfileJson: { nativeWorkspaceSync: reference } }]),
|
|
transaction: async (callback: (tx: unknown) => Promise<unknown>) =>
|
|
callback({
|
|
select: () => rows([{ metadata: { retained: true } }]),
|
|
update: () => ({
|
|
set: (value: { metadata: Record<string, unknown> }) => ({
|
|
where: async () => {
|
|
persistedLeaseMetadata = value.metadata;
|
|
},
|
|
}),
|
|
}),
|
|
}),
|
|
};
|
|
const execute = vi
|
|
.fn()
|
|
.mockResolvedValueOnce({ timedOut: false, exitCode: 1, stdout: "" })
|
|
.mockResolvedValueOnce({ timedOut: false, exitCode: 0, stdout: "" });
|
|
|
|
await expect(
|
|
resumeNativeWorkspaceSync({
|
|
db: db as never,
|
|
runId: descriptor.binding.runId,
|
|
target: {
|
|
kind: "remote",
|
|
transport: "sandbox",
|
|
remoteCwd: descriptor.binding.remoteCwd,
|
|
sandboxLeaseAcquisition: {
|
|
providerLeaseId: descriptor.binding.providerLeaseId,
|
|
},
|
|
runner: { execute },
|
|
} as never,
|
|
}),
|
|
).resolves.toBe(true);
|
|
|
|
expect(execute).toHaveBeenCalledTimes(2);
|
|
expect(persistedLeaseMetadata).toMatchObject({
|
|
retained: true,
|
|
nativeWorkspaceSync: {
|
|
schema: "paperclip.native-workspace-stamp/v1",
|
|
workspaceId: descriptor.binding.workspaceId,
|
|
providerLeaseId: descriptor.binding.providerLeaseId,
|
|
remoteCwd: descriptor.binding.remoteCwd,
|
|
hostSha256: finalHostSha256,
|
|
finalizedRunId: descriptor.binding.runId,
|
|
},
|
|
});
|
|
});
|
|
});
|