Files
PaperClipAI/server/src/__tests__/native-workspace-sync.test.ts
T
Dotta 0c1e7504c0 fix(runner): persist warm Daytona workspaces across turns (#12904)
## Thinking Path

> - Daytona preserves a stopped sandbox filesystem, but deleting or
replacing a sandbox removes its only remote copy.
> - Warm reuse therefore improves latency but cannot be Paperclip's
durability boundary.
> - The host execution workspace must remain authoritative after every
successful turn, while same-run recovery must avoid overwriting
unexported remote work.
> - Result proposal, workspace export/merge, and terminal completion
need a durable, replayable ordering so a crash never starts a duplicate
provider turn.
> - A paid browser acceptance suite must exercise both legacy Codex and
Runner Codex for three real turns on one continuously warm Daytona
sandbox.

## Linked Issues or Issue Description

Refs #12901.

Runner Codex did not previously export successful Daytona workspace
changes back to the authoritative host workspace. That made warm reuse
depend on Daytona's remote filesystem and left deleted/replacement
sandboxes without a reliable reconstruction path. The existing paid
fixture also lacked a focused three-turn continuity case for both Codex
adapters.

## What Changed

- Persist versioned, atomic native workspace-sync descriptors and
durable seeds in `PAPERCLIP_HOME`, without credentials or a database
migration.
- Classify fresh, warm, replacement, and same-run-recovery workspace
preparation explicitly; ambiguous lease/root/digest evidence fails
closed.
- Finalize native workspace export/merge after semantic result proposal
and before run completion, with idempotent replay that never submits a
second provider turn.
- Surface legacy Codex workspace restoration failures instead of masking
them, while preserving an earlier provider error when both fail.
- Keep healthy reusable Daytona leases warm for legacy and native
adapters, stamp finalized workspace generations, and retain existing
cleanup behavior for per-turn or unhealthy leases.
- Preserve Runner Codex's provider process/session across warm turns,
including bounded post-terminal tail draining and exact authority
rotation.
- Add the exact paid `daytona-warm-continuity` matrix:
  - `legacy-codex × daytona × warm-three-turn`
  - `runner-codex × daytona × warm-three-turn`
- Drive all three turns through the browser, verify ordered file
continuity and stable lease/workspace/runtime identities, capture
per-turn timings, and delete the sandbox immediately after assertions.
- Document `pnpm test:e2e:runner -- --suite daytona-warm-continuity`; no
package script was added.

## Verification

- `pnpm typecheck` — passed, including migration safety (no migration
added)
- Focused server/runner Vitest coverage — 144 passed
- `pnpm test:e2e:runner:unit` — 114 passed
- `pnpm test:e2e:runner:typecheck` — passed
- `pnpm --filter @paperclipai/paperclip-runner test:codex` — 66 passed,
1 helper ignored
- `native-session-executor.test.ts` — 139 passed, including safe
fail-closed cleanup after remote runner identity capture failure
- Paid local browser acceptance, exact post-rebase Linux/amd64 runner
binary:
- Runner Codex — passed in 1.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- Legacy Codex — passed in 2.7m; 3 runs; lease outcomes `created,
resumed, resumed`; 10/10 matchers; cleanup passed
- [Protected paid GitHub Actions
campaign](https://github.com/paperclipai/paperclip/actions/runs/34026735033)
against `7da42a91b95fa7fb2df126668ef7e37afb3b2b9d` — passed 2/2:
- Runner Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
- Legacy Codex — 3 runs; lease outcomes `created, resumed, resumed`;
evidence and cleanup passed
  - Merge/enforcement, S3 history, and Pages publication jobs passed
- Paid result artifacts were scanned for both provider credentials;
neither secret was present.
- Current PR checks — 31 passed, 1 expected Storybook skip; Greptile
5/5; Superagent security scan passed
- `git diff --check origin/master...HEAD` — passed
- Confirmed no `package.json`, lockfile, migration, or SQL changes.

## Risks

- Workspace synchronization now sits on the terminal-success path, so a
remote export failure deliberately prevents false success. Retryable
state retains its lease/seed; loss of the only unexported remote copy
fails closed.
- Warm provider reuse has strict identity and quiescence checks.
Mismatched or ambiguous evidence blocks reuse rather than risking
concurrent provider work.
- The paid suite incurs Daytona and Codex cost only in the existing
protected scheduled/manual workflow and explicitly destroys its sandbox
after each cell.

## Model Used

OpenAI Codex with GPT-5 agentic reasoning, repository inspection, real
browser E2E execution, Rust/TypeScript test execution, and GitHub
Actions diagnostics.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked an existing issue or described the issue
in-PR
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name contains no internal ticket id
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have documented the dedicated suite invocation without adding a
package script
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green on the current revision
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
on the current revision
- [x] I will address all reviewer comments before requesting merge
2026-09-06 08:16:12 -05:00

306 lines
9.6 KiB
TypeScript

import { mkdtemp, readdir, rm } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import {
directorySnapshotSha256,
serializeDirectorySnapshot,
} from "@paperclipai/adapter-utils/workspace-restore-merge";
import {
classifyNativeWorkspaceInbound,
nativeWorkspaceSyncInternals,
readNativeWorkspaceSyncReference,
resumeNativeWorkspaceSync,
} from "../services/native-runtime/native-workspace-sync.js";
const digest = "a".repeat(64);
describe("native workspace sync durable metadata", () => {
const originalPaperclipHome = process.env.PAPERCLIP_HOME;
const originalPaperclipInstanceId = process.env.PAPERCLIP_INSTANCE_ID;
const cleanupDirs: string[] = [];
afterEach(async () => {
if (originalPaperclipHome === undefined) delete process.env.PAPERCLIP_HOME;
else process.env.PAPERCLIP_HOME = originalPaperclipHome;
if (originalPaperclipInstanceId === undefined)
delete process.env.PAPERCLIP_INSTANCE_ID;
else process.env.PAPERCLIP_INSTANCE_ID = originalPaperclipInstanceId;
await Promise.all(
cleanupDirs
.splice(0)
.map((directory) => rm(directory, { recursive: true, force: true })),
);
});
it("classifies fresh, warm, replacement, and same-run recovery inputs", () => {
expect(
classifyNativeWorkspaceInbound({
kind: "new_run",
acquisition: "created",
hasPriorStamp: false,
}),
).toBe("host_current");
expect(
classifyNativeWorkspaceInbound({
kind: "new_run",
acquisition: "resumed",
hasPriorStamp: true,
}),
).toBe("adopt_remote");
expect(
classifyNativeWorkspaceInbound({
kind: "new_run",
acquisition: "resumed",
hasPriorStamp: false,
}),
).toBe("host_current");
expect(
classifyNativeWorkspaceInbound({
kind: "new_run",
acquisition: "replacement",
hasPriorStamp: true,
}),
).toBe("host_current");
expect(
classifyNativeWorkspaceInbound({
kind: "existing_run",
restartRecovery: true,
sameProviderLease: true,
}),
).toBe("adopt_remote");
expect(
classifyNativeWorkspaceInbound({
kind: "existing_run",
restartRecovery: true,
sameProviderLease: false,
}),
).toBe("durable_seed");
expect(
classifyNativeWorkspaceInbound({
kind: "existing_run",
restartRecovery: false,
sameRunRecovery: true,
sameProviderLease: true,
}),
).toBe("adopt_remote");
expect(() =>
classifyNativeWorkspaceInbound({
kind: "existing_run",
restartRecovery: false,
sameProviderLease: true,
}),
).toThrow("native_workspace_sync_unexpected_existing_descriptor");
});
it("reads backward-compatible references and the resource disposition", () => {
const base = {
schema: "paperclip.native-workspace-sync/v1",
state: "prepared",
descriptorSha256: digest,
baselineSha256: digest,
finalHostSha256: null,
workspaceId: "workspace-1",
leaseId: "lease-1",
providerLeaseId: "sandbox-1",
remoteCwd: "/workspace",
};
expect(readNativeWorkspaceSyncReference(base)).toEqual({
...base,
resourceDisposition: null,
});
expect(
readNativeWorkspaceSyncReference({
...base,
resourceDisposition: "keep_running",
}),
).toEqual({ ...base, resourceDisposition: "keep_running" });
expect(
readNativeWorkspaceSyncReference({
...base,
resourceDisposition: "delete_everything",
}),
).toBeNull();
});
it("rejects traversal before constructing a durable state path", () => {
expect(() =>
nativeWorkspaceSyncInternals.descriptorPath("../run", digest),
).toThrow("native_workspace_sync_invalid_run_id");
expect(() =>
nativeWorkspaceSyncInternals.descriptorPath("run-1", "../descriptor"),
).toThrow("native_workspace_sync_descriptor_digest_invalid");
});
it("writes one immutable descriptor when the same state is replayed", async () => {
const paperclipHome = await mkdtemp(
path.join(os.tmpdir(), "paperclip-native-workspace-sync-"),
);
cleanupDirs.push(paperclipHome);
process.env.PAPERCLIP_HOME = paperclipHome;
process.env.PAPERCLIP_INSTANCE_ID = "descriptor-test";
const baseline = {
exclude: [".paperclip-runtime"],
entries: new Map([
[
"continuity.txt",
{ kind: "file" as const, mode: 0o644, hash: digest },
],
]),
};
const descriptor = {
schema: "paperclip.native-workspace-sync/v1" as const,
binding: {
runId: "run-idempotent",
companyId: "company-1",
workspaceId: "workspace-1",
leaseId: "lease-1",
providerLeaseId: "sandbox-1",
localCwd: path.join(paperclipHome, "workspace"),
remoteCwd: "/workspace",
},
state: "prepared" as const,
baselineSha256: directorySnapshotSha256(baseline),
baseline: serializeDirectorySnapshot(baseline),
gitSnapshot: null,
seed: null,
createdAt: "2026-01-01T00:00:00.000Z",
finalizedAt: null,
finalHostSha256: null,
resourceDisposition: "keep_running" as const,
};
const first =
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
const second =
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
expect(second).toEqual(first);
const files = await readdir(
path.dirname(
nativeWorkspaceSyncInternals.descriptorPath(
descriptor.binding.runId,
first.descriptorSha256,
),
),
);
expect(files.filter((file) => file.endsWith(".json"))).toEqual([
`descriptor.${first.descriptorSha256}.json`,
]);
await expect(
nativeWorkspaceSyncInternals.readDescriptor({
runId: descriptor.binding.runId,
reference: first,
}),
).resolves.toMatchObject({ descriptor });
});
it("repairs finalized remote and lease stamps after an interrupted commit", async () => {
const paperclipHome = await mkdtemp(
path.join(os.tmpdir(), "paperclip-native-workspace-sync-repair-"),
);
cleanupDirs.push(paperclipHome);
process.env.PAPERCLIP_HOME = paperclipHome;
process.env.PAPERCLIP_INSTANCE_ID = "descriptor-repair-test";
const baseline = {
exclude: [".paperclip-runtime"],
entries: new Map([
[
"continuity.txt",
{ kind: "file" as const, mode: 0o644, hash: digest },
],
]),
};
const finalHostSha256 = "b".repeat(64);
const descriptor = {
schema: "paperclip.native-workspace-sync/v1" as const,
binding: {
runId: "run-finalized-repair",
companyId: "company-1",
workspaceId: "workspace-1",
leaseId: "lease-1",
providerLeaseId: "sandbox-1",
localCwd: path.join(paperclipHome, "workspace"),
remoteCwd: "/workspace",
},
state: "finalized" as const,
baselineSha256: directorySnapshotSha256(baseline),
baseline: serializeDirectorySnapshot(baseline),
gitSnapshot: null,
seed: null,
createdAt: "2026-01-01T00:00:00.000Z",
finalizedAt: "2026-01-01T00:01:00.000Z",
finalHostSha256,
resourceDisposition: "keep_running" as const,
};
const reference =
await nativeWorkspaceSyncInternals.writeDescriptor(descriptor);
const rows = (values: unknown[]) => {
const query = {
from: () => query,
where: () => query,
for: () => query,
limit: () => query,
then: <TResult1 = unknown, TResult2 = never>(
onfulfilled?:
((value: unknown[]) => TResult1 | PromiseLike<TResult1>) | null,
onrejected?:
((reason: unknown) => TResult2 | PromiseLike<TResult2>) | null,
) => Promise.resolve(values).then(onfulfilled, onrejected),
};
return query;
};
let persistedLeaseMetadata: Record<string, unknown> | null = null;
const db = {
select: () =>
rows([{ runnerProfileJson: { nativeWorkspaceSync: reference } }]),
transaction: async (callback: (tx: unknown) => Promise<unknown>) =>
callback({
select: () => rows([{ metadata: { retained: true } }]),
update: () => ({
set: (value: { metadata: Record<string, unknown> }) => ({
where: async () => {
persistedLeaseMetadata = value.metadata;
},
}),
}),
}),
};
const execute = vi
.fn()
.mockResolvedValueOnce({ timedOut: false, exitCode: 1, stdout: "" })
.mockResolvedValueOnce({ timedOut: false, exitCode: 0, stdout: "" });
await expect(
resumeNativeWorkspaceSync({
db: db as never,
runId: descriptor.binding.runId,
target: {
kind: "remote",
transport: "sandbox",
remoteCwd: descriptor.binding.remoteCwd,
sandboxLeaseAcquisition: {
providerLeaseId: descriptor.binding.providerLeaseId,
},
runner: { execute },
} as never,
}),
).resolves.toBe(true);
expect(execute).toHaveBeenCalledTimes(2);
expect(persistedLeaseMetadata).toMatchObject({
retained: true,
nativeWorkspaceSync: {
schema: "paperclip.native-workspace-stamp/v1",
workspaceId: descriptor.binding.workspaceId,
providerLeaseId: descriptor.binding.providerLeaseId,
remoteCwd: descriptor.binding.remoteCwd,
hostSha256: finalHostSha256,
finalizedRunId: descriptor.binding.runId,
},
});
});
});