Files
PaperClipAI/server/src/services/native-runtime/native-workspace-finalizer.ts
T
Dotta 1dceee9a4e fix(runner): persist warm Daytona workspaces (#12901)
## Thinking Path

> - Paperclip manages AI agent work and the execution state for each
task.
> - Remote agents run in sandbox environments such as Daytona.
> - Daytona keeps files while a sandbox is stopped, but deletion removes
those files.
> - Runner Codex did not copy successful remote workspace changes back
to the host workspace.
> - A warm sandbox could therefore hide data loss until Daytona replaced
or deleted the sandbox.
> - This pull request makes the host workspace durable after every
successful turn and keeps verified reusable sandboxes warm.
> - The benefit is reliable multi-turn work across warm reuse, restart,
stop, and sandbox replacement.

## Linked Issues or Issue Description

**What happened?**

A successful native Codex turn in Daytona could leave workspace changes
only in the remote sandbox. A later warm turn appeared to work because
it reused that filesystem. A replacement sandbox could start from stale
host data and lose the successful changes.

**Expected behavior**

Paperclip must merge each successful remote turn into the authoritative
host workspace before it completes the run. A verified warm lease may
reuse its remote files. A replacement lease must reconstruct the exact
durable workspace seed.

**Steps to reproduce**

1. Run Codex in a reusable Daytona environment.
2. Write a file during one successful turn.
3. Replace the Daytona sandbox before the next turn.
4. Observe that the next turn can start without the prior file on the
unpatched code.

Related remote workspace foundation: #10070.

## What Changed

- Added explicit `host_current`, `durable_seed`, and `adopt_remote`
workspace preparation modes.
- Added atomic, versioned native workspace descriptors and seed archives
under `PAPERCLIP_HOME`.
- Added real native sandbox export and three-way host merge before
terminal result completion.
- Added workspace-only recovery after a proposed result. Recovery does
not submit another provider turn or consume the provider retry budget.
- Added fail-closed handling when a sandbox with unexported changes is
gone.
- Kept healthy reusable Daytona sandboxes started for legacy Codex and
Runner Codex.
- Kept the Runner Codex process and provider session across verified
warm turns.
- Added the paid `daytona-warm-continuity` browser suite. It contains
exactly the legacy Codex and Runner Codex cells. Each cell performs
three measured turns.
- Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`.
No package script was added.
- Added no database migration. The metadata format is backward
compatible and idempotent.

## Verification

- `pnpm typecheck`
- `pnpm test:e2e:runner:unit` — 114 passed
- Native workspace, finalizer, session, and environment tests — 232
passed
- Daytona provider tests — 150 passed
- Workspace staging and merge tests — 98 passed
- Runner transport tests — 63 passed
- Legacy Codex restore tests — 5 passed
- Rust format and compile checks pass through root typecheck
- The paid Daytona suite was not run locally because the required
Daytona, OpenAI, and immutable image credentials are not present.

## Risks

- The main risk is an incorrect workspace identity or merge after a
crash. Durable descriptors bind the run, workspace, lease, provider
lease, local root, remote root, and baseline digest. Ambiguous evidence
fails closed.
- The host merge may conflict with concurrent host edits. The existing
three-way merge and exclusion rules handle this case and surface
failures.
- A deleted sandbox cannot recover unexported bytes. Paperclip now
blocks with `workspace_sync_out_unrecoverable` instead of reporting
success or rerunning the provider.
- There is no database migration. Descriptor writes and recovery are
atomic and idempotent.

## Model Used

OpenAI Codex with GPT-5. The run used agentic reasoning, repository
inspection, code execution, test execution, Git, and GitHub CLI tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-09-05 13:00:57 -05:00

241 lines
8.4 KiB
TypeScript

import fs from "node:fs/promises";
import { and, desc, eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import {
executionWorkspaces,
heartbeatRuns,
issues,
nativeRunFinalizations,
workspaceOperations,
} from "@paperclipai/db";
import { workspaceOperationService } from "../workspace-operations.js";
import { inspectManagedGitWorktreeBranch } from "../workspace-runtime.js";
import { environmentService } from "../environments.js";
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
import { resolveEnvironmentExecutionTarget } from "../environment-execution-target.js";
import {
readNativeWorkspaceSyncReference,
resumeNativeWorkspaceSync,
} from "./native-workspace-sync.js";
function record(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? value as Record<string, unknown>
: {};
}
function readString(value: unknown) {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function workspaceSyncFailure(
code: "workspace_sync_out_failed" | "workspace_sync_out_unrecoverable",
) {
return {
status: "failed" as const,
exitCode: 1,
stderr: `${code}\n`,
metadata: { workspaceSync: { code } },
};
}
/**
* Resume the real workspace-finalization action for a result-bearing native
* run. Durable sandbox-backed runs export and merge the remote workspace;
* older runs retain the local workspace validation path. The caller owns the
* durable retry and failure policy.
*/
export async function resumeNativeWorkspaceFinalization(input: {
db: Db;
runId: string;
environmentRuntime?: EnvironmentRuntimeService;
}) {
const bound = await input.db.select({
companyId: heartbeatRuns.companyId,
runtimeMode: heartbeatRuns.runtimeMode,
runnerProfileJson: heartbeatRuns.runnerProfileJson,
issueId: nativeRunFinalizations.issueId,
resultId: nativeRunFinalizations.resultId,
issueWorkspaceId: issues.executionWorkspaceId,
})
.from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.innerJoin(issues, and(
eq(issues.id, nativeRunFinalizations.issueId),
eq(issues.companyId, heartbeatRuns.companyId),
))
.where(eq(heartbeatRuns.id, input.runId))
.limit(1)
.then((rows) => rows[0] ?? null);
if (!bound || bound.runtimeMode !== "native" || !bound.resultId) {
throw new Error("native_workspace_finalization_binding_missing");
}
const previous = await input.db.select().from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, bound.companyId),
eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.phase, "workspace_finalize"),
)).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null);
const persistedInput = record(
record(bound.runnerProfileJson).nativeExecutionInput,
);
const nativeWorkspaceSync = readNativeWorkspaceSyncReference(
record(bound.runnerProfileJson).nativeWorkspaceSync,
);
const binding = record(persistedInput.binding);
const workspaceId = previous?.executionWorkspaceId
?? bound.issueWorkspaceId
?? readString(binding.executionWorkspaceId);
const workspace = workspaceId
? await input.db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.id, workspaceId),
eq(executionWorkspaces.companyId, bound.companyId),
)).limit(1).then((rows) => rows[0] ?? null)
: null;
const cwd = workspace?.providerRef ?? workspace?.cwd ?? previous?.cwd ?? null;
const recorder = workspaceOperationService(input.db).createRecorder({
companyId: bound.companyId,
heartbeatRunId: input.runId,
executionWorkspaceId: workspace?.id ?? workspaceId,
issueId: bound.issueId,
});
return recorder.recordOperation({
phase: "workspace_finalize",
cwd,
metadata: {
resumedFromOperationId: previous?.id ?? null,
owningService: "native_workspace_finalizer",
observation: workspace?.strategyType === "git_worktree"
? "managed_git_worktree_branch"
: "workspace_directory",
},
run: async () => {
if (nativeWorkspaceSync) {
if (!input.environmentRuntime) {
return {
status: "failed",
exitCode: 1,
stderr:
"Native workspace finalization cannot access the environment runtime.\n",
};
}
const environmentsSvc = environmentService(input.db);
const lease = await environmentsSvc.getLeaseById(
nativeWorkspaceSync.leaseId,
);
const environment = lease?.environmentId
? await environmentsSvc.getById(lease.environmentId)
: null;
if (
!lease ||
!environment ||
lease.companyId !== bound.companyId ||
environment.id !== lease.environmentId
) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
if (
lease.status === "expired" ||
lease.status === "failed" ||
lease.status === "pending_cleanup" ||
!lease.providerLeaseId ||
lease.providerLeaseId !== nativeWorkspaceSync.providerLeaseId
) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
const target = await resolveEnvironmentExecutionTarget({
db: input.db,
companyId: bound.companyId,
adapterType: "paperclip_runner",
environment,
leaseId: lease.id,
leaseMetadata: lease.metadata,
lease,
environmentRuntime: input.environmentRuntime,
});
if (!target) {
return {
status: "failed",
exitCode: 1,
stderr: "Native workspace finalization target is unavailable.\n",
};
}
try {
const restored = await resumeNativeWorkspaceSync({
db: input.db,
runId: input.runId,
target,
});
if (!restored) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
return {
status: "succeeded",
exitCode: 0,
system:
"Native workspace finalization restored the remote workspace.\n",
metadata: {
workspaceSync: {
schema: nativeWorkspaceSync.schema,
workspaceId: nativeWorkspaceSync.workspaceId,
leaseId: nativeWorkspaceSync.leaseId,
},
},
};
} catch (error) {
const code =
error instanceof Error &&
(error.message === "workspace_sync_out_unrecoverable" ||
error.message.includes("daytona_sandbox_not_found"))
? "workspace_sync_out_unrecoverable"
: "workspace_sync_out_failed";
return workspaceSyncFailure(code);
}
}
if (!cwd) {
return {
status: "failed",
exitCode: 1,
stderr: "Native workspace finalization cannot resolve the persisted workspace path.\n",
};
}
const isDirectory = await fs.stat(cwd).then((stat) => stat.isDirectory()).catch(() => false);
if (!isDirectory) {
return {
status: "failed",
exitCode: 1,
stderr: `Native workspace finalization could not access workspace directory ${cwd}.\n`,
};
}
if (workspace?.strategyType === "git_worktree") {
const inspection = await inspectManagedGitWorktreeBranch({
worktreePath: cwd,
expectedBranchName: workspace.branchName,
});
if (!inspection.valid) {
return {
status: "failed",
exitCode: 1,
stderr: `Managed git worktree validation failed: ${inspection.reason ?? "unknown failure"}.\n`,
metadata: { managedGitWorktreeBranch: inspection },
};
}
return {
status: "succeeded",
exitCode: 0,
system: "Native workspace finalization validated the managed git worktree.\n",
metadata: { managedGitWorktreeBranch: inspection },
};
}
return {
status: "succeeded",
exitCode: 0,
system: "Native workspace finalization validated the persisted workspace directory.\n",
metadata: { observedWorkspacePath: cwd },
};
},
});
}