Files
PaperClipAI/server/src/services/native-runtime/native-workspace-finalizer.ts
T
Nicky LeachandPaperclip 7f69ef8b44 Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path

> - Paperclip lets plugins own managed Git workspaces.
> - A portable Runner task can move to a different physical allocation.
> - The checkout must be present on the allocation that launches the
Runner.
> - Workspace preparation and finalization need exact placement when
they retry.
> - This change checks the placement at Runner admission and records it
before finalization.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Workspace preparation and finalization for portable Runner tasks.

**Current behavior**
A task replacement can change the allocation after Git preparation. A
lost finalization reply can leave retry without the original placement.

**Proposed behavior**
Prepare the checkout on the actual allocation before admission. Record
finalization placement so an exact retry can use it.

Portable ownership and controller recovery are in
[#21066](https://github.com/paperclipai/paperclip/pull/21066).

## What Changed

- Reserve the managed workspace before acquiring the portable task
lease.
- Pass the task placement to the workspace provider. Reuse the described
task while placement stays the same and prepare again before admitting a
replacement.
- Record the initial preparation's run, agent, generation, and optional
task placement before dispatch. Inspection and retries on the same
Launcher replay the same request; VM and sandbox preparation use the
same recovery path.
- Keep provider request identity separate from Git credential authority:
reservation reconciliation uses the existing project-consent path in a
host-only scope bound to the plugin and workspace. Foreground
preparation still requires an active run; consent, repository, and
revocation checks are unchanged.
- Acquire the current Launcher before retrying preparation. If placement
changes, use ordinary preparation on the replacement so the provider can
settle the prior writer and recover the retained checkout. No additional
provider discovery call is added.
- Select finalization placement before publishing the operation record,
then dispatch and recover using that saved choice. Retain the Launcher
during bounded finalization retries.
- Preserve interrupted pre-upgrade finalization requests without
placement, including their original operation IDs and provider routing.
No migration or new durable mode flag is required.

## Verification

- Final revision `74ea3e6712`: server typecheck and exact SDK source-pin
verification (538 inputs) passed.
- All 185 tests passed across the full plugin worker manager, project
Git authorization, and plugin workspace integration suites (including 56
workspace tests). Independent verification reran the three lost-reply
cases: inspection, same-Launcher retry, and replacement after deleting
the original Launcher. The tests assert the correct request identity,
durable intent, and clearing that intent after success.
- Before the final retry-accounting fix, all 323 tests passed across
workspace admission, runtime execution, native workspace finalization,
plugin workspace integration, and terminal cleanup retry. Regressions
exercise crashes after operation insertion but before dispatch in both
finalization paths, exact replay after lease release, and retry after
ambiguous placement clears without publishing an incomplete operation.
Pre-upgrade interrupted requests retain their operation IDs despite an
available portable lease.
- Final retry-accounting fix: all 84 native workspace finalizer,
failure, and supersession tests passed. A real missing-plugin regression
first failed on the prior commit, then verified healthy-run progress,
three-attempt exhaustion, terminal cleanup, and absence of a misleading
workspace operation. Independent verification also passed the legacy and
crash-before-dispatch cases.
- Worker-process tests cover both preparation RPC methods: the provider
still receives the original run and agent, maintenance receives
project-scoped credential authority, and foreground or mismatched-scope
requests remain denied. Database tests confirm a stopped run cannot
authorize foreground Git access while valid project consent can
authorize maintenance.
- Existing metadata protection coverage proves callers cannot overwrite
the private preparation intent or read it through workspace responses.
- Two existing Launcher Git tests passed: retry preserves dirty files
and the original base commit without fetching, and retry resumes the
saved checkout intent while rejecting a changed pending branch.
- Implementation was delegated to GPT-6.1 Sol and independently
reviewed. The GitHub stack's automatic rebase preserved the original
workspace-placement patch.

- [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706)
passed on `74ea3e6712`, including server and workspace tests, Runner,
end-to-end tests, typecheck, build, and canary dry run. Ordinary
Greptile and
[Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388)
both scored `74ea3e6712` 5/5, with no unresolved findings.

## Risks

- Live Daytona end-to-end qualification is pending. The checkout record
persists on the mounted volume; replacement preparation relies on the
provider confirming the previous writer is settled before using that
retained checkout. Local tests cover Core request routing and Launcher's
durable checkout recovery separately.

## Model Used

- OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and
independent review.

## Checklist

- [x] Thinking path and model are included.
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] The issue is described using the enhancement template.
- [x] Local checks passed and new behavior has tests.
- [x] All Paperclip CI gates are green.
- [x] Greptile is 5/5 with no open findings.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-12 12:12:41 -08:00

413 lines
18 KiB
TypeScript

import { classifyNativeWorkspaceFailure } from "./native-workspace-failure.js";
import { readRemoteWorkspaceAuthority, saveRemoteWorkspace } from "../remote-workspaces.js";
import { remoteWorkspaceFailureCause } from "../remote-workspace-transport.js";
import { recordNativeFinalizationFailure } from "./native-run-finalizer.js";
import { inspectPluginWorkspaceForFinalization } from "./native-plugin-workspace-finalization.js";
import { resolvePluginWorkspaceFinalization, type WorkspaceWorkers } from "../plugin-workspaces.js";
import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js";
import { withNativeWorkspaceFinalizationOwnership, type NativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js";
import { settleSupersededNativeWorkspaceFinalizations } from "./native-workspace-finalization-supersession.js";
import { logger } from "../../middleware/logger.js";
import fs from "node:fs/promises";
import { and, desc, eq } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { pluginWorkspaceParamsSchema } from "@paperclipai/plugin-sdk";
import {
executionWorkspaces,
heartbeatRuns,
issues,
nativeRunFinalizations,
workspaceOperations,
} from "@paperclipai/db";
import { workspaceOperationService } from "../workspace-operations.js";
import { inspectManagedGitWorktreeBranch } from "../workspace-runtime.js";
import { environmentService } from "../environments.js";
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
import { resolveEnvironmentExecutionTarget } from "../environment-execution-target.js";
import { measureWorkspaceSyncPhase } from "../execution-preparation-timing.js";
import {
readNativeWorkspaceSyncReference,
resumeNativeWorkspaceSync,
} from "./native-workspace-sync.js";
function record(value: unknown): Record<string, unknown> {
return value && typeof value === "object" && !Array.isArray(value)
? value as Record<string, unknown>
: {};
}
function readString(value: unknown) {
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
}
function workspaceSyncFailure(
code: "workspace_sync_out_failed" | "workspace_sync_out_unrecoverable",
cause?: string,
) {
return {
status: "failed" as const,
exitCode: 1,
stderr: cause ? `${code}\n${cause}\n` : `${code}\n`,
metadata: { workspaceSync: cause ? { code, cause } : { code } },
};
}
/**
* Resume the real workspace-finalization action for a result-bearing native
* run. Durable sandbox-backed runs export and merge the remote workspace;
* older runs retain the local workspace validation path. The caller owns the
* durable retry and failure policy.
*/
export async function attemptNativeWorkspaceFinalization(input: {
db: Db;
runId: string;
environmentRuntime?: EnvironmentRuntimeService;
pluginWorkerManager?: WorkspaceWorkers;
}) {
const bound = await input.db.select({
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
runtimeMode: heartbeatRuns.runtimeMode,
runnerProfileJson: heartbeatRuns.runnerProfileJson,
issueId: nativeRunFinalizations.issueId,
resultId: nativeRunFinalizations.resultId,
issueWorkspaceId: issues.executionWorkspaceId,
})
.from(heartbeatRuns)
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
.innerJoin(issues, and(
eq(issues.id, nativeRunFinalizations.issueId),
eq(issues.companyId, heartbeatRuns.companyId),
))
.where(eq(heartbeatRuns.id, input.runId))
.limit(1)
.then((rows) => rows[0] ?? null);
if (!bound || bound.runtimeMode !== "native" || !bound.resultId) {
throw new Error("native_workspace_finalization_binding_missing");
}
// An attempt interrupted by a controller crash stays `running`; once this
// owner records a terminal attempt, settle the orphan under the same proof.
const settleSuperseded = async (ownership: NativeWorkspaceFinalizationOwnership,
superseding: { id: string; status: string }) => {
try {
await ownership.assertHeld();
await settleSupersededNativeWorkspaceFinalizations(input.db, {
companyId: bound.companyId, runId: input.runId, issueId: bound.issueId, superseding,
});
} catch (error) {
// Settlement is bookkeeping; it must not change the finalization outcome.
logger.warn({ runId: input.runId, error }, "Superseded workspace finalization could not be settled");
}
};
const owned = await withNativeWorkspaceFinalizationOwnership({
db: input.db, companyId: bound.companyId, runId: input.runId,
}, async (ownership) => {
// A sweep can be admitted before the live owner publishes a permanent
// failure or retry delay, then acquire ownership after that owner releases.
// Recheck admission inside the lock before any copyback or operation receipt.
const admission = await input.db.select({
phase: nativeRunFinalizations.phase,
nextAttemptAt: nativeRunFinalizations.nextAttemptAt,
resultId: nativeRunFinalizations.resultId,
}).from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.runId, input.runId),
eq(nativeRunFinalizations.companyId, bound.companyId),
eq(nativeRunFinalizations.issueId, bound.issueId),
)).limit(1).then((rows) => rows[0] ?? null);
if (!admission || admission.resultId !== bound.resultId) {
throw new Error("native_workspace_finalization_binding_missing");
}
if (admission.phase === "terminal_failure") return { kind: "inactive" } as const;
if (admission.nextAttemptAt && admission.nextAttemptAt > new Date()) {
return { kind: "retry", retryAt: admission.nextAttemptAt } as const;
}
const successful = await input.db.select().from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, bound.companyId),
eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.issueId, bound.issueId),
eq(workspaceOperations.phase, "workspace_finalize"),
eq(workspaceOperations.status, "succeeded"),
)).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null);
// A stale failure from a pre-fencing controller cannot invalidate exported work.
if (successful) {
await settleSuperseded(ownership, successful);
return { kind: "completed", operation: successful } as const;
}
const previous = await input.db.select().from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, bound.companyId),
eq(workspaceOperations.heartbeatRunId, input.runId),
eq(workspaceOperations.issueId, bound.issueId),
eq(workspaceOperations.phase, "workspace_finalize"),
)).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null);
const persistedInput = record(
record(bound.runnerProfileJson).nativeExecutionInput,
);
const nativeWorkspaceSync = readNativeWorkspaceSyncReference(
record(bound.runnerProfileJson).nativeWorkspaceSync,
);
const binding = record(persistedInput.binding);
const workspaceId = previous?.executionWorkspaceId
?? bound.issueWorkspaceId
?? readString(binding.executionWorkspaceId);
const workspace = workspaceId
? await input.db.select().from(executionWorkspaces).where(and(
eq(executionWorkspaces.id, workspaceId),
eq(executionWorkspaces.companyId, bound.companyId),
)).limit(1).then((rows) => rows[0] ?? null)
: null;
const cwd = workspace?.providerRef ?? workspace?.cwd ?? previous?.cwd ?? null;
let pluginFinalizationPlan: Awaited<ReturnType<typeof resolvePluginWorkspaceFinalization>>;
if (workspace && !nativeWorkspaceSync && !readRemoteWorkspaceAuthority(workspace.metadata)) {
try {
const taskLease = previous
? pluginWorkspaceParamsSchema.shape.taskLease.parse(previous.metadata?.pluginWorkspaceTaskLease) ?? null
: undefined;
pluginFinalizationPlan = await resolvePluginWorkspaceFinalization(input.db, workspace, {
runId: input.runId, agentId: bound.agentId, taskLease,
});
} catch {
await ownership.assertHeld();
return { kind: "failed", operation: { stderrExcerpt: "plugin_workspace_inspection_failed" } } as const;
}
}
const recordedTaskLease = pluginFinalizationPlan?.kind === "finalize" ? pluginFinalizationPlan.taskLease : undefined;
const recorder = workspaceOperationService(input.db).createRecorder({
companyId: bound.companyId,
heartbeatRunId: input.runId,
// The native binding also uses this field as a directory-only containment token.
// Only attach it to the operation when it resolves to a company-owned row, because
// workspace_operations.execution_workspace_id is a real execution-workspace FK.
executionWorkspaceId: workspace?.id ?? null,
issueId: bound.issueId,
});
const operation = await recorder.recordOperation({
phase: "workspace_finalize",
cwd,
metadata: {
resumedFromOperationId: previous?.id ?? null,
owningService: "native_workspace_finalizer",
...(recordedTaskLease ? { pluginWorkspaceTaskLease: recordedTaskLease } : {}),
observation: workspace?.strategyType === "git_worktree"
? "managed_git_worktree_branch"
: "workspace_directory",
},
run: async () => {
await ownership.assertHeld();
const remoteAuthority = readRemoteWorkspaceAuthority(workspace?.metadata);
if (remoteAuthority && workspace) {
if (!input.environmentRuntime || !remoteAuthority.location) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
const location = remoteAuthority.location;
const environmentsSvc = environmentService(input.db);
const lease = await environmentsSvc.getLeaseById(location.leaseId);
const environment = lease?.environmentId ? await environmentsSvc.getById(lease.environmentId) : null;
if (!lease || !environment || lease.companyId !== bound.companyId
|| lease.heartbeatRunId !== input.runId || lease.executionWorkspaceId !== workspace.id
|| lease.providerLeaseId !== location.providerLeaseId || environment.id !== location.environmentId
|| ["expired", "failed", "pending_cleanup"].includes(lease.status)) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
const target = await resolveEnvironmentExecutionTarget({
db: input.db, companyId: bound.companyId, adapterType: "paperclip_runner",
environment, leaseId: lease.id, leaseMetadata: lease.metadata, lease,
environmentRuntime: input.environmentRuntime,
});
if (!target) return workspaceSyncFailure("workspace_sync_out_unrecoverable");
try {
const saved = await saveRemoteWorkspace({
db: input.db, workspace, runId: input.runId, lease, target,
assertOwned: ownership.assertHeld,
});
await ownership.assertHeld();
return { status: "succeeded", exitCode: 0,
system: "Native workspace files were saved to durable storage.\n",
metadata: { remoteWorkspace: saved } };
} catch (error) {
await ownership.assertHeld();
// The code keeps the retry policy. The redacted remote cause (for
// example a full sandbox disk or the failed capture step) is named on
// the operation; other error text stays off the record.
return workspaceSyncFailure(classifyNativeWorkspaceFailure(error).code === "workspace_sync_out_unrecoverable"
? "workspace_sync_out_unrecoverable" : "workspace_sync_out_failed", remoteWorkspaceFailureCause(error) ?? undefined);
}
}
if (nativeWorkspaceSync) {
if (!input.environmentRuntime) {
return {
status: "failed",
exitCode: 1,
stderr:
"Native workspace finalization cannot access the environment runtime.\n",
};
}
const environmentsSvc = environmentService(input.db);
const lease = await environmentsSvc.getLeaseById(
nativeWorkspaceSync.leaseId,
);
const environment = lease?.environmentId
? await environmentsSvc.getById(lease.environmentId)
: null;
if (
!lease ||
!environment ||
lease.companyId !== bound.companyId ||
environment.id !== lease.environmentId
) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
if (
lease.status === "expired" ||
lease.status === "failed" ||
lease.status === "pending_cleanup" ||
!lease.providerLeaseId ||
lease.providerLeaseId !== nativeWorkspaceSync.providerLeaseId
) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
const target = await resolveEnvironmentExecutionTarget({
db: input.db,
companyId: bound.companyId,
adapterType: "paperclip_runner",
environment,
leaseId: lease.id,
leaseMetadata: lease.metadata,
lease,
environmentRuntime: input.environmentRuntime,
});
if (!target) {
return {
status: "failed",
exitCode: 1,
stderr: "Native workspace finalization target is unavailable.\n",
};
}
try {
const restored = await restoreNativeWorkspaceBestEffort({
db: input.db,
runId: input.runId,
assertOwnership: ownership.assertHeld,
restore: () => resumeNativeWorkspaceSync({
db: input.db, runId: input.runId, target, assertOwnership: ownership.assertHeld,
onPhase: measureWorkspaceSyncPhase,
}),
});
await ownership.assertHeld();
if (restored === false) {
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
}
return {
status: "succeeded",
exitCode: 0,
system:
restored
? "Native workspace finalization restored the remote workspace.\n"
: "Unsafe workspace export omitted; finalization completed.\n",
metadata: {
workspaceSync: {
schema: nativeWorkspaceSync.schema,
workspaceId: nativeWorkspaceSync.workspaceId,
leaseId: nativeWorkspaceSync.leaseId,
},
},
};
} catch (error) {
await ownership.assertHeld();
const code = classifyNativeWorkspaceFailure(error).code === "workspace_sync_out_unrecoverable"
? "workspace_sync_out_unrecoverable" : "workspace_sync_out_failed";
return workspaceSyncFailure(code);
}
}
let pluginWorkspace: Record<string, unknown> | null;
try {
pluginWorkspace = workspace
? await inspectPluginWorkspaceForFinalization(input.db, input.pluginWorkerManager, workspace, bound.agentId, input.runId, {
taskLease: null,
plan: pluginFinalizationPlan,
})
: null;
} catch (error) {
await ownership.assertHeld();
const message = error instanceof Error ? error.message : null;
const code = message === "plugin_workspace_branch_mismatch" || message === "plugin_workspace_owner_mismatch"
? message : "plugin_workspace_inspection_failed";
return { status: "failed", exitCode: 1, stderr: `${code}\n` };
}
await ownership.assertHeld();
if (pluginWorkspace) return { status: "succeeded", exitCode: 0,
system: "Native workspace finalization used the plugin workspace owner.\n",
metadata: { pluginWorkspace } };
if (!cwd) {
return {
status: "failed",
exitCode: 1,
stderr: "Native workspace finalization cannot resolve the persisted workspace path.\n",
};
}
const isDirectory = await fs.stat(cwd).then((stat) => stat.isDirectory()).catch(() => false);
await ownership.assertHeld();
if (!isDirectory) {
return {
status: "failed",
exitCode: 1,
stderr: `Native workspace finalization could not access workspace directory ${cwd}.\n`,
};
}
if (workspace?.strategyType === "git_worktree") {
const inspection = await inspectManagedGitWorktreeBranch({
worktreePath: cwd,
expectedBranchName: workspace.branchName,
});
if (!inspection.valid) {
return {
status: "failed",
exitCode: 1,
stderr: `Managed git worktree validation failed: ${inspection.reason ?? "unknown failure"}.\n`,
metadata: { managedGitWorktreeBranch: inspection },
};
}
return {
status: "succeeded",
exitCode: 0,
system: "Native workspace finalization validated the managed git worktree.\n",
metadata: { managedGitWorktreeBranch: inspection },
};
}
return {
status: "succeeded",
exitCode: 0,
system: "Native workspace finalization validated the persisted workspace directory.\n",
metadata: { observedWorkspacePath: cwd },
};
},
});
await settleSuperseded(ownership, operation);
return { kind: "completed", operation } as const;
});
return owned.acquired ? owned.value : { kind: "waiting", reason: owned.reason } as const;
}
export async function resumeNativeWorkspaceFinalization(input: Parameters<typeof attemptNativeWorkspaceFinalization>[0]) {
const result = await attemptNativeWorkspaceFinalization(input);
return result.kind === "completed" ? result.operation : null;
}
export function recordNativeWorkspaceFinalizationFailure(db: Db, runId: string,
operation: { stderrExcerpt: string | null; metadata?: Record<string, unknown> | null }) {
const unrecoverable = operation.stderrExcerpt?.includes("workspace_sync_out_unrecoverable");
const pluginFailure = operation.stderrExcerpt?.trim();
const pluginRepair = pluginFailure === "plugin_workspace_branch_mismatch" || pluginFailure === "plugin_workspace_owner_mismatch";
return recordNativeFinalizationFailure({
db, runId,
error: new Error(pluginRepair ? `native_${pluginFailure}` : unrecoverable
? "native_workspace_sync_out_unrecoverable" : "native_workspace_sync_out_failed"),
projectRunStatus: true, failureScope: "workspace", permanent: unrecoverable,
failureCause: readString(record(record(operation.metadata).workspaceSync).cause),
});
}