mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-17 21:34:48 +01:00
Prepare and finalize managed workspaces on the active Launcher (#20743)
## Thinking Path > - Paperclip lets plugins own managed Git workspaces. > - A portable Runner task can move to a different physical allocation. > - The checkout must be present on the allocation that launches the Runner. > - Workspace preparation and finalization need exact placement when they retry. > - This change checks the placement at Runner admission and records it before finalization. ## Linked Issues or Issue Description **What existing behavior does this improve?** Workspace preparation and finalization for portable Runner tasks. **Current behavior** A task replacement can change the allocation after Git preparation. A lost finalization reply can leave retry without the original placement. **Proposed behavior** Prepare the checkout on the actual allocation before admission. Record finalization placement so an exact retry can use it. Portable ownership and controller recovery are in [#21066](https://github.com/paperclipai/paperclip/pull/21066). ## What Changed - Reserve the managed workspace before acquiring the portable task lease. - Pass the task placement to the workspace provider. Reuse the described task while placement stays the same and prepare again before admitting a replacement. - Record the initial preparation's run, agent, generation, and optional task placement before dispatch. Inspection and retries on the same Launcher replay the same request; VM and sandbox preparation use the same recovery path. - Keep provider request identity separate from Git credential authority: reservation reconciliation uses the existing project-consent path in a host-only scope bound to the plugin and workspace. Foreground preparation still requires an active run; consent, repository, and revocation checks are unchanged. - Acquire the current Launcher before retrying preparation. If placement changes, use ordinary preparation on the replacement so the provider can settle the prior writer and recover the retained checkout. No additional provider discovery call is added. - Select finalization placement before publishing the operation record, then dispatch and recover using that saved choice. Retain the Launcher during bounded finalization retries. - Preserve interrupted pre-upgrade finalization requests without placement, including their original operation IDs and provider routing. No migration or new durable mode flag is required. ## Verification - Final revision `74ea3e6712`: server typecheck and exact SDK source-pin verification (538 inputs) passed. - All 185 tests passed across the full plugin worker manager, project Git authorization, and plugin workspace integration suites (including 56 workspace tests). Independent verification reran the three lost-reply cases: inspection, same-Launcher retry, and replacement after deleting the original Launcher. The tests assert the correct request identity, durable intent, and clearing that intent after success. - Before the final retry-accounting fix, all 323 tests passed across workspace admission, runtime execution, native workspace finalization, plugin workspace integration, and terminal cleanup retry. Regressions exercise crashes after operation insertion but before dispatch in both finalization paths, exact replay after lease release, and retry after ambiguous placement clears without publishing an incomplete operation. Pre-upgrade interrupted requests retain their operation IDs despite an available portable lease. - Final retry-accounting fix: all 84 native workspace finalizer, failure, and supersession tests passed. A real missing-plugin regression first failed on the prior commit, then verified healthy-run progress, three-attempt exhaustion, terminal cleanup, and absence of a misleading workspace operation. Independent verification also passed the legacy and crash-before-dispatch cases. - Worker-process tests cover both preparation RPC methods: the provider still receives the original run and agent, maintenance receives project-scoped credential authority, and foreground or mismatched-scope requests remain denied. Database tests confirm a stopped run cannot authorize foreground Git access while valid project consent can authorize maintenance. - Existing metadata protection coverage proves callers cannot overwrite the private preparation intent or read it through workspace responses. - Two existing Launcher Git tests passed: retry preserves dirty files and the original base commit without fetching, and retry resumes the saved checkout intent while rejecting a changed pending branch. - Implementation was delegated to GPT-6.1 Sol and independently reviewed. The GitHub stack's automatic rebase preserved the original workspace-placement patch. - [Full CI](https://github.com/paperclipai/paperclip/actions/runs/43293046205/job/131452117706) passed on `74ea3e6712`, including server and workspace tests, Runner, end-to-end tests, typecheck, build, and canary dry run. Ordinary Greptile and [Apex](https://github.com/paperclipai/paperclip/pull/20743#issuecomment-6788920388) both scored `74ea3e6712` 5/5, with no unresolved findings. ## Risks - Live Daytona end-to-end qualification is pending. The checkout record persists on the mounted volume; replacement preparation relies on the provider confirming the previous writer is settled before using that retained checkout. Local tests cover Core request routing and Launcher's durable checkout recovery separately. ## Model Used - OpenAI Codex GPT-6 with GPT-6.1 Sol agents for implementation and independent review. ## Checklist - [x] Thinking path and model are included. - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] The issue is described using the enhancement template. - [x] Local checks passed and new behavior has tests. - [x] All Paperclip CI gates are green. - [x] Greptile is 5/5 with no open findings. --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
ea400ecb07
commit
7f69ef8b44
23 files changed
+1029
-243
No files matched your search
@@ -57,7 +57,7 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { isNewTerminalRunFailure, reportRunFailure } from "../run-failure-report.js";
|
||||
import { resumeNativeWorkspaceFinalization, recordNativeWorkspaceFinalizationFailure } from "./native-workspace-finalizer.js";
|
||||
import { attemptNativeWorkspaceFinalization, recordNativeWorkspaceFinalizationFailure } from "./native-workspace-finalizer.js";
|
||||
import { dismissObsoleteNativePolicyReviews } from "./obsolete-policy-reviews.js";
|
||||
import {
|
||||
cleanupNativeWorkspaceSync,
|
||||
@@ -654,23 +654,21 @@ async function reconcileNativeFinalizationsPass(
|
||||
if (!recoveryDecision.effects.some((effect) => effect.kind === "resume_workspace_operation")) {
|
||||
throw new Error("native_reconciliation_workspace_resume_policy_missing");
|
||||
}
|
||||
const operation = await resumeNativeWorkspaceFinalization({
|
||||
const outcome = await attemptNativeWorkspaceFinalization({
|
||||
db,
|
||||
runId: row.runId,
|
||||
environmentRuntime: options.environmentRuntime,
|
||||
pluginWorkerManager: options.pluginWorkerManager,
|
||||
});
|
||||
// Busy is ownership, not another failed export or retry-budget debit.
|
||||
if (!operation) continue;
|
||||
const workspaceFinalizeStatus =
|
||||
operation.status === "succeeded" ? "succeeded" : "failed";
|
||||
if (workspaceFinalizeStatus === "failed") {
|
||||
const failure = await recordNativeWorkspaceFinalizationFailure(db, row.runId, operation);
|
||||
if (outcome.kind !== "completed" && outcome.kind !== "failed") continue;
|
||||
if (outcome.kind === "failed" || outcome.operation.status !== "succeeded") {
|
||||
const failure = await recordNativeWorkspaceFinalizationFailure(db, row.runId, outcome.operation);
|
||||
results.push({
|
||||
...failure,
|
||||
reconciliationAction: "resume_workspace_operation" as const,
|
||||
workspaceOperationId: operation.id,
|
||||
workspaceFinalizeStatus,
|
||||
...(outcome.kind === "completed" ? { workspaceOperationId: outcome.operation.id } : {}),
|
||||
workspaceFinalizeStatus: "failed" as const,
|
||||
reconciliationDecision: recoveryDecision,
|
||||
});
|
||||
if (failure.phase === "terminal_failure") {
|
||||
@@ -685,6 +683,8 @@ async function reconcileNativeFinalizationsPass(
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const { operation } = outcome;
|
||||
const workspaceFinalizeStatus = "succeeded" as const;
|
||||
if (options.skipNativeResultCommit && row.coordinatorPhase !== "committed") continue;
|
||||
try {
|
||||
const finalized = await finalizeNativeRun({
|
||||
|
||||
@@ -8,9 +8,10 @@ export async function inspectPluginWorkspaceForFinalization(
|
||||
workspace: typeof executionWorkspaces.$inferSelect,
|
||||
agentId: string,
|
||||
runId: string,
|
||||
placement?: Pick<Parameters<typeof finalizePluginWorkspace>[3], "taskLease" | "plan">,
|
||||
): Promise<Record<string, unknown> | null> {
|
||||
if (workspace.providerType === "plugin") {
|
||||
const inspection = await finalizePluginWorkspace(db, workers, workspace, { agentId, runId });
|
||||
const inspection = await finalizePluginWorkspace(db, workers, workspace, { agentId, runId, ...placement });
|
||||
if (workspace.strategyType === "git_worktree"
|
||||
&& (!workspace.branchName || inspection.branchName !== workspace.branchName)) {
|
||||
throw new Error("plugin_workspace_branch_mismatch");
|
||||
|
||||
@@ -92,7 +92,7 @@ export function nativeWorkspaceFinalizationWork(db: Db, options: {
|
||||
}
|
||||
return null;
|
||||
}
|
||||
if (outcome.kind !== "completed" || outcome.operation.status === "succeeded") return null;
|
||||
if (outcome.kind === "inactive" || (outcome.kind === "completed" && outcome.operation.status === "succeeded")) return null;
|
||||
const failure = await recordNativeWorkspaceFinalizationFailure(db, current.id, outcome.operation);
|
||||
if (failure.phase === "terminal_failure") {
|
||||
await options.onWorkspaceSettled({ runId: row.id, companyId: row.companyId, agentId: row.agentId, succeeded: false });
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
import { armCrashPointForTest } from "@paperclipai/shared/crash-points";
|
||||
import { DELIVERY_QUEUES, subscribeDeliveryWork } from "../delivery-work-notifications.js";
|
||||
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
||||
import * as pluginWorkspaces from "../plugin-workspaces.js";
|
||||
@@ -23,6 +24,8 @@ import {
|
||||
companies,
|
||||
completionContracts,
|
||||
createDb,
|
||||
environmentLeases,
|
||||
environments,
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
issues,
|
||||
@@ -30,6 +33,7 @@ import {
|
||||
nativeRunResults,
|
||||
issueRecoveryActions,
|
||||
issueThreadInteractions,
|
||||
plugins,
|
||||
projects,
|
||||
workspaceOperations,
|
||||
workAssessments,
|
||||
@@ -37,6 +41,7 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import { startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
||||
import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js";
|
||||
import { nativeWorkspaceFinalizationWork } from "./native-workspace-finalization-work.js";
|
||||
import { reconcileNativeFinalizations } from "./native-finalization-reconciler.js";
|
||||
|
||||
describe("native workspace finalization recovery", () => {
|
||||
@@ -44,7 +49,6 @@ describe("native workspace finalization recovery", () => {
|
||||
let connectionString: string;
|
||||
let db: ReturnType<typeof createDb>;
|
||||
let workspaceRoot: string;
|
||||
let priorLogRoot: string | undefined;
|
||||
|
||||
const companyId = randomUUID();
|
||||
const foreignCompanyId = randomUUID();
|
||||
@@ -55,10 +59,11 @@ describe("native workspace finalization recovery", () => {
|
||||
async function seedRun(input: {
|
||||
executionWorkspaceId: string;
|
||||
title: string;
|
||||
runId?: string;
|
||||
}) {
|
||||
const issueId = randomUUID();
|
||||
const contractId = randomUUID();
|
||||
const runId = randomUUID();
|
||||
const runId = input.runId ?? randomUUID();
|
||||
const resultId = randomUUID();
|
||||
|
||||
await db.insert(issues).values({
|
||||
@@ -123,11 +128,7 @@ describe("native workspace finalization recovery", () => {
|
||||
workspaceRoot = await fs.mkdtemp(
|
||||
path.join(os.tmpdir(), "paperclip-native-workspace-finalizer-"),
|
||||
);
|
||||
priorLogRoot = process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = path.join(
|
||||
workspaceRoot,
|
||||
"operation-logs",
|
||||
);
|
||||
vi.stubEnv("WORKSPACE_OPERATION_LOG_BASE_PATH", path.join(workspaceRoot, "operation-logs"));
|
||||
const external = resolveExternalTestDatabaseUrl("PAPERCLIP_NATIVE_WORKSPACE_FINALIZER_TEST_DATABASE_URL");
|
||||
if (!external) temporary = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-finalizer-");
|
||||
connectionString = external ?? temporary!.connectionString;
|
||||
@@ -168,11 +169,7 @@ describe("native workspace finalization recovery", () => {
|
||||
await db?.$client.end({ timeout: 1 });
|
||||
await temporary?.cleanup();
|
||||
await fs.rm(workspaceRoot, { recursive: true, force: true });
|
||||
if (priorLogRoot === undefined) {
|
||||
delete process.env.WORKSPACE_OPERATION_LOG_BASE_PATH;
|
||||
} else {
|
||||
process.env.WORKSPACE_OPERATION_LOG_BASE_PATH = priorLogRoot;
|
||||
}
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
async function seedAbandonedCopyback() {
|
||||
@@ -752,14 +749,70 @@ describe("native workspace finalization recovery", () => {
|
||||
mode: "isolated_workspace", strategyType: providerType === "plugin" ? "git_worktree" : "project_primary",
|
||||
branchName: "task-branch", metadata: providerType === "plugin" ? {} : { runnerTaskWorkspaceOwner: { agentId } } }).returning();
|
||||
const seeded = await seedRun({ executionWorkspaceId: workspace.id, title: "Plugin result before controller restart" });
|
||||
const plan = vi.spyOn(pluginWorkspaces, "resolvePluginWorkspaceFinalization").mockResolvedValue({ kind: "inspect" });
|
||||
const inspect = vi.spyOn(pluginWorkspaces, "finalizePluginWorkspace").mockResolvedValue({ kind: "inspection",
|
||||
branchName: "task-branch", headSha: "a".repeat(40), hasDirtyTrackedFiles: true, hasUntrackedFiles: false,
|
||||
hasLocalOnlyData: true, isMergedIntoBase: false, remotelyRecoverable: false });
|
||||
try {
|
||||
expect(await resumeNativeWorkspaceFinalization({ db, runId: seeded.runId })).toMatchObject({ status: "succeeded" });
|
||||
if (providerType === "plugin") expect(inspect).toHaveBeenCalledWith(db, undefined, expect.objectContaining({ id: workspace.id }), { agentId, runId: seeded.runId });
|
||||
if (providerType === "plugin") expect(inspect).toHaveBeenCalledWith(db, undefined,
|
||||
expect.objectContaining({ id: workspace.id }), expect.objectContaining({ agentId, runId: seeded.runId }));
|
||||
else expect(inspect).not.toHaveBeenCalled();
|
||||
} finally { inspect.mockRestore(); }
|
||||
} finally { inspect.mockRestore(); plan.mockRestore(); }
|
||||
});
|
||||
|
||||
it.each(["pre-placement request", "crash before dispatch"] as const)("replays the recorded route after %s", async scenario => {
|
||||
const pluginId = randomUUID(), environmentId = randomUUID(), workspaceId = randomUUID();
|
||||
const binding = { pluginId, projectId, executionWorkspaceId: workspaceId, projectWorkspaceId: randomUUID(),
|
||||
sourceRevision: "fixture-revision", workspaceRef: "provider:pre-placement" };
|
||||
await db.insert(plugins).values({
|
||||
id: pluginId, pluginKey: pluginId, packageName: "test-storage", version: "1.0.0", status: "ready",
|
||||
manifestJson: { capabilities: ["workspaces.manage"], resourceLifecycle: ["project"],
|
||||
workspaceProvider: { finalizationVersion: 1 },
|
||||
environmentDrivers: [{ driverKey: "portable", sessionHandoffVersion: 1 }] } as never,
|
||||
});
|
||||
await db.insert(environments).values({ id: environmentId, name: `Portable tasks ${environmentId}`, driver: "plugin" });
|
||||
const [workspace] = await db.insert(executionWorkspaces).values({
|
||||
id: workspaceId, companyId, projectId, name: "Pre-placement worktree", providerType: "plugin", providerRef: binding.workspaceRef,
|
||||
mode: "isolated_workspace", strategyType: "git_worktree", branchName: "task-branch", metadata: { workspaceProvider: binding },
|
||||
}).returning();
|
||||
const { runId, issueId } = await seedRun({ executionWorkspaceId: workspaceId, title: "Workspace finalization interrupted during upgrade" });
|
||||
await db.insert(environmentLeases).values({
|
||||
companyId, environmentId, executionWorkspaceId: workspaceId, heartbeatRunId: runId, issueId,
|
||||
status: "active", provider: "plugin:fixture:portable", providerLeaseId: "live-task",
|
||||
metadata: { pluginId, agentId, driverKey: "portable", runnerTaskLease: true },
|
||||
});
|
||||
const result = { kind: "finalized", branchName: "task-branch", headSha: "a".repeat(40) };
|
||||
const worker = vi.fn().mockResolvedValue(result);
|
||||
const workers = { getWorker: () => ({ supportedMethods: ["workspaceOperation"] }), call: worker } as unknown as pluginWorkspaces.WorkspaceWorkers;
|
||||
const resume = () => resumeNativeWorkspaceFinalization({ db, runId, pluginWorkerManager: workers });
|
||||
if (scenario === "pre-placement request") {
|
||||
worker.mockRejectedValueOnce(new Error("fixture reply lost"));
|
||||
await expect(pluginWorkspaces.callWorkspaceProvider(db, workers, {
|
||||
workspace, runId, agentId, operation: { kind: "finalize", binding },
|
||||
})).rejects.toThrow();
|
||||
await db.insert(workspaceOperations).values({
|
||||
companyId, executionWorkspaceId: workspaceId, heartbeatRunId: runId, issueId,
|
||||
phase: "workspace_finalize", status: "running", metadata: {},
|
||||
});
|
||||
await expect(resume()).resolves.toMatchObject({ status: "succeeded" });
|
||||
const requests = worker.mock.calls.map(([, , request]) => request);
|
||||
expect(requests).toHaveLength(2);
|
||||
expect(requests[1]?.operationId).toBe(requests[0]?.operationId);
|
||||
expect(requests.every(request => request.taskLease === undefined)).toBe(true);
|
||||
} else {
|
||||
const disarm = armCrashPointForTest("workspace.finalize.after_start_before_copyback");
|
||||
try { await expect(resume()).rejects.toThrow("workspace.finalize.after_start_before_copyback"); }
|
||||
finally { disarm(); }
|
||||
expect(worker).not.toHaveBeenCalled();
|
||||
const taskLease = { environmentId, driverKey: "portable", providerLeaseId: "live-task" };
|
||||
const [interrupted] = await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, runId));
|
||||
expect(interrupted).toMatchObject({ status: "running", metadata: { pluginWorkspaceTaskLease: taskLease } });
|
||||
await db.update(environmentLeases).set({ status: "released" }).where(eq(environmentLeases.heartbeatRunId, runId));
|
||||
await expect(resume()).resolves.toMatchObject({ status: "succeeded", metadata: { pluginWorkspaceTaskLease: taskLease } });
|
||||
expect(worker).toHaveBeenCalledExactlyOnceWith(pluginId, "workspaceOperation",
|
||||
expect.objectContaining({ taskLease, runId, agentId, operation: { kind: "finalize", binding } }), 40_000);
|
||||
}
|
||||
});
|
||||
|
||||
it.each(["synthetic plugin unavailable", "plugin_workspace_branch_mismatch", "plugin_workspace_owner_mismatch"])("defers plugin failure without aborting other result recovery: %s", async message => {
|
||||
@@ -771,6 +824,7 @@ describe("native workspace finalization recovery", () => {
|
||||
]).returning();
|
||||
const blocked = await seedRun({ executionWorkspaceId: unavailable.id, title: "Plugin is unavailable after restart" });
|
||||
const ready = await seedRun({ executionWorkspaceId: healthy.id, title: "Other result can recover" });
|
||||
const plan = vi.spyOn(pluginWorkspaces, "resolvePluginWorkspaceFinalization").mockResolvedValue({ kind: "inspect" });
|
||||
const inspect = vi.spyOn(pluginWorkspaces, "finalizePluginWorkspace").mockRejectedValue(new Error(message));
|
||||
const code = message.startsWith("plugin_workspace_") ? message : "plugin_workspace_inspection_failed";
|
||||
try {
|
||||
@@ -783,7 +837,43 @@ describe("native workspace finalization recovery", () => {
|
||||
.toMatchObject([{ status: "failed", stderrExcerpt: `${code}\n` }]);
|
||||
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, ready.runId)))
|
||||
.toMatchObject([{ status: "succeeded" }]);
|
||||
} finally { inspect.mockRestore(); }
|
||||
} finally { inspect.mockRestore(); plan.mockRestore(); }
|
||||
});
|
||||
|
||||
it("accounts for a removed workspace plugin without blocking other accepted runs", async () => {
|
||||
const workspaceId = randomUUID();
|
||||
const binding = { pluginId: randomUUID(), projectId, executionWorkspaceId: workspaceId, projectWorkspaceId: randomUUID(),
|
||||
sourceRevision: "fixture-revision", workspaceRef: "provider:removed-plugin" };
|
||||
const [unavailable, healthy] = await db.insert(executionWorkspaces).values([
|
||||
{ id: workspaceId, companyId, projectId, name: "Removed plugin", providerType: "plugin", providerRef: binding.workspaceRef,
|
||||
mode: "isolated_workspace", strategyType: "git_worktree", branchName: "task-branch", metadata: { workspaceProvider: binding } },
|
||||
{ companyId, projectId, name: "Healthy task", providerType: "adapter_managed", cwd: "/workspaces/healthy-task",
|
||||
mode: "shared_workspace", strategyType: "project_primary", metadata: { runnerTaskWorkspaceOwner: { agentId } } },
|
||||
]).returning();
|
||||
const blocked = await seedRun({ executionWorkspaceId: unavailable.id, title: "Removed plugin", runId: `0${randomUUID().slice(1)}` });
|
||||
const ready = await seedRun({ executionWorkspaceId: healthy.id, title: "Healthy finalization", runId: `f${randomUUID().slice(1)}` });
|
||||
await expect(reconcileNativeFinalizations(db, [blocked.runId, ready.runId])).resolves.toBeDefined();
|
||||
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, ready.runId)))
|
||||
.toMatchObject([{ status: "succeeded" }]);
|
||||
const coordinator = () => db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, blocked.runId))
|
||||
.then(([row]) => row);
|
||||
expect(await coordinator()).toMatchObject({
|
||||
phase: "retryable_failure", nextAttemptAt: expect.any(Date),
|
||||
failureCode: "native_workspace_sync_out_failed", failureDetail: { workspaceFinalizeAttempt: 1 },
|
||||
});
|
||||
|
||||
const onWorkspaceSettled = vi.fn(async () => {});
|
||||
const work = nativeWorkspaceFinalizationWork(db, { onWorkspaceSettled });
|
||||
for (const attempt of [2, 3]) {
|
||||
await db.update(nativeRunFinalizations).set({ nextAttemptAt: null }).where(eq(nativeRunFinalizations.runId, blocked.runId));
|
||||
const [entry] = await work.load({ companyId, runId: blocked.runId });
|
||||
expect(await work.check(entry)).toEqual(attempt === 2 ? expect.objectContaining({ id: blocked.runId }) : null);
|
||||
expect(await coordinator()).toMatchObject({
|
||||
phase: attempt === 2 ? "retryable_failure" : "terminal_failure", failureDetail: { workspaceFinalizeAttempt: attempt },
|
||||
});
|
||||
}
|
||||
expect(onWorkspaceSettled).toHaveBeenCalledExactlyOnceWith({ runId: blocked.runId, companyId, agentId, succeeded: false });
|
||||
expect(await db.select().from(workspaceOperations).where(eq(workspaceOperations.heartbeatRunId, blocked.runId))).toHaveLength(0);
|
||||
});
|
||||
|
||||
it("records directory-only recovery without inventing an execution-workspace foreign key", async () => {
|
||||
|
||||
@@ -3,7 +3,7 @@ import { readRemoteWorkspaceAuthority, saveRemoteWorkspace } from "../remote-wor
|
||||
import { remoteWorkspaceFailureCause } from "../remote-workspace-transport.js";
|
||||
import { recordNativeFinalizationFailure } from "./native-run-finalizer.js";
|
||||
import { inspectPluginWorkspaceForFinalization } from "./native-plugin-workspace-finalization.js";
|
||||
import type { WorkspaceWorkers } from "../plugin-workspaces.js";
|
||||
import { resolvePluginWorkspaceFinalization, type WorkspaceWorkers } from "../plugin-workspaces.js";
|
||||
import { restoreNativeWorkspaceBestEffort } from "./native-workspace-best-effort.js";
|
||||
import { withNativeWorkspaceFinalizationOwnership, type NativeWorkspaceFinalizationOwnership } from "./native-workspace-finalization-ownership.js";
|
||||
import { settleSupersededNativeWorkspaceFinalizations } from "./native-workspace-finalization-supersession.js";
|
||||
@@ -11,6 +11,7 @@ import { logger } from "../../middleware/logger.js";
|
||||
import fs from "node:fs/promises";
|
||||
import { and, desc, eq } from "drizzle-orm";
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { pluginWorkspaceParamsSchema } from "@paperclipai/plugin-sdk";
|
||||
import {
|
||||
executionWorkspaces,
|
||||
heartbeatRuns,
|
||||
@@ -157,6 +158,21 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
)).limit(1).then((rows) => rows[0] ?? null)
|
||||
: null;
|
||||
const cwd = workspace?.providerRef ?? workspace?.cwd ?? previous?.cwd ?? null;
|
||||
let pluginFinalizationPlan: Awaited<ReturnType<typeof resolvePluginWorkspaceFinalization>>;
|
||||
if (workspace && !nativeWorkspaceSync && !readRemoteWorkspaceAuthority(workspace.metadata)) {
|
||||
try {
|
||||
const taskLease = previous
|
||||
? pluginWorkspaceParamsSchema.shape.taskLease.parse(previous.metadata?.pluginWorkspaceTaskLease) ?? null
|
||||
: undefined;
|
||||
pluginFinalizationPlan = await resolvePluginWorkspaceFinalization(input.db, workspace, {
|
||||
runId: input.runId, agentId: bound.agentId, taskLease,
|
||||
});
|
||||
} catch {
|
||||
await ownership.assertHeld();
|
||||
return { kind: "failed", operation: { stderrExcerpt: "plugin_workspace_inspection_failed" } } as const;
|
||||
}
|
||||
}
|
||||
const recordedTaskLease = pluginFinalizationPlan?.kind === "finalize" ? pluginFinalizationPlan.taskLease : undefined;
|
||||
|
||||
const recorder = workspaceOperationService(input.db).createRecorder({
|
||||
companyId: bound.companyId,
|
||||
@@ -173,6 +189,7 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
metadata: {
|
||||
resumedFromOperationId: previous?.id ?? null,
|
||||
owningService: "native_workspace_finalizer",
|
||||
...(recordedTaskLease ? { pluginWorkspaceTaskLease: recordedTaskLease } : {}),
|
||||
observation: workspace?.strategyType === "git_worktree"
|
||||
? "managed_git_worktree_branch"
|
||||
: "workspace_directory",
|
||||
@@ -307,7 +324,10 @@ export async function attemptNativeWorkspaceFinalization(input: {
|
||||
let pluginWorkspace: Record<string, unknown> | null;
|
||||
try {
|
||||
pluginWorkspace = workspace
|
||||
? await inspectPluginWorkspaceForFinalization(input.db, input.pluginWorkerManager, workspace, bound.agentId, input.runId)
|
||||
? await inspectPluginWorkspaceForFinalization(input.db, input.pluginWorkerManager, workspace, bound.agentId, input.runId, {
|
||||
taskLease: null,
|
||||
plan: pluginFinalizationPlan,
|
||||
})
|
||||
: null;
|
||||
} catch (error) {
|
||||
await ownership.assertHeld();
|
||||
|
||||
Reference in new issue
Block a user