mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-27 02:44:50 +01:00
fix(runner): persist warm Daytona workspaces (#12901)
## Thinking Path > - Paperclip manages AI agent work and the execution state for each task. > - Remote agents run in sandbox environments such as Daytona. > - Daytona keeps files while a sandbox is stopped, but deletion removes those files. > - Runner Codex did not copy successful remote workspace changes back to the host workspace. > - A warm sandbox could therefore hide data loss until Daytona replaced or deleted the sandbox. > - This pull request makes the host workspace durable after every successful turn and keeps verified reusable sandboxes warm. > - The benefit is reliable multi-turn work across warm reuse, restart, stop, and sandbox replacement. ## Linked Issues or Issue Description **What happened?** A successful native Codex turn in Daytona could leave workspace changes only in the remote sandbox. A later warm turn appeared to work because it reused that filesystem. A replacement sandbox could start from stale host data and lose the successful changes. **Expected behavior** Paperclip must merge each successful remote turn into the authoritative host workspace before it completes the run. A verified warm lease may reuse its remote files. A replacement lease must reconstruct the exact durable workspace seed. **Steps to reproduce** 1. Run Codex in a reusable Daytona environment. 2. Write a file during one successful turn. 3. Replace the Daytona sandbox before the next turn. 4. Observe that the next turn can start without the prior file on the unpatched code. Related remote workspace foundation: #10070. ## What Changed - Added explicit `host_current`, `durable_seed`, and `adopt_remote` workspace preparation modes. - Added atomic, versioned native workspace descriptors and seed archives under `PAPERCLIP_HOME`. - Added real native sandbox export and three-way host merge before terminal result completion. - Added workspace-only recovery after a proposed result. Recovery does not submit another provider turn or consume the provider retry budget. - Added fail-closed handling when a sandbox with unexported changes is gone. - Kept healthy reusable Daytona sandboxes started for legacy Codex and Runner Codex. - Kept the Runner Codex process and provider session across verified warm turns. - Added the paid `daytona-warm-continuity` browser suite. It contains exactly the legacy Codex and Runner Codex cells. Each cell performs three measured turns. - Documented `pnpm test:e2e:runner -- --suite daytona-warm-continuity`. No package script was added. - Added no database migration. The metadata format is backward compatible and idempotent. ## Verification - `pnpm typecheck` - `pnpm test:e2e:runner:unit` — 114 passed - Native workspace, finalizer, session, and environment tests — 232 passed - Daytona provider tests — 150 passed - Workspace staging and merge tests — 98 passed - Runner transport tests — 63 passed - Legacy Codex restore tests — 5 passed - Rust format and compile checks pass through root typecheck - The paid Daytona suite was not run locally because the required Daytona, OpenAI, and immutable image credentials are not present. ## Risks - The main risk is an incorrect workspace identity or merge after a crash. Durable descriptors bind the run, workspace, lease, provider lease, local root, remote root, and baseline digest. Ambiguous evidence fails closed. - The host merge may conflict with concurrent host edits. The existing three-way merge and exclusion rules handle this case and surface failures. - A deleted sandbox cannot recover unexported bytes. Paperclip now blocks with `workspace_sync_out_unrecoverable` instead of reporting success or rerunning the provider. - There is no database migration. Descriptor writes and recovery are atomic and idempotent. ## Model Used OpenAI Codex with GPT-5. The run used agentic reasoning, repository inspection, code execution, test execution, Git, and GitHub CLI tools. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge
This commit is contained in:
Dotta authored and GitHub committed
1 parent
be407f3456
commit
1dceee9a4e
47 files changed
+4296
-359
No files matched your search
@@ -9,4 +9,5 @@ export * from "./paperclip-control-plane-port.js";
|
||||
export * from "./native-run-finalizer.js";
|
||||
export * from "./native-finalization-reconciler.js";
|
||||
export * from "./native-restart-recovery.js";
|
||||
export * from "./native-workspace-sync.js";
|
||||
export * from "./status-arbiter.js";
|
||||
@@ -24,6 +24,11 @@ import { issueRecoveryActionService } from "../issue-recovery-actions.js";
|
||||
import { issueService } from "../issues.js";
|
||||
import { emitAgentTaskRun } from "../agent-task-run-telemetry.js";
|
||||
import { resumeNativeWorkspaceFinalization } from "./native-workspace-finalizer.js";
|
||||
import {
|
||||
cleanupNativeWorkspaceSync,
|
||||
readNativeWorkspaceSyncReference,
|
||||
} from "./native-workspace-sync.js";
|
||||
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
|
||||
import { classifyNativeEvidence } from "./evidence-classifier.js";
|
||||
import { recordNativeWorkAssessment } from "./work-assessments.js";
|
||||
import {
|
||||
@@ -373,19 +378,33 @@ export async function claimNativeSessionResumptions(input: {
|
||||
}
|
||||
|
||||
/** Recovery is keyed only by persisted mode/coordinator state, never the live flag. */
|
||||
export async function reconcileNativeFinalizations(db: Db, runIds?: string[]) {
|
||||
const rows = await db.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
issueId: nativeRunFinalizations.issueId,
|
||||
issueStatus: issues.status,
|
||||
issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId,
|
||||
coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId,
|
||||
decisionId: nativeRunFinalizations.decisionId,
|
||||
})
|
||||
export async function reconcileNativeFinalizations(
|
||||
db: Db,
|
||||
runIds?: string[],
|
||||
options: {
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
onWorkspaceSettled?: (input: {
|
||||
runId: string;
|
||||
companyId: string;
|
||||
agentId: string;
|
||||
succeeded: boolean;
|
||||
}) => Promise<void>;
|
||||
} = {},
|
||||
) {
|
||||
const rows = await db
|
||||
.select({
|
||||
runId: heartbeatRuns.id,
|
||||
companyId: heartbeatRuns.companyId,
|
||||
agentId: heartbeatRuns.agentId,
|
||||
issueId: nativeRunFinalizations.issueId,
|
||||
issueStatus: issues.status,
|
||||
issueStatusVersion: issues.statusVersion,
|
||||
issueDecisionId: issues.lastStatusDecisionId,
|
||||
coordinatorPhase: nativeRunFinalizations.phase,
|
||||
assessmentId: nativeRunFinalizations.assessmentId,
|
||||
decisionId: nativeRunFinalizations.decisionId,
|
||||
runnerProfileJson: heartbeatRuns.runnerProfileJson,
|
||||
})
|
||||
.from(heartbeatRuns)
|
||||
.innerJoin(nativeRunFinalizations, eq(nativeRunFinalizations.runId, heartbeatRuns.id))
|
||||
.innerJoin(issues, and(
|
||||
@@ -617,14 +636,59 @@ export async function reconcileNativeFinalizations(db: Db, runIds?: string[]) {
|
||||
if (!recoveryDecision.effects.some((effect) => effect.kind === "resume_workspace_operation")) {
|
||||
throw new Error("native_reconciliation_workspace_resume_policy_missing");
|
||||
}
|
||||
const operation = await resumeNativeWorkspaceFinalization({ db, runId: row.runId });
|
||||
const workspaceFinalizeStatus = operation.status === "succeeded" ? "succeeded" : "failed";
|
||||
const operation = await resumeNativeWorkspaceFinalization({
|
||||
db,
|
||||
runId: row.runId,
|
||||
environmentRuntime: options.environmentRuntime,
|
||||
});
|
||||
const workspaceFinalizeStatus =
|
||||
operation.status === "succeeded" ? "succeeded" : "failed";
|
||||
if (workspaceFinalizeStatus === "failed") {
|
||||
const unrecoverable = operation.stderrExcerpt?.includes(
|
||||
"workspace_sync_out_unrecoverable",
|
||||
);
|
||||
const failure = await recordNativeFinalizationFailure({
|
||||
db,
|
||||
runId: row.runId,
|
||||
error: new Error(
|
||||
unrecoverable
|
||||
? "native_workspace_sync_out_unrecoverable"
|
||||
: "native_workspace_sync_out_failed",
|
||||
),
|
||||
projectRunStatus: true,
|
||||
failureScope: "workspace",
|
||||
permanent: unrecoverable,
|
||||
});
|
||||
results.push({
|
||||
...failure,
|
||||
reconciliationAction: "resume_workspace_operation" as const,
|
||||
workspaceOperationId: operation.id,
|
||||
workspaceFinalizeStatus,
|
||||
reconciliationDecision: recoveryDecision,
|
||||
});
|
||||
if (failure.phase === "terminal_failure") {
|
||||
await options
|
||||
.onWorkspaceSettled?.({
|
||||
runId: row.runId,
|
||||
companyId: row.companyId,
|
||||
agentId: row.agentId,
|
||||
succeeded: false,
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
try {
|
||||
const finalized = await finalizeNativeRun({
|
||||
db,
|
||||
runId: row.runId,
|
||||
workspaceFinalizeStatus,
|
||||
projectRunStatus: true,
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
record(row.runnerProfileJson).nativeWorkspaceSync,
|
||||
),
|
||||
),
|
||||
});
|
||||
results.push({
|
||||
...finalized,
|
||||
@@ -633,6 +697,20 @@ export async function reconcileNativeFinalizations(db: Db, runIds?: string[]) {
|
||||
workspaceFinalizeStatus,
|
||||
reconciliationDecision: recoveryDecision,
|
||||
});
|
||||
if (
|
||||
workspaceFinalizeStatus === "succeeded" &&
|
||||
finalized.phase === "committed"
|
||||
) {
|
||||
await cleanupNativeWorkspaceSync(row.runId).catch(() => undefined);
|
||||
await options
|
||||
.onWorkspaceSettled?.({
|
||||
runId: row.runId,
|
||||
companyId: row.companyId,
|
||||
agentId: row.agentId,
|
||||
succeeded: true,
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
} catch (error) {
|
||||
const failure = await recordNativeFinalizationFailure({
|
||||
db,
|
||||
@@ -659,12 +737,29 @@ export async function reconcileNativeFinalizations(db: Db, runIds?: string[]) {
|
||||
throw new Error("native_reconciliation_replay_policy_invalid");
|
||||
}
|
||||
try {
|
||||
results.push(await finalizeNativeRun({
|
||||
const finalized = await finalizeNativeRun({
|
||||
db,
|
||||
runId: row.runId,
|
||||
workspaceFinalizeStatus: barrier.status as "succeeded" | "failed",
|
||||
projectRunStatus: true,
|
||||
}));
|
||||
preserveProviderAttempt: Boolean(
|
||||
readNativeWorkspaceSyncReference(
|
||||
record(row.runnerProfileJson).nativeWorkspaceSync,
|
||||
),
|
||||
),
|
||||
});
|
||||
results.push(finalized);
|
||||
if (finalized.phase === "committed") {
|
||||
await cleanupNativeWorkspaceSync(row.runId).catch(() => undefined);
|
||||
await options
|
||||
.onWorkspaceSettled?.({
|
||||
runId: row.runId,
|
||||
companyId: row.companyId,
|
||||
agentId: row.agentId,
|
||||
succeeded: true,
|
||||
})
|
||||
.catch(() => undefined);
|
||||
}
|
||||
} catch (error) {
|
||||
results.push(await recordNativeFinalizationFailure({
|
||||
db,
|
||||
|
||||
@@ -279,6 +279,93 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a
|
||||
expect(run?.status).toBe("running");
|
||||
});
|
||||
|
||||
it("bounds workspace-only retries without consuming the provider attempt", async () => {
|
||||
const fixture = await seedNativeRun();
|
||||
await db.insert(nativeRunFinalizations).values({
|
||||
runId: fixture.runId,
|
||||
companyId,
|
||||
issueId: fixture.issueId,
|
||||
phase: "observed",
|
||||
attempt: 1,
|
||||
});
|
||||
|
||||
for (
|
||||
let expectedAttempt = 1;
|
||||
expectedAttempt <= 3;
|
||||
expectedAttempt += 1
|
||||
) {
|
||||
await recordNativeFinalizationFailure({
|
||||
db,
|
||||
runId: fixture.runId,
|
||||
error: new Error("native_workspace_sync_out_failed"),
|
||||
projectRunStatus: true,
|
||||
failureScope: "workspace",
|
||||
});
|
||||
const coordinator = await db
|
||||
.select()
|
||||
.from(nativeRunFinalizations)
|
||||
.where(eq(nativeRunFinalizations.runId, fixture.runId))
|
||||
.then((rows) => rows[0]!);
|
||||
expect(coordinator.attempt).toBe(1);
|
||||
expect(coordinator.failureDetail).toMatchObject({
|
||||
workspaceFinalizeAttempt: expectedAttempt,
|
||||
});
|
||||
expect(coordinator.phase).toBe(
|
||||
expectedAttempt === 3 ? "terminal_failure" : "retryable_failure",
|
||||
);
|
||||
}
|
||||
|
||||
await expect(
|
||||
db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, fixture.runId))
|
||||
.then((rows) => rows[0]?.status),
|
||||
).resolves.toBe("failed");
|
||||
});
|
||||
|
||||
it("blocks immediately when the sandbox with unexported changes is gone", async () => {
|
||||
const fixture = await seedNativeRun();
|
||||
await db.insert(nativeRunFinalizations).values({
|
||||
runId: fixture.runId,
|
||||
companyId,
|
||||
issueId: fixture.issueId,
|
||||
phase: "workspace_finalizing",
|
||||
attempt: 1,
|
||||
resultId: null,
|
||||
});
|
||||
|
||||
const failure = await recordNativeFinalizationFailure({
|
||||
db,
|
||||
runId: fixture.runId,
|
||||
error: new Error("native_workspace_sync_out_unrecoverable"),
|
||||
projectRunStatus: true,
|
||||
failureScope: "workspace",
|
||||
permanent: true,
|
||||
});
|
||||
|
||||
expect(failure).toMatchObject({
|
||||
phase: "terminal_failure",
|
||||
failureCode: "native_workspace_sync_out_unrecoverable",
|
||||
nextAttemptAt: null,
|
||||
attempt: 1,
|
||||
});
|
||||
await expect(
|
||||
db
|
||||
.select({ status: heartbeatRuns.status })
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, fixture.runId))
|
||||
.then((rows) => rows[0]?.status),
|
||||
).resolves.toBe("failed");
|
||||
await expect(
|
||||
db
|
||||
.select({ status: issues.status })
|
||||
.from(issues)
|
||||
.where(eq(issues.id, fixture.issueId))
|
||||
.then((rows) => rows[0]?.status),
|
||||
).resolves.toBe("blocked");
|
||||
});
|
||||
|
||||
it("emits exactly one event for a cancel_continuations write (trap 2: :685/:518 overlap)", async () => {
|
||||
const fixture = await seedNativeRun();
|
||||
// Build the minimal real rows commitNativeStatusDecision's foreign keys
|
||||
@@ -329,7 +416,10 @@ describeEmbeddedPostgres("native run finalizer / status decision committer — a
|
||||
toStatus: "cancelled",
|
||||
reasonCode: "cancellation_issue_authorized",
|
||||
unblockDescriptor: null,
|
||||
effects: [{ kind: "release_checkout" }, { kind: "cancel_continuations" }],
|
||||
effects: [
|
||||
{ kind: "release_checkout" },
|
||||
{ kind: "cancel_continuations" },
|
||||
],
|
||||
};
|
||||
|
||||
const callsBefore = mockTelemetryClient.track.mock.calls.length;
|
||||
|
||||
@@ -126,7 +126,11 @@ async function acceptedInteractionFromRun(input: {
|
||||
.then((rows) => rows[0] ?? null);
|
||||
}
|
||||
|
||||
async function claimCoordinator(input: { db: Db; runId: string }) {
|
||||
async function claimCoordinator(input: {
|
||||
db: Db;
|
||||
runId: string;
|
||||
preserveProviderAttempt?: boolean;
|
||||
}) {
|
||||
const leaseOwner = `native-finalizer:${randomUUID()}`;
|
||||
const now = new Date();
|
||||
const claimed = await input.db.transaction(async (tx) => {
|
||||
@@ -152,23 +156,33 @@ async function claimCoordinator(input: { db: Db; runId: string }) {
|
||||
}
|
||||
if (coordinator.phase === "terminal_failure") throw new Error("native_finalization_terminal_failure");
|
||||
if (
|
||||
coordinator.leaseOwner
|
||||
&& coordinator.leaseExpiresAt
|
||||
&& coordinator.leaseExpiresAt > now
|
||||
&& coordinator.leaseOwner !== leaseOwner
|
||||
) throw new Error("native_finalization_lease_busy");
|
||||
const [updated] = await tx.update(nativeRunFinalizations).set({
|
||||
leaseOwner,
|
||||
leaseExpiresAt: new Date(now.getTime() + 5 * 60_000),
|
||||
attempt: coordinator.attempt + 1,
|
||||
phase: coordinator.phase === "retryable_failure"
|
||||
? coordinator.assessmentId ? "arbitrating" : "workspace_finalizing"
|
||||
: coordinator.phase,
|
||||
failureCode: null,
|
||||
failureDetail: null,
|
||||
nextAttemptAt: null,
|
||||
updatedAt: now,
|
||||
}).where(eq(nativeRunFinalizations.runId, input.runId)).returning();
|
||||
coordinator.leaseOwner &&
|
||||
coordinator.leaseExpiresAt &&
|
||||
coordinator.leaseExpiresAt > now &&
|
||||
coordinator.leaseOwner !== leaseOwner
|
||||
)
|
||||
throw new Error("native_finalization_lease_busy");
|
||||
const [updated] = await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
leaseOwner,
|
||||
leaseExpiresAt: new Date(now.getTime() + 5 * 60_000),
|
||||
attempt: input.preserveProviderAttempt
|
||||
? coordinator.attempt
|
||||
: coordinator.attempt + 1,
|
||||
phase:
|
||||
coordinator.phase === "retryable_failure"
|
||||
? coordinator.assessmentId
|
||||
? "arbitrating"
|
||||
: "workspace_finalizing"
|
||||
: coordinator.phase,
|
||||
failureCode: null,
|
||||
failureDetail: null,
|
||||
nextAttemptAt: null,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(eq(nativeRunFinalizations.runId, input.runId))
|
||||
.returning();
|
||||
if (!updated) throw new Error("native_finalization_claim_failed");
|
||||
return { coordinator: updated, leaseOwner };
|
||||
});
|
||||
@@ -183,6 +197,8 @@ async function recordRetryableFailure(input: {
|
||||
message: string;
|
||||
nextAction: string;
|
||||
projectRunStatus?: boolean;
|
||||
failureScope?: "provider" | "workspace";
|
||||
permanent?: boolean;
|
||||
}) {
|
||||
const now = new Date();
|
||||
const nextAttemptAt = new Date(now.getTime() + 30_000);
|
||||
@@ -228,50 +244,87 @@ async function recordRetryableFailure(input: {
|
||||
const supersededByNewerRun = Boolean(
|
||||
latestDecisionRun && latestDecisionRun.createdAt > input.run.createdAt,
|
||||
);
|
||||
const exhausted = input.coordinator.attempt >= 3;
|
||||
const phase = supersededByNewerRun || exhausted
|
||||
? "terminal_failure" as const
|
||||
: "retryable_failure" as const;
|
||||
const priorFailureDetail = record(input.coordinator.failureDetail);
|
||||
const workspaceFinalizeAttempt =
|
||||
input.failureScope === "workspace"
|
||||
? (typeof priorFailureDetail.workspaceFinalizeAttempt === "number" &&
|
||||
Number.isInteger(priorFailureDetail.workspaceFinalizeAttempt) &&
|
||||
priorFailureDetail.workspaceFinalizeAttempt >= 0
|
||||
? priorFailureDetail.workspaceFinalizeAttempt
|
||||
: 0) + 1
|
||||
: null;
|
||||
const exhausted =
|
||||
input.permanent === true ||
|
||||
(workspaceFinalizeAttempt !== null
|
||||
? workspaceFinalizeAttempt >= 3
|
||||
: input.coordinator.attempt >= 3);
|
||||
const phase =
|
||||
supersededByNewerRun || exhausted
|
||||
? ("terminal_failure" as const)
|
||||
: ("retryable_failure" as const);
|
||||
const failureCode = supersededByNewerRun
|
||||
? "native_finalization_superseded"
|
||||
: exhausted
|
||||
? "native_finalization_retry_exhausted"
|
||||
: input.failureCode;
|
||||
await tx.update(nativeRunFinalizations).set({
|
||||
phase,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
failureCode,
|
||||
failureDetail: {
|
||||
message: input.message.slice(0, 2_000),
|
||||
originalFailureCode: input.failureCode,
|
||||
recoveryOwner: supersededByNewerRun
|
||||
? { kind: "none", reason: "newer_native_decision" }
|
||||
: exhausted
|
||||
? { kind: "board" }
|
||||
: { kind: "agent", agentId: input.run.agentId },
|
||||
nextAction: input.nextAction,
|
||||
},
|
||||
nextAttemptAt: supersededByNewerRun || exhausted ? null : nextAttemptAt,
|
||||
updatedAt: now,
|
||||
}).where(eq(nativeRunFinalizations.runId, input.run.id));
|
||||
const projectsTerminalStatus = exhausted && !supersededByNewerRun && input.projectRunStatus;
|
||||
const [updatedRun] = await tx.update(heartbeatRuns).set({
|
||||
...(projectsTerminalStatus ? {
|
||||
status: exhaustedRunStatus,
|
||||
finishedAt: input.run.finishedAt ?? now,
|
||||
} : {}),
|
||||
nativePhase: phase,
|
||||
nativePhaseUpdatedAt: now,
|
||||
resultJson: {
|
||||
...record(input.run.resultJson),
|
||||
finalizationPhase: phase,
|
||||
: input.permanent
|
||||
? input.failureCode
|
||||
: exhausted
|
||||
? input.failureScope === "workspace"
|
||||
? "native_workspace_sync_out_retry_exhausted"
|
||||
: "native_finalization_retry_exhausted"
|
||||
: input.failureCode;
|
||||
await tx
|
||||
.update(nativeRunFinalizations)
|
||||
.set({
|
||||
phase,
|
||||
leaseOwner: null,
|
||||
leaseExpiresAt: null,
|
||||
failureCode,
|
||||
originalFailureCode: input.failureCode,
|
||||
nextAttemptAt: supersededByNewerRun || exhausted ? null : nextAttemptAt.toISOString(),
|
||||
},
|
||||
updatedAt: now,
|
||||
}).where(eq(heartbeatRuns.id, input.run.id)).returning();
|
||||
failureDetail: {
|
||||
message: input.message.slice(0, 2_000),
|
||||
originalFailureCode: input.failureCode,
|
||||
...(workspaceFinalizeAttempt === null
|
||||
? {}
|
||||
: { workspaceFinalizeAttempt }),
|
||||
recoveryOwner: supersededByNewerRun
|
||||
? { kind: "none", reason: "newer_native_decision" }
|
||||
: exhausted
|
||||
? { kind: "board" }
|
||||
: { kind: "agent", agentId: input.run.agentId },
|
||||
nextAction: input.nextAction,
|
||||
},
|
||||
nextAttemptAt: supersededByNewerRun || exhausted ? null : nextAttemptAt,
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(eq(nativeRunFinalizations.runId, input.run.id));
|
||||
const projectsTerminalStatus =
|
||||
exhausted && !supersededByNewerRun && input.projectRunStatus;
|
||||
const [updatedRun] = await tx
|
||||
.update(heartbeatRuns)
|
||||
.set({
|
||||
...(projectsTerminalStatus
|
||||
? {
|
||||
status:
|
||||
input.failureScope === "workspace"
|
||||
? "failed"
|
||||
: exhaustedRunStatus,
|
||||
finishedAt: input.run.finishedAt ?? now,
|
||||
}
|
||||
: {}),
|
||||
nativePhase: phase,
|
||||
nativePhaseUpdatedAt: now,
|
||||
resultJson: {
|
||||
...record(input.run.resultJson),
|
||||
finalizationPhase: phase,
|
||||
failureCode,
|
||||
originalFailureCode: input.failureCode,
|
||||
nextAttemptAt:
|
||||
supersededByNewerRun || exhausted
|
||||
? null
|
||||
: nextAttemptAt.toISOString(),
|
||||
},
|
||||
updatedAt: now,
|
||||
})
|
||||
.where(eq(heartbeatRuns.id, input.run.id))
|
||||
.returning();
|
||||
if (projectsTerminalStatus) terminalRunToEmit = updatedRun ?? null;
|
||||
if (supersededByNewerRun) {
|
||||
await issueRecoveryActionService(tx as unknown as Db).resolveActiveForIssue({
|
||||
@@ -284,9 +337,16 @@ async function recordRetryableFailure(input: {
|
||||
resolutionNote: "A newer native run already committed the authoritative issue decision; the stale finalizer was retired without changing issue state.",
|
||||
}, tx);
|
||||
} else if (exhausted) {
|
||||
await issueService(tx as unknown as Db).update(input.coordinator.issueId, {
|
||||
status: "in_review",
|
||||
}, tx);
|
||||
await issueService(tx as unknown as Db).update(
|
||||
input.coordinator.issueId,
|
||||
{
|
||||
status:
|
||||
input.permanent && input.failureScope === "workspace"
|
||||
? "blocked"
|
||||
: "in_review",
|
||||
},
|
||||
tx,
|
||||
);
|
||||
}
|
||||
if (!supersededByNewerRun) {
|
||||
await issueRecoveryActionService(tx as unknown as Db).upsertSourceScoped({
|
||||
@@ -301,10 +361,15 @@ async function recordRetryableFailure(input: {
|
||||
evidence: {
|
||||
runId: input.run.id,
|
||||
coordinatorAttempt: input.coordinator.attempt,
|
||||
...(workspaceFinalizeAttempt === null
|
||||
? {}
|
||||
: { workspaceFinalizeAttempt }),
|
||||
originalFailureCode: input.failureCode,
|
||||
},
|
||||
nextAction: exhausted
|
||||
? `Finalization retry budget exhausted. ${input.nextAction}`
|
||||
? input.permanent
|
||||
? input.nextAction
|
||||
: `Finalization retry budget exhausted. ${input.nextAction}`
|
||||
: input.nextAction,
|
||||
wakePolicy: exhausted
|
||||
? null
|
||||
@@ -328,6 +393,8 @@ export async function recordNativeFinalizationFailure(input: {
|
||||
runId: string;
|
||||
error: unknown;
|
||||
projectRunStatus?: boolean;
|
||||
failureScope?: "provider" | "workspace";
|
||||
permanent?: boolean;
|
||||
}) {
|
||||
const [run, coordinator] = await Promise.all([
|
||||
input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))
|
||||
@@ -344,8 +411,15 @@ export async function recordNativeFinalizationFailure(input: {
|
||||
coordinator,
|
||||
failureCode,
|
||||
message,
|
||||
nextAction: "Repair the persisted native result or contract discriminator, then resume finalization from the coordinator.",
|
||||
nextAction:
|
||||
input.failureScope === "workspace"
|
||||
? input.permanent
|
||||
? "Restore the exact sandbox containing the unexported workspace changes, or resolve the task manually from durable evidence."
|
||||
: "Retry workspace export and merge from the retained sandbox; do not submit another provider turn."
|
||||
: "Repair the persisted native result or contract discriminator, then resume finalization from the coordinator.",
|
||||
projectRunStatus: input.projectRunStatus,
|
||||
failureScope: input.failureScope,
|
||||
permanent: input.permanent,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -395,12 +469,23 @@ export async function finalizeNativeRun(input: {
|
||||
workspaceFinalizeStatus: "succeeded" | "failed";
|
||||
/** Reconciliation owns terminal run projection; the live heartbeat does it afterward. */
|
||||
projectRunStatus?: boolean;
|
||||
/** Workspace-only replay must not consume the provider recovery budget. */
|
||||
preserveProviderAttempt?: boolean;
|
||||
failpoint?: NativeStatusCommitFailpoint;
|
||||
}) {
|
||||
const run = await input.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, input.runId))
|
||||
.limit(1).then((rows) => rows[0] ?? null);
|
||||
if (!run || run.runtimeMode !== "native") throw new Error("native_finalization_run_missing");
|
||||
const claim = await claimCoordinator({ db: input.db, runId: input.runId });
|
||||
const run = await input.db
|
||||
.select()
|
||||
.from(heartbeatRuns)
|
||||
.where(eq(heartbeatRuns.id, input.runId))
|
||||
.limit(1)
|
||||
.then((rows) => rows[0] ?? null);
|
||||
if (!run || run.runtimeMode !== "native")
|
||||
throw new Error("native_finalization_run_missing");
|
||||
const claim = await claimCoordinator({
|
||||
db: input.db,
|
||||
runId: input.runId,
|
||||
preserveProviderAttempt: input.preserveProviderAttempt,
|
||||
});
|
||||
const coordinator = claim.coordinator;
|
||||
if (!claim.leaseOwner && coordinator.phase === "committed") {
|
||||
if (input.projectRunStatus) await projectCommittedRun({ db: input.db, run, coordinator });
|
||||
|
||||
@@ -2880,7 +2880,7 @@ describe("native warm session supervision", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("rehydrates a runnerd warm session from its checkpoint under a fresh run authority", async () => {
|
||||
it("reattaches a live runnerd warm session under a fresh run authority", async () => {
|
||||
const stateBase = await mkdtemp(
|
||||
join(tmpdir(), "paperclip-runnerd-warm-authority-"),
|
||||
);
|
||||
@@ -2889,9 +2889,7 @@ describe("native warm session supervision", () => {
|
||||
process.env.PAPERCLIP_RUNNER_STATE_DIR = stateBase;
|
||||
process.env.PAPERCLIP_HOME = stateBase;
|
||||
const firstClose = vi.fn(async () => undefined);
|
||||
const secondClose = vi.fn(async () => undefined);
|
||||
const firstSession = { close: firstClose };
|
||||
const secondSession = { close: secondClose };
|
||||
const first = {
|
||||
...execution,
|
||||
binding: {
|
||||
@@ -2947,18 +2945,8 @@ describe("native warm session supervision", () => {
|
||||
return result;
|
||||
})
|
||||
.mockImplementationOnce(async (options) => {
|
||||
expect(options.existingSession).toBeUndefined();
|
||||
expect(options.persistedSession).toEqual(
|
||||
expect.objectContaining({
|
||||
identity: expect.objectContaining({
|
||||
runId: second.binding.runId,
|
||||
sessionId: second.session.normalizedSessionId,
|
||||
}),
|
||||
providerSessionId: "provider-runnerd-warm",
|
||||
activeTurnId: null,
|
||||
}),
|
||||
);
|
||||
options.onSession?.(secondSession);
|
||||
expect(options.existingSession).toBe(firstSession);
|
||||
expect(options.persistedSession).toBeUndefined();
|
||||
return result;
|
||||
});
|
||||
|
||||
@@ -3014,10 +3002,10 @@ describe("native warm session supervision", () => {
|
||||
runnerInstanceId: "runner-runnerd-warm",
|
||||
useRunnerd: true,
|
||||
});
|
||||
expect(firstClose).toHaveBeenCalledWith({
|
||||
reason: "warm native session authority epoch rotated",
|
||||
});
|
||||
await vi.waitFor(() => expect(secondClose).toHaveBeenCalled(), {
|
||||
expect(firstClose).not.toHaveBeenCalled();
|
||||
await vi.waitFor(() => expect(firstClose).toHaveBeenCalledWith({
|
||||
reason: "warm native session idle timeout",
|
||||
}), {
|
||||
timeout: 500,
|
||||
});
|
||||
} finally {
|
||||
@@ -5392,9 +5380,13 @@ describe("runnerd provider runtime wiring", () => {
|
||||
|
||||
const firstOptions = state.createBackend.mock.calls[0]![1];
|
||||
const continuationOptions = state.createBackend.mock.calls[1]![1];
|
||||
await expect(firstOptions.dynamicToolHandler!({})).rejects.toThrow(
|
||||
"native_tool_authority_epoch_revoked",
|
||||
);
|
||||
// The retained runner backend owns one stable callback. After run.attach,
|
||||
// that callback routes through the session-scope authority registry to
|
||||
// the new run; stale provider calls are rejected earlier by runnerd's
|
||||
// turn identity boundary.
|
||||
await expect(firstOptions.dynamicToolHandler!({})).resolves.toEqual({
|
||||
runId: continuation.binding.runId,
|
||||
});
|
||||
await expect(
|
||||
continuationOptions.dynamicToolHandler!({}),
|
||||
).resolves.toEqual({ runId: continuation.binding.runId });
|
||||
|
||||
@@ -2623,7 +2623,13 @@ export async function nativeProviderRecoveryEvidence(input: {
|
||||
const durableEvents = await input.db
|
||||
.select({ eventType: heartbeatRunEvents.eventType })
|
||||
.from(heartbeatRunEvents)
|
||||
.where(eq(heartbeatRunEvents.runId, input.runId));
|
||||
.where(
|
||||
and(
|
||||
eq(heartbeatRunEvents.runId, input.runId),
|
||||
inArray(heartbeatRunEvents.eventType, [...PROVIDER_DURABLE_EVENT_TYPES]),
|
||||
),
|
||||
)
|
||||
.limit(1);
|
||||
const providerEventsExist = durableEvents.some((event) =>
|
||||
PROVIDER_DURABLE_EVENT_TYPES.has(event.eventType),
|
||||
);
|
||||
@@ -4171,23 +4177,7 @@ async function executePaperclipNativeSessionWithinScope(
|
||||
entry.busy = true;
|
||||
if (entry.idleTimer !== null) clearTimeout(entry.idleTimer);
|
||||
entry.idleTimer = null;
|
||||
if (input.useRunnerd) {
|
||||
// A retained driver closes over the tool authority from the run that
|
||||
// created it. Preserve provider continuity through its durable
|
||||
// checkpoint, but rebuild the runnerd backend so the next run gets a
|
||||
// fresh, immutable authority epoch. Reusing the live driver would
|
||||
// either retain stale authority or require rebinding old callbacks.
|
||||
warmNativeSessions.delete(warmSessionId);
|
||||
await entry.session.close({
|
||||
reason: "warm native session authority epoch rotated",
|
||||
});
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(
|
||||
input.execution,
|
||||
warmConfigDigest,
|
||||
);
|
||||
} else {
|
||||
existingWarmSession = entry.session;
|
||||
}
|
||||
existingWarmSession = entry.session;
|
||||
}
|
||||
} else {
|
||||
persistedWarmSession = loadWarmNativeCheckpoint(
|
||||
@@ -7235,6 +7225,13 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
input.restartRecovery?.kind === "reattach_existing_runner"
|
||||
? input.restartRecovery.process
|
||||
: null;
|
||||
const executeCurrentToolAuthority = (
|
||||
call: Parameters<SessionToolAuthorityEpoch["execute"]>[0],
|
||||
) => {
|
||||
const current = sessionToolAuthorityEpochs.get(sessionScopeId);
|
||||
if (!current) throw new Error("native_session_tool_authority_unavailable");
|
||||
return current.execute(call);
|
||||
};
|
||||
const backend = createNativeSessionBackend(runnerExecution, {
|
||||
runnerInstanceId: input.runnerInstanceId,
|
||||
environment: effectiveRunnerEnvironment,
|
||||
@@ -7243,8 +7240,8 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
: "local_filesystem",
|
||||
onSpawn: input.onSpawn,
|
||||
dynamicTools,
|
||||
dynamicToolHandler: (call) => authorityEpoch.execute(call),
|
||||
acpxDynamicToolHandler: (call) => authorityEpoch.execute(call),
|
||||
dynamicToolHandler: executeCurrentToolAuthority,
|
||||
acpxDynamicToolHandler: executeCurrentToolAuthority,
|
||||
opencodeRuntimeDirectory: resolve(
|
||||
resolvePaperclipInstanceRoot(),
|
||||
"runtime",
|
||||
@@ -7419,7 +7416,7 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
turnId: `turn-${input.execution.binding.runId}`,
|
||||
itemId: `item-${input.execution.binding.runId}`,
|
||||
},
|
||||
controlPlaneRegistration: (authority) =>
|
||||
controlPlaneRegistration: (authority, attachmentIdentity) =>
|
||||
measureNativeRunnerSpan(
|
||||
input.trace,
|
||||
"runner.transport.connect",
|
||||
@@ -7446,7 +7443,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
() =>
|
||||
registerRunnerPrpAuthority({
|
||||
companyId: input.execution.binding.companyId,
|
||||
runId: input.execution.binding.runId,
|
||||
runId:
|
||||
attachmentIdentity?.runId ??
|
||||
input.execution.binding.runId,
|
||||
authority,
|
||||
}),
|
||||
);
|
||||
@@ -7469,7 +7468,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
() =>
|
||||
resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: input.execution.binding.runId,
|
||||
runId:
|
||||
attachmentIdentity?.runId ??
|
||||
input.execution.binding.runId,
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: input.runnerPublicUrl,
|
||||
runnerCaBundlePath: input.runnerCaBundlePath,
|
||||
@@ -7505,7 +7506,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
() =>
|
||||
resolvePaperclipRunnerTransport({
|
||||
target,
|
||||
runId: input.execution.binding.runId,
|
||||
runId:
|
||||
attachmentIdentity?.runId ??
|
||||
input.execution.binding.runId,
|
||||
localConnectUrl: "ws://127.0.0.1/unused",
|
||||
runnerPublicUrl: input.runnerPublicUrl,
|
||||
runnerCaBundlePath: input.runnerCaBundlePath,
|
||||
@@ -7541,7 +7544,9 @@ async function createRunnerdBackendWithinSessionClaim(
|
||||
() =>
|
||||
registerRunnerPrpAuthority({
|
||||
companyId: input.execution.binding.companyId,
|
||||
runId: input.execution.binding.runId,
|
||||
runId:
|
||||
attachmentIdentity?.runId ??
|
||||
input.execution.binding.runId,
|
||||
authority,
|
||||
}),
|
||||
{ parentName: "runner.transport.connect" },
|
||||
|
||||
@@ -10,6 +10,13 @@ import {
|
||||
} from "@paperclipai/db";
|
||||
import { workspaceOperationService } from "../workspace-operations.js";
|
||||
import { inspectManagedGitWorktreeBranch } from "../workspace-runtime.js";
|
||||
import { environmentService } from "../environments.js";
|
||||
import type { EnvironmentRuntimeService } from "../environment-runtime.js";
|
||||
import { resolveEnvironmentExecutionTarget } from "../environment-execution-target.js";
|
||||
import {
|
||||
readNativeWorkspaceSyncReference,
|
||||
resumeNativeWorkspaceSync,
|
||||
} from "./native-workspace-sync.js";
|
||||
|
||||
function record(value: unknown): Record<string, unknown> {
|
||||
return value && typeof value === "object" && !Array.isArray(value)
|
||||
@@ -21,14 +28,27 @@ function readString(value: unknown) {
|
||||
return typeof value === "string" && value.trim().length > 0 ? value.trim() : null;
|
||||
}
|
||||
|
||||
function workspaceSyncFailure(
|
||||
code: "workspace_sync_out_failed" | "workspace_sync_out_unrecoverable",
|
||||
) {
|
||||
return {
|
||||
status: "failed" as const,
|
||||
exitCode: 1,
|
||||
stderr: `${code}\n`,
|
||||
metadata: { workspaceSync: { code } },
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Resume the real workspace-finalization action for a result-bearing native
|
||||
* run. This service observes the workspace; it never fabricates a successful
|
||||
* marker. The caller decides whether a failed observation is retryable.
|
||||
* run. Durable sandbox-backed runs export and merge the remote workspace;
|
||||
* older runs retain the local workspace validation path. The caller owns the
|
||||
* durable retry and failure policy.
|
||||
*/
|
||||
export async function resumeNativeWorkspaceFinalization(input: {
|
||||
db: Db;
|
||||
runId: string;
|
||||
environmentRuntime?: EnvironmentRuntimeService;
|
||||
}) {
|
||||
const bound = await input.db.select({
|
||||
companyId: heartbeatRuns.companyId,
|
||||
@@ -57,7 +77,12 @@ export async function resumeNativeWorkspaceFinalization(input: {
|
||||
eq(workspaceOperations.phase, "workspace_finalize"),
|
||||
)).orderBy(desc(workspaceOperations.createdAt)).limit(1).then((rows) => rows[0] ?? null);
|
||||
|
||||
const persistedInput = record(record(bound.runnerProfileJson).nativeExecutionInput);
|
||||
const persistedInput = record(
|
||||
record(bound.runnerProfileJson).nativeExecutionInput,
|
||||
);
|
||||
const nativeWorkspaceSync = readNativeWorkspaceSyncReference(
|
||||
record(bound.runnerProfileJson).nativeWorkspaceSync,
|
||||
);
|
||||
const binding = record(persistedInput.binding);
|
||||
const workspaceId = previous?.executionWorkspaceId
|
||||
?? bound.issueWorkspaceId
|
||||
@@ -87,6 +112,88 @@ export async function resumeNativeWorkspaceFinalization(input: {
|
||||
: "workspace_directory",
|
||||
},
|
||||
run: async () => {
|
||||
if (nativeWorkspaceSync) {
|
||||
if (!input.environmentRuntime) {
|
||||
return {
|
||||
status: "failed",
|
||||
exitCode: 1,
|
||||
stderr:
|
||||
"Native workspace finalization cannot access the environment runtime.\n",
|
||||
};
|
||||
}
|
||||
const environmentsSvc = environmentService(input.db);
|
||||
const lease = await environmentsSvc.getLeaseById(
|
||||
nativeWorkspaceSync.leaseId,
|
||||
);
|
||||
const environment = lease?.environmentId
|
||||
? await environmentsSvc.getById(lease.environmentId)
|
||||
: null;
|
||||
if (
|
||||
!lease ||
|
||||
!environment ||
|
||||
lease.companyId !== bound.companyId ||
|
||||
environment.id !== lease.environmentId
|
||||
) {
|
||||
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
|
||||
}
|
||||
if (
|
||||
lease.status === "expired" ||
|
||||
lease.status === "failed" ||
|
||||
lease.status === "pending_cleanup" ||
|
||||
!lease.providerLeaseId ||
|
||||
lease.providerLeaseId !== nativeWorkspaceSync.providerLeaseId
|
||||
) {
|
||||
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
|
||||
}
|
||||
const target = await resolveEnvironmentExecutionTarget({
|
||||
db: input.db,
|
||||
companyId: bound.companyId,
|
||||
adapterType: "paperclip_runner",
|
||||
environment,
|
||||
leaseId: lease.id,
|
||||
leaseMetadata: lease.metadata,
|
||||
lease,
|
||||
environmentRuntime: input.environmentRuntime,
|
||||
});
|
||||
if (!target) {
|
||||
return {
|
||||
status: "failed",
|
||||
exitCode: 1,
|
||||
stderr: "Native workspace finalization target is unavailable.\n",
|
||||
};
|
||||
}
|
||||
try {
|
||||
const restored = await resumeNativeWorkspaceSync({
|
||||
db: input.db,
|
||||
runId: input.runId,
|
||||
target,
|
||||
});
|
||||
if (!restored) {
|
||||
return workspaceSyncFailure("workspace_sync_out_unrecoverable");
|
||||
}
|
||||
return {
|
||||
status: "succeeded",
|
||||
exitCode: 0,
|
||||
system:
|
||||
"Native workspace finalization restored the remote workspace.\n",
|
||||
metadata: {
|
||||
workspaceSync: {
|
||||
schema: nativeWorkspaceSync.schema,
|
||||
workspaceId: nativeWorkspaceSync.workspaceId,
|
||||
leaseId: nativeWorkspaceSync.leaseId,
|
||||
},
|
||||
},
|
||||
};
|
||||
} catch (error) {
|
||||
const code =
|
||||
error instanceof Error &&
|
||||
(error.message === "workspace_sync_out_unrecoverable" ||
|
||||
error.message.includes("daytona_sandbox_not_found"))
|
||||
? "workspace_sync_out_unrecoverable"
|
||||
: "workspace_sync_out_failed";
|
||||
return workspaceSyncFailure(code);
|
||||
}
|
||||
}
|
||||
if (!cwd) {
|
||||
return {
|
||||
status: "failed",
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
Reference in new issue
Block a user