Files
PaperClipAI/server/src/services/native-runtime/native-reconstruction-intervening-executions.ts
T
DottaandPaperclip 6bec153410 fix(runtime): retain cancelled workspaces before reconstruction (#16700)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs must preserve files when Stop ends a turn before its
result is saved.
> - The cancellation path could skip file retention and leave a prepared
remote workspace active.
> - An older controller could also admit an extra run while an earlier
lost-source hold was unresolved.
> - Rejecting every later run then prevented recovery even after that
extra run was stopped and its files were reconciled.
> - This change retains cancelled workspaces and records the exact proof
for a safely reconciled intervening cancellation.
> - Recovery can proceed without changing cancelled history, replaying
actions, or weakening execution gates.

## Linked Issues or Issue Description

Refs: #16583, #16690, #16696.

**What happened?**

Stop could acknowledge a native turn without recording its unsaved file
obligation. Reconstruction also rejected a later cancelled run after its
exact environment and files had been reconciled. The task could remain
inactive with an `in_progress` status, which the reconstruction endpoint
rejected.

**Expected behavior**

Retain unsaved files after Stop. Permit a board operator to reconstruct
from saved sources only after each intervening cancellation has a
confirmed stop and a file-only repair record. Preserve the original
failed export and the cancelled run.

**Steps to reproduce**

1. Prepare a native workspace and cancel the turn before a structured
result exists.
2. Inspect its file retention and remote lease state.
3. For a task with an older destroyed export, reconcile the later
cancelled run's exact files and unperformed actions.
4. Request reconstruction from verified same-task attachments.

## What Changed

- Pin prepared native files before cancellation settlement. Defer
teardown if pinning fails. Use the existing stop-only cleanup worker.
- Repair active warm and ephemeral sources, plus stopping and
confirmed-stopped historical cancellations in bounded startup and
periodic batches. Preserve the existing exact-source checks. Bound
row-lock waits and advance past a busy source.
- Accept only same-owner native run-only cancellations with no result or
current coordinator, confirmed stopped leases, and board file-only
repair with `not_performed` action outcomes.
- Save hashes of each intervening repair and recheck them at queue claim
and final dispatch. Reject changed evidence or another later run.
- Expose saved-source reconstruction in the issue API for inactive
`blocked` and `in_progress` tasks, and hide it when an execution or
checkout owner exists. Permit the same inactive source at submission.
Keep the source, owner, version, attachment, budget, pause and retry
checks.
- Update the shared evidence type, recovery runbook and capability maps.
Add an optional cancelled-native no-action confirmation to file
recovery, resetting evidence when its source changes. No migration, new
endpoint, permission or dependency.

## Verification

- The new fixture fails on the original validator with
`workspace_reconstruction_stale` after valid cancelled-run file repair.
- Focused source and Stop retention tests: 94 passed before the final
contention case. The subsequent three-suite run passed 123 tests,
including real PostgreSQL lock contention and public
startup/API/final-dispatch paths. The final narrow-query rerun passed
all 123 tests. Review correction: all 143 tests passed across four
suites, including real lock contention through the teardown boundary and
stopping/stopped allocations.
- Six broader cancellation, executor, heartbeat and restart suites: 831
passed, 9 existing skips.
- Final correction: all 145 backend tests and all 36 recovery UI tests
passed. Real contention covers both warm reusable and ephemeral sources.
Optional no-action confirmation defaults to uncertified, and changing
source clears evidence. Workspace `pnpm -r typecheck` and token gates
passed on these sources.
- Maps, module boundaries and Git whitespace checks: passed. Full `pnpm
build` passed on the final `4e8c0e73ac` revision as well as the prior
reviewed heads. A full local run of the preceding fix completed with
21,724 passes, 109 skips and three failures in webhook body limits, a
status-card timeout, and a port-allocation fixture. All three suites
then passed unchanged in an isolated rerun (209 tests). The full local
run remains recorded as failed; the final head's full CI passed (55
completed checks, no failures). A fresh full local run is also in
progress on the frozen reviewed head.
- Preserved failures: the first attempt lacked a built plugin SDK; after
building prerequisites, the regression reached the expected validator
rejection. Treatment tests caught the file-only action's deliberately
null continuation owner. A real lock test then caught the database
driver's wrapped timeout error. Both defects were corrected and
retested. Independent review then found a teardown race after a failed
retention write. A second review found active warm sources excluded by
the historical policy filter; the regression reproduced both warm
failures before correcting the query. Its real-row-lock fixture
reproduced the failure; the fixed release guard and historical
stopping/stopped cases pass. No checks were weakened.
- A pre-deployment walkthrough caught the issue API still hiding
reconstruction for an inactive `in_progress` task. The status/capability
baseline reproduced three failures, including inappropriate capability
flags for execution/checkout owners. The fix uses the same
inactive-state requirements as submission; its positive fixture reads
the assembled issue API before reconstructing. All 117 tests in those
three recovery suites pass, and all 28 heartbeat-recovery tests pass
separately. Full workspace typecheck passed again. Full build passed
with the exact `4e8c0e73ac` stamp; latest-head independent review is 5/5
with no open findings. All 55 latest-head checks completed with no
failures.
- Live tenant acceptance remains pending. No successful reconstruction,
export or warm-process proof is claimed by these local tests.

## Risks

- Stop retains unfinished work, so storage remains until file recovery
and explicit cleanup complete.
- Reconstruction remains held for unknown or performed external actions,
foreign owners, live or rebound environments, accepted results, changed
evidence, and unrelated later runs.
- The sweep examines at most five candidates per pass. It uses a fair
cursor and bounded transaction waits. It performs no provider call and
starts no new turn.
- Intervening verification reads only the needed JSON fields. The source
transaction still serializes recovery for the same task.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution.
The session does not expose an exact backend model ID or context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-18 01:33:11 -05:00

73 lines
5.3 KiB
TypeScript

import { and, eq, inArray, ne, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import type { ExecutionReconciliation } from "@paperclipai/shared";
import { hasAcknowledgedNativeStopIntent } from "../acknowledged-native-stop.js";
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js";
import { nativeSha256 } from "./canonical.js";
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
type RecordedExecution = NonNullable<NonNullable<ExecutionReconciliation["workspaceReconstruction"]>["interveningExecutions"]>;
type InterveningRun = Pick<typeof heartbeatRuns.$inferSelect,
"id" | "companyId" | "agentId" | "nativeIssueId" | "runtimeMode" | "status" | "finishedAt" | "runnerProfileJson" | "resultJson">;
/** A later run is not made harmless by its terminal status. Only an audited
* run-only cancellation with independently recorded file repair can be kept
* alongside the older loss decision. Preserve its exact proof through dispatch. */
export async function reconciledInterveningExecutions(db: Db, source: typeof heartbeatRuns.$inferSelect,
runs: InterveningRun[]): Promise<RecordedExecution | null> {
const recorded: RecordedExecution = [];
for (const run of runs) {
if (run.companyId !== source.companyId || run.agentId !== source.agentId || run.nativeIssueId !== source.nativeIssueId
|| run.runtimeMode !== "native" || run.status !== "cancelled" || !run.finishedAt
|| !hasAcknowledgedNativeStopIntent(run) || !hasRequiredWorkspaceRecovery(run.resultJson)
|| run.runnerProfileJson?.nativeWorkspaceFinalizationOwner) return null;
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
eq(nativeRunFinalizations.issueId, run.nativeIssueId!),
)).for("share");
if (!coordinator || coordinator.phase !== "terminal_failure" || coordinator.failureCode !== "native_retry_cancelled"
|| coordinator.leaseOwner || coordinator.resultId || coordinator.nextAttemptAt || coordinator.failureDetail?.successorRunId) return null;
const [result] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id),
)).limit(1);
if (result) return null;
const reference = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
const leases = await db.select().from(environmentLeases).where(and(
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
)).for("share");
if (!reference || !leases.some(lease => lease.id === reference.leaseId && lease.providerLeaseId === reference.providerLeaseId)
|| leases.some(lease => lease.issueId !== run.nativeIssueId || !hasRemoteTerminationReceipt(lease)
|| lease.metadata?.pendingCleanupInFlight === true
|| (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state !== "stopped")) return null;
for (const lease of leases) {
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!),
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]),
)).limit(1);
if (other) return null;
}
const actions = await db.select().from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, run.nativeIssueId!),
eq(issueRecoveryActions.status, "resolved"), eq(issueRecoveryActions.outcome, "restored"),
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
)).for("share");
const repairs = actions.filter(action => {
const decision = action.evidence.executionReconciliation as ExecutionReconciliation | undefined;
return action.evidence.runId === run.id && action.kind === "active_run_watchdog" && action.ownerType === "board"
&& action.returnOwnerAgentId === null && hasRequiredWorkspaceRecovery(action.evidence)
&& action.evidence.continuationDelivery === "not_requested" && decision?.runId === run.id
&& decision.providerStopped === true && decision.actionOutcome === "not_performed"
&& typeof decision.workspaceRepairEvidence === "string" && decision.workspaceRepairEvidence.trim().length >= 20;
});
if (repairs.length !== 1) return null;
const repair = repairs[0];
recorded.push({ runId: run.id, recoveryActionId: repair.id, evidenceSha256: nativeSha256({
runId: run.id, finishedAt: run.finishedAt.toISOString(), reference, recovery: run.resultJson?.workspaceRestoreRecovery,
reconciliation: repair.evidence.executionReconciliation,
leases: leases.map(lease => ({ id: lease.id, receipt: lease.metadata?.remoteExecutionTermination })).sort((a, b) => a.id.localeCompare(b.id)),
}) });
}
return recorded.sort((a, b) => a.runId.localeCompare(b.runId));
}