mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 20:29:18 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native runs must preserve files when Stop ends a turn before its result is saved. > - The cancellation path could skip file retention and leave a prepared remote workspace active. > - An older controller could also admit an extra run while an earlier lost-source hold was unresolved. > - Rejecting every later run then prevented recovery even after that extra run was stopped and its files were reconciled. > - This change retains cancelled workspaces and records the exact proof for a safely reconciled intervening cancellation. > - Recovery can proceed without changing cancelled history, replaying actions, or weakening execution gates. ## Linked Issues or Issue Description Refs: #16583, #16690, #16696. **What happened?** Stop could acknowledge a native turn without recording its unsaved file obligation. Reconstruction also rejected a later cancelled run after its exact environment and files had been reconciled. The task could remain inactive with an `in_progress` status, which the reconstruction endpoint rejected. **Expected behavior** Retain unsaved files after Stop. Permit a board operator to reconstruct from saved sources only after each intervening cancellation has a confirmed stop and a file-only repair record. Preserve the original failed export and the cancelled run. **Steps to reproduce** 1. Prepare a native workspace and cancel the turn before a structured result exists. 2. Inspect its file retention and remote lease state. 3. For a task with an older destroyed export, reconcile the later cancelled run's exact files and unperformed actions. 4. Request reconstruction from verified same-task attachments. ## What Changed - Pin prepared native files before cancellation settlement. Defer teardown if pinning fails. Use the existing stop-only cleanup worker. - Repair active warm and ephemeral sources, plus stopping and confirmed-stopped historical cancellations in bounded startup and periodic batches. Preserve the existing exact-source checks. Bound row-lock waits and advance past a busy source. - Accept only same-owner native run-only cancellations with no result or current coordinator, confirmed stopped leases, and board file-only repair with `not_performed` action outcomes. - Save hashes of each intervening repair and recheck them at queue claim and final dispatch. Reject changed evidence or another later run. - Expose saved-source reconstruction in the issue API for inactive `blocked` and `in_progress` tasks, and hide it when an execution or checkout owner exists. Permit the same inactive source at submission. Keep the source, owner, version, attachment, budget, pause and retry checks. - Update the shared evidence type, recovery runbook and capability maps. Add an optional cancelled-native no-action confirmation to file recovery, resetting evidence when its source changes. No migration, new endpoint, permission or dependency. ## Verification - The new fixture fails on the original validator with `workspace_reconstruction_stale` after valid cancelled-run file repair. - Focused source and Stop retention tests: 94 passed before the final contention case. The subsequent three-suite run passed 123 tests, including real PostgreSQL lock contention and public startup/API/final-dispatch paths. The final narrow-query rerun passed all 123 tests. Review correction: all 143 tests passed across four suites, including real lock contention through the teardown boundary and stopping/stopped allocations. - Six broader cancellation, executor, heartbeat and restart suites: 831 passed, 9 existing skips. - Final correction: all 145 backend tests and all 36 recovery UI tests passed. Real contention covers both warm reusable and ephemeral sources. Optional no-action confirmation defaults to uncertified, and changing source clears evidence. Workspace `pnpm -r typecheck` and token gates passed on these sources. - Maps, module boundaries and Git whitespace checks: passed. Full `pnpm build` passed on the final `4e8c0e73ac` revision as well as the prior reviewed heads. A full local run of the preceding fix completed with 21,724 passes, 109 skips and three failures in webhook body limits, a status-card timeout, and a port-allocation fixture. All three suites then passed unchanged in an isolated rerun (209 tests). The full local run remains recorded as failed; the final head's full CI passed (55 completed checks, no failures). A fresh full local run is also in progress on the frozen reviewed head. - Preserved failures: the first attempt lacked a built plugin SDK; after building prerequisites, the regression reached the expected validator rejection. Treatment tests caught the file-only action's deliberately null continuation owner. A real lock test then caught the database driver's wrapped timeout error. Both defects were corrected and retested. Independent review then found a teardown race after a failed retention write. A second review found active warm sources excluded by the historical policy filter; the regression reproduced both warm failures before correcting the query. Its real-row-lock fixture reproduced the failure; the fixed release guard and historical stopping/stopped cases pass. No checks were weakened. - A pre-deployment walkthrough caught the issue API still hiding reconstruction for an inactive `in_progress` task. The status/capability baseline reproduced three failures, including inappropriate capability flags for execution/checkout owners. The fix uses the same inactive-state requirements as submission; its positive fixture reads the assembled issue API before reconstructing. All 117 tests in those three recovery suites pass, and all 28 heartbeat-recovery tests pass separately. Full workspace typecheck passed again. Full build passed with the exact `4e8c0e73ac` stamp; latest-head independent review is 5/5 with no open findings. All 55 latest-head checks completed with no failures. - Live tenant acceptance remains pending. No successful reconstruction, export or warm-process proof is claimed by these local tests. ## Risks - Stop retains unfinished work, so storage remains until file recovery and explicit cleanup complete. - Reconstruction remains held for unknown or performed external actions, foreign owners, live or rebound environments, accepted results, changed evidence, and unrelated later runs. - The sweep examines at most five candidates per pass. It uses a fair cursor and bounded transaction waits. It performs no provider call and starts no new turn. - Intervening verification reads only the needed JSON fields. The source transaction still serializes recovery for the same task. ## Model Used OpenAI Codex, GPT-6 family, with reasoning, tool use and code execution. The session does not expose an exact backend model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
73 lines
5.3 KiB
TypeScript
73 lines
5.3 KiB
TypeScript
import { and, eq, inArray, ne, sql } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
|
|
import type { ExecutionReconciliation } from "@paperclipai/shared";
|
|
import { hasAcknowledgedNativeStopIntent } from "../acknowledged-native-stop.js";
|
|
import { hasRemoteTerminationReceipt } from "../remote-execution-termination.js";
|
|
import { hasRequiredWorkspaceRecovery } from "../workspace-restore-recovery-state.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import { readNativeWorkspaceSyncReference } from "./native-workspace-sync.js";
|
|
|
|
type RecordedExecution = NonNullable<NonNullable<ExecutionReconciliation["workspaceReconstruction"]>["interveningExecutions"]>;
|
|
type InterveningRun = Pick<typeof heartbeatRuns.$inferSelect,
|
|
"id" | "companyId" | "agentId" | "nativeIssueId" | "runtimeMode" | "status" | "finishedAt" | "runnerProfileJson" | "resultJson">;
|
|
|
|
/** A later run is not made harmless by its terminal status. Only an audited
|
|
* run-only cancellation with independently recorded file repair can be kept
|
|
* alongside the older loss decision. Preserve its exact proof through dispatch. */
|
|
export async function reconciledInterveningExecutions(db: Db, source: typeof heartbeatRuns.$inferSelect,
|
|
runs: InterveningRun[]): Promise<RecordedExecution | null> {
|
|
const recorded: RecordedExecution = [];
|
|
for (const run of runs) {
|
|
if (run.companyId !== source.companyId || run.agentId !== source.agentId || run.nativeIssueId !== source.nativeIssueId
|
|
|| run.runtimeMode !== "native" || run.status !== "cancelled" || !run.finishedAt
|
|
|| !hasAcknowledgedNativeStopIntent(run) || !hasRequiredWorkspaceRecovery(run.resultJson)
|
|
|| run.runnerProfileJson?.nativeWorkspaceFinalizationOwner) return null;
|
|
const [coordinator] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, run.companyId), eq(nativeRunFinalizations.runId, run.id),
|
|
eq(nativeRunFinalizations.issueId, run.nativeIssueId!),
|
|
)).for("share");
|
|
if (!coordinator || coordinator.phase !== "terminal_failure" || coordinator.failureCode !== "native_retry_cancelled"
|
|
|| coordinator.leaseOwner || coordinator.resultId || coordinator.nextAttemptAt || coordinator.failureDetail?.successorRunId) return null;
|
|
const [result] = await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
|
|
eq(nativeRunResults.companyId, run.companyId), eq(nativeRunResults.runId, run.id),
|
|
)).limit(1);
|
|
if (result) return null;
|
|
const reference = readNativeWorkspaceSyncReference(run.runnerProfileJson?.nativeWorkspaceSync);
|
|
const leases = await db.select().from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, run.companyId), eq(environmentLeases.heartbeatRunId, run.id),
|
|
)).for("share");
|
|
if (!reference || !leases.some(lease => lease.id === reference.leaseId && lease.providerLeaseId === reference.providerLeaseId)
|
|
|| leases.some(lease => lease.issueId !== run.nativeIssueId || !hasRemoteTerminationReceipt(lease)
|
|
|| lease.metadata?.pendingCleanupInFlight === true
|
|
|| (lease.metadata?.remoteExecutionTermination as { state?: string } | undefined)?.state !== "stopped")) return null;
|
|
for (const lease of leases) {
|
|
const [other] = await db.select({ id: environmentLeases.id }).from(environmentLeases).where(and(
|
|
ne(environmentLeases.id, lease.id), eq(environmentLeases.provider, lease.provider!),
|
|
eq(environmentLeases.providerLeaseId, lease.providerLeaseId!), inArray(environmentLeases.status, ["active", "retained", "pending_cleanup"]),
|
|
)).limit(1);
|
|
if (other) return null;
|
|
}
|
|
const actions = await db.select().from(issueRecoveryActions).where(and(
|
|
eq(issueRecoveryActions.companyId, run.companyId), eq(issueRecoveryActions.sourceIssueId, run.nativeIssueId!),
|
|
eq(issueRecoveryActions.status, "resolved"), eq(issueRecoveryActions.outcome, "restored"),
|
|
sql`${issueRecoveryActions.evidence}->>'runId' = ${run.id}`,
|
|
)).for("share");
|
|
const repairs = actions.filter(action => {
|
|
const decision = action.evidence.executionReconciliation as ExecutionReconciliation | undefined;
|
|
return action.evidence.runId === run.id && action.kind === "active_run_watchdog" && action.ownerType === "board"
|
|
&& action.returnOwnerAgentId === null && hasRequiredWorkspaceRecovery(action.evidence)
|
|
&& action.evidence.continuationDelivery === "not_requested" && decision?.runId === run.id
|
|
&& decision.providerStopped === true && decision.actionOutcome === "not_performed"
|
|
&& typeof decision.workspaceRepairEvidence === "string" && decision.workspaceRepairEvidence.trim().length >= 20;
|
|
});
|
|
if (repairs.length !== 1) return null;
|
|
const repair = repairs[0];
|
|
recorded.push({ runId: run.id, recoveryActionId: repair.id, evidenceSha256: nativeSha256({
|
|
runId: run.id, finishedAt: run.finishedAt.toISOString(), reference, recovery: run.resultJson?.workspaceRestoreRecovery,
|
|
reconciliation: repair.evidence.executionReconciliation,
|
|
leases: leases.map(lease => ({ id: lease.id, receipt: lease.metadata?.remoteExecutionTermination })).sort((a, b) => a.id.localeCompare(b.id)),
|
|
}) });
|
|
}
|
|
return recorded.sort((a, b) => a.runId.localeCompare(b.runId));
|
|
}
|