mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-08 12:33:05 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Native task recovery must save unfinished workspace exports before continuation. > - A prepared export can still refer to a sandbox with a warm reuse policy. > - The strict recovery inspector rejects that policy and exhausts source verification. > - Ordinary reuse must not give the same unfinished allocation to another run. > - An existing user Retry can authorize bounded file maintenance after exact ownership checks. > - This change retains that exact source and preserves the original failure record. > - Recovery completes only when files, stop evidence and policy settlement agree. ## Linked Issues or Issue Description **What happened?** A failed native run has a prepared `keep_running` workspace export. Its exact sandbox lease still has `reuse_by_environment` policy. Native export cleanup can preserve that policy while it stops the sandbox. Generic workspace recovery then rejects the source with `lease_policy_not_retained`. A saved user Retry remains blocked before file export. **Expected behavior** Keep unfinished exports out of ordinary warm reuse. With current user authority and an unchanged saved Retry, retain the exact source for one bounded maintenance generation. Preserve files and return continuation to normal admission. Keep all ambiguous sources held. **Steps to reproduce** 1. Prepare a native workspace export with a warm sandbox and `keep_running` disposition. 2. Save a normal board Retry for the failed source run. 3. Persist its first exhausted source-verification failure before file export. 4. Run the normal maintenance worker. Before this change, source policy rejection leaves the saved files unrestored. Refs #17773 for source diagnostics. Related #17839 records a missing preservation obligation during blocked restart. This change handles an existing prepared warm export whose verification already failed. ## What Changed - Exclude unfinished physical allocations from reuse and obsolete-lease cleanup across every historical lease row. Reuse a clean candidate or acquire a fresh sandbox through normal selection. Preserve exact same-run reattachment, workspace-less execution and ordinary task admission checks. - Preserve the newer prepared remote-workspace writer proof for native and legacy runs. Share its binding check between single-source recovery and one batched allocation read. - Keep persisted recovery-action IDs when constructing continuation source checks. Synthetic sources cannot grant snapshot receipt authority. - Permit one audited maintenance adoption for the first exhausted source-verification attempt. Require an unadmitted canonical saved board Retry, current user authority, unchanged assignment and latest source run. - Keep all company, source, allocation, workspace writer, controller and physical finalization ownership checks. The public read-only inspector keeps its strict policy check. - Pin the exact source for file maintenance. Preserve the previous generation and reuse the existing bounded recovery worker. Create no provider turn or new user request. - Restore the warm policy in the same transaction as verified native copyback completion. A failed write or changed settlement keeps a recovery obligation. - When local merge metadata is missing, settle the exact verified snapshot while keeping its unfinished allocation retained. Deliver the saved Retry through normal fresh isolation and attach the verified files. Keep the accepted result as history. - Require native copyback proof before reopening an accepted result's original workspace finalizer. - Validate snapshot receipts through one shared boundary for warm, retained and ephemeral sources. Bind the receipt to the current generation's operation IDs, company, issue, run, recovery action, lease and provider allocation. - Retire fixture input before removing recovery proof, and use the same configured heartbeat for HTTP and worker dispatch. Keep the exact catalog refusal and no-provider-work assertions. - Document the recovery contract in technical documentation. Leave capability maps unchanged for this bug fix. ## Verification - Candidate-selection RED: all six clean-candidate and fresh-allocation controls fail on unchanged `f54e7baf538` because a pending export aborts the whole acquisition. GREEN: the focused suite passes **15/15 cases** after candidate filtering. Both provider paths preserve the protected lease and never resume or destroy its allocation. Obsolete fingerprints cannot send that allocation to cleanup. Historical-row fencing, modern native/legacy writers, exact same-run reuse, workspace-less execution and the single-read allocation-history check remain covered. - The existing reusable-runtime selection cases passed **40/40**. Driver/capability contract suites passed **5/5**. Reusable service controls passed **2/2**. After merging current master, the combined lease/provider suites pass 61 cases and the recovery integration selection passes 36 cases, including the upstream inherited-session and resolved-interruption assertions. This gives 97 passing targeted cases on the final source. A duplicate test-app declaration from the automatic merge was removed while retaining the upstream assertions. - The source-history repair has meaningful red/green evidence: three valid retired-handoff cases could not complete under the old blanket history-row rejection. Settlement now uses the same full source inspector as adoption. It accepts exact retired handoffs and still rejects a predecessor that becomes active after export. The 26 focused history, rollback, snapshot and ambiguity cases passed before this last merge. - Server typecheck passes on the final merged source. Module boundaries and `git diff --check` pass. Full local repository tests and a successful current-head full build are not claimed. Current-head CI is complete on `0ef974d76483e11fb0b0304bcb05bbd3b56e2ef9`: both required gates pass, with 54 successful checks, three expected skips, one neutral Paperclip review check and a successful security status. [Full CI evidence](https://github.com/paperclipai/paperclip/actions/runs/41904339370). Greptile reviewed that exact head at **5/5** with no actionable findings. Its check completed successfully. [Ponytail also reviewed the exact head at 5/5](https://github.com/paperclipai/paperclip/pull/17916#issuecomment-6640083408). No review threads remain unresolved, and the branch is mergeable. - Earlier local broad runs exposed a dropped recovery-action ID in the merged source projection and deferred test input being revisited after its records were removed. Both were repaired. Later local disk exhaustion caused manifest and saved-file reserve failures; generated caches were removed only from these checkouts. The affected saved-file control passed unchanged after cleanup. The original logs are retained. No disk reserve or timeout was lowered. - On the prior head, the first [full CI attempt](https://github.com/paperclipai/paperclip/actions/runs/41899038530) received simultaneous runner shutdowns. Four jobs were cancelled, and two late jobs have lost-runner annotations with unavailable logs. The Runner job had five earlier ACPX failures whose full assertion report was lost on interruption. Their cause remains unproven. Identical Runner source passed the [#19004 control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429725), including all seven enabled tests in that file. - That attempt's [browser shard](https://github.com/paperclipai/paperclip/actions/runs/41899038530/job/126817404595) returned to the exact saved task URL, but its conversation was still loading at the unchanged heading deadline. The [artifact](https://github.com/paperclipai/paperclip/actions/runs/41899038530/artifacts/13358840931) is retained. All five Cloud-auth cases passed locally on the unchanged head and in the [same-source CI control](https://github.com/paperclipai/paperclip/actions/runs/41898305521/job/126814429966). The delay remains unexplained. One rerun of failed/interrupted jobs was submitted after the workflow completed. This new source revision requires fresh CI. - The earlier [original-head workflow](https://github.com/paperclipai/paperclip/actions/runs/40552228175) passed all 48 jobs, and prior head `f54e7baf538` received Greptile 5/5. These results do not establish readiness for the new head. The earlier Superagent scan was neutral after a pinned-base checkout failure; its external analysis remains unverified. - These tests use real PostgreSQL and controlled provider boundaries. Hosted rollout and original-task acceptance remain unverified. ## Risks - Maintenance can resume and stop the exact sandbox for file export. It cannot admit a provider turn, choose another allocation or bypass normal continuation gates. - A saved Retry alone is insufficient. A changed source, newer run, active writer, competing lease, lost user authority or previous adoption keeps the hold. - Policy restoration can fail. Completion and restoration now commit together, so normal recovery remains responsible for a retry. - Snapshot preservation cannot repair missing local merge metadata. Its old allocation remains explicitly retained. A snapshot receipt grants no permission to reuse or destroy that allocation, or to replay the accepted result. - The allocation check uses one projected join per candidate over the existing provider lease index. It adds no polling timer or recurring scan. A fresh fallback can create another sandbox while the unfinished source remains retained; it grants no authority to execute a provider turn. - This change does not establish the cause of a prior provider failure or prove that any specific hosted sandbox still contains its original files. ## Model Used OpenAI Codex, GPT-6 family, for the original change. OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), xhigh reasoning, for the merge and integration repairs. Both used repository tools and PostgreSQL verification. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing> Co-authored-by: Devin Foley <devin@paperclip.ing>
99 lines
6.9 KiB
TypeScript
99 lines
6.9 KiB
TypeScript
import { and, eq, sql } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRuns, executionWorkspaces, executionWorkspaceSnapshots, type Db } from "@paperclipai/db";
|
|
import { readRemoteWorkspaceAuthority, remoteWorkspaceRunReferenceSchema, type EnvironmentLease } from "@paperclipai/shared";
|
|
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
|
|
|
type Workspace = Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "metadata">;
|
|
type Transaction = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
|
export type DestroyRemoteWorkspaceLease = (input: {
|
|
companyId: string; workspaceId: string; leaseId: string; environmentId: string; providerLeaseId: string;
|
|
}) => Promise<EnvironmentLease | null>;
|
|
|
|
export type WorkspaceSourceLease = Pick<typeof environmentLeases.$inferSelect,
|
|
"id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId">;
|
|
|
|
export async function preparedRemoteWorkspaceSource(db: Db, lease: WorkspaceSourceLease) {
|
|
if (!lease.heartbeatRunId || !lease.provider || lease.provider === "local" || !lease.providerLeaseId) return null;
|
|
const [bound] = await db.select({ lease: environmentLeases, run: {
|
|
id: heartbeatRuns.id, runtimeMode: heartbeatRuns.runtimeMode,
|
|
controllerBootId: heartbeatRuns.controllerBootId,
|
|
issueId: sql<string | null>`coalesce(${heartbeatRuns.nativeIssueId}::text, ${heartbeatRuns.issueId}::text,
|
|
${heartbeatRuns.contextSnapshot}->>'issueId')`,
|
|
reference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'`,
|
|
legacyReference: sql<unknown>`${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'`,
|
|
} }).from(environmentLeases)
|
|
.innerJoin(heartbeatRuns, and(eq(heartbeatRuns.id, environmentLeases.heartbeatRunId),
|
|
eq(heartbeatRuns.companyId, environmentLeases.companyId)))
|
|
.where(and(eq(environmentLeases.id, lease.id), eq(environmentLeases.companyId, lease.companyId),
|
|
eq(environmentLeases.heartbeatRunId, lease.heartbeatRunId), eq(environmentLeases.provider, lease.provider),
|
|
eq(environmentLeases.providerLeaseId, lease.providerLeaseId),
|
|
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'state' = 'prepared'`)).limit(1);
|
|
if (!bound) return null;
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(bound.run.reference);
|
|
if (!parsed.success || parsed.data.state !== "prepared" || bound.run.legacyReference != null) return null;
|
|
const [workspace] = await db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, lease.companyId), eq(executionWorkspaces.id, parsed.data.workspaceId))).limit(1);
|
|
return matchPreparedRemoteWorkspaceSource(bound, workspace);
|
|
}
|
|
|
|
type PreparedSourceBinding = {
|
|
lease: Pick<typeof environmentLeases.$inferSelect, "id" | "companyId" | "heartbeatRunId" | "provider" | "providerLeaseId"
|
|
| "issueId" | "executionWorkspaceId" | "environmentId" | "metadata">;
|
|
run: { id: string; runtimeMode: string; controllerBootId: string | null; issueId: string | null;
|
|
reference: unknown; legacyReference: unknown };
|
|
};
|
|
|
|
/** Shared binding proof for single-source inspection and batched allocation fencing. */
|
|
export function matchPreparedRemoteWorkspaceSource<T extends PreparedSourceBinding>(bound: T,
|
|
workspace: Pick<typeof executionWorkspaces.$inferSelect, "id" | "companyId" | "sourceIssueId" | "mode" | "metadata"> | null | undefined) {
|
|
const parsed = remoteWorkspaceRunReferenceSchema.safeParse(bound.run.reference);
|
|
if (!parsed.success || parsed.data.state !== "prepared" || bound.run.legacyReference != null) return null;
|
|
const reference = parsed.data;
|
|
const issueId = bound.run.issueId;
|
|
if (typeof issueId !== "string" || bound.lease.issueId !== issueId || reference.leaseId !== bound.lease.id
|
|
|| reference.providerLeaseId !== bound.lease.providerLeaseId || reference.remoteCwd !== bound.lease.metadata?.remoteCwd
|
|
|| bound.lease.executionWorkspaceId && bound.lease.executionWorkspaceId !== reference.workspaceId) return null;
|
|
const authority = readRemoteWorkspaceAuthority(workspace?.metadata);
|
|
if (!workspace || workspace.companyId !== bound.lease.companyId || workspace.id !== reference.workspaceId
|
|
|| workspace.sourceIssueId !== issueId && workspace.mode !== "shared_workspace"
|
|
|| !authority || !authority.dirty || authority.writerRunId !== bound.run.id || authority.epoch !== reference.epoch
|
|
|| authority.snapshotId !== reference.snapshotId || !authority.location
|
|
|| authority.location.environmentId !== bound.lease.environmentId || authority.location.leaseId !== bound.lease.id
|
|
|| authority.location.providerLeaseId !== bound.lease.providerLeaseId || authority.location.remoteCwd !== reference.remoteCwd) return null;
|
|
return { ...bound, workspace, reference, issueId };
|
|
}
|
|
|
|
export async function inspectRemoteWorkspacePersistence(db: Db | Transaction, workspace: Workspace) {
|
|
const authority = readRemoteWorkspaceAuthority(workspace.metadata);
|
|
if (!authority || authority.dirty || authority.writerRunId || !authority.snapshotId) {
|
|
return { ready: false, reason: "The workspace still has unsaved work or an unfinished writer.", authority };
|
|
}
|
|
const [snapshot] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
|
|
eq(executionWorkspaceSnapshots.companyId, workspace.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, workspace.id),
|
|
eq(executionWorkspaceSnapshots.id, authority.snapshotId), eq(executionWorkspaceSnapshots.status, "committed"),
|
|
));
|
|
return { ready: Boolean(snapshot), reason: snapshot ? null : "The saved workspace is unavailable. Its remote files were retained.", authority };
|
|
}
|
|
|
|
/** Called within the workspace lifecycle fence; the saved workspace remains durable. */
|
|
export async function cleanupRemoteWorkspaceAllocation(input: {
|
|
db: Db; workspace: Workspace; destroyLease: DestroyRemoteWorkspaceLease;
|
|
}) {
|
|
const [workspace] = await input.db.select().from(executionWorkspaces).where(and(
|
|
eq(executionWorkspaces.companyId, input.workspace.companyId), eq(executionWorkspaces.id, input.workspace.id),
|
|
));
|
|
if (!workspace || !["archived", "cleanup_failed"].includes(workspace.status)) throw new Error("remote_workspace_cleanup_binding_changed");
|
|
const persistence = await inspectRemoteWorkspacePersistence(input.db, workspace);
|
|
if (!persistence.ready || !persistence.authority) throw new Error(persistence.reason ?? "remote_workspace_saved_state_required");
|
|
const location = persistence.authority.location;
|
|
if (location) {
|
|
const released = await input.destroyLease({ companyId: workspace.companyId, workspaceId: workspace.id,
|
|
leaseId: location.leaseId, environmentId: location.environmentId, providerLeaseId: location.providerLeaseId });
|
|
if (!released || !hasRemoteTerminationReceipt(released)
|
|
|| (released.metadata?.remoteExecutionTermination as { state?: unknown } | undefined)?.state !== "destroyed") {
|
|
throw new Error("remote_workspace_cleanup_unconfirmed");
|
|
}
|
|
}
|
|
return { cleaned: true, warnings: [] as string[] };
|
|
}
|