mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 15:55:46 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Each run of an agent with managed agent files gets a private copy of
the agent directory. Remote copies serialize their prepare, checkpoint,
collect, and release operations through one Postgres advisory lock per
agent, taken with `pg_try_advisory_xact_lock`
> - When the lock is busy, the operation fails at once with "Agent
directory operation is already active"
> - On staging, a deploy restarted the controller while three runs of
the same agent were active. When one run resumed, another run of that
agent was checkpointing its directory. The resume failed, used its last
attempt, and the task became blocked with "Retry or send a message"
> - A run must not fail because another run of the same agent is briefly
using the agent directory
> - This pull request keeps the fail-fast lock and, when it is busy
during preparation, puts the same native run back on the existing
same-run retry schedule a few seconds later
> - The benefit is that lock contention delays a run by seconds instead
of ending it as failed
## Linked Issues or Issue Description
**What happened?**
A task's run failed with `native_continuation_requires_reconciliation`
and the error "Agent directory operation is already active". The task
was blocked and asked the user to retry. The same error ended at least
two other runs of the same agent within one day.
**Steps to reproduce**
1. Use an agent with managed agent files in a remote sandbox.
2. Let one run of the agent checkpoint or collect its agent directory.
3. At the same time, let another run of the same agent prepare (for
example, a same-run resume after a controller restart).
4. The second run fails immediately with "Agent directory operation is
already active".
**Expected behavior**
The second run waits until the first operation is done, then continues.
It does not end as failed.
## What Changed
- `server/src/services/agent-directory-working-copies.ts`: The busy lock
now throws `AgentDirectoryOperationBusyError`, still a 409 with the same
message. Lock behavior is unchanged from `master`: a fail-fast
`pg_try_advisory_xact_lock` held for the life of the holder's
transaction.
- `server/src/services/heartbeat/agent-directory-busy.ts`:
`prepareWithAgentDirectoryDeferral` wraps only instruction preparation
and turns the busy error into `AgentDirectoryPreparationDeferral`
("Another run of this agent is still using its files"). A busy lock met
while collecting files after a turn is not wrapped and keeps its
existing path.
- **Before native admission** (a brand-new run, such as the first run of
a task): `server/src/services/heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` cancels the run as an expected
control-plane wait (`providerWorkStarted: false`) and schedules an
`agent_directory_busy` retry 5–15 seconds later. The task keeps its
execution lock on the retry. The lane spends no failure retries
(`execution-recovery-attempt.ts`), its count is the retry run's
`scheduledRetryAttempt`, and it stops after 30 attempts.
`pre-dispatch-execution-evidence.ts` classifies this wait like the
existing workspace and AI-connection waits, so retry scheduling does not
ask for reconciliation.
- **A resumed native run**:
`server/src/services/native-runtime/native-controller-recovery.ts`
`deferNativeRunForBusyAgentDirectory` keeps the same run on the existing
same-run retry schedule without spending an attempt. It uses the same
guards as `deferNativeControllerRecovery`: it locks the task, requires
the task's execution run to be this run, requires the current lease
owner, and refuses after a Stop, a startup cancellation or a pending
native cancellation, so those settle normally. Its count is stored in
the run's `contextSnapshot`, which a resume claim does not clear, and it
stops after 30.
- `server/src/services/heartbeat.ts`: routes a preparation deferral to
the same-run path or, before admission, to the scheduled-retry path.
After the bound it follows the normal failure path with the plain
message.
- No schema or migration. Postgres releases a dead holder's advisory
lock with its session. The staging evidence showed a live holder
(another run's checkpoint), not a stale one.
- `doc/DEVELOPING.md`: Documented both paths and the bound.
Earlier revisions of this PR waited on the lock and added idle-session
expiry, keepalive, ownership fences, fenced remote removal and attempt
slots. This revision removes all of them.
### How the directory-wait retry keeps the original wake
The retry no longer replaces the wake. `scheduleBoundedRetryForRun`
takes `preserveWakeContext` for this wait: the retry keeps the original
`wakeReason`, `retryReason` and the whole wake context (interaction,
comment, continuation and admission fields), and its wake request keeps
the original reason. The wait is recorded separately as
`preparationWait: { kind: "agent_directory_busy", attempt, nextAttemptAt
}`, next to the carried `executionRetryAccounting`. Gates that key on
the wake reason therefore treat the retry exactly as they treated the
original wake.
| # | Site | Treatment for the directory wait | Why |
| --- | --- | --- | --- |
| 1 | `heartbeat/retries.ts` `scheduleBoundedRetryForRun` | Changed:
`preserveWakeContext` keeps the original wake context and reason and
adds `preparationWait` | Root cause: the retry overwrote the wake reason
|
| 2 | `run-dispatch/domain/wake-context.ts`
`isResolvedInteractionContinuationWakeContext` /
`PREPARATION_WAIT_RETRY_REASONS` | Special case removed | The original
wake context is kept, so the continuation is classified from it |
| 3 | `heartbeat/retries.ts` `interactionRunnerWait` | Special case
removed | The interaction fields are kept in the context |
| 4 | `run-dispatch/adapters/postgres.ts` scheduled-retry dependency
facts | Changed: a preparation-wait retry whose original wake was
admitted as a dependency-blocked interaction
(`dependencyBlockedInteraction`, recorded at wake time) is admitted the
same way | Without the wait the original wake would have run; this gate
is only met because of the wait |
| 5 | `run-dispatch/domain/wake-context.ts`
`isNonAssigneeWorkspaceBusyRetry` + `heartbeat/retries.ts`
`finalizeAgentDirectoryBusyDeferral` | Kept:
`preparationWaitDeferredWhileAssignee`, derived only from the recorded
admission authority | The scheduled-retry ownership gate has no
wake-reason path; the workspace and AI waits use the same recorded flag
|
| 6 | `heartbeat/retries.ts` explicit user continuation admission |
Kept: admits the `agent_directory_busy` scheduling reason | The
successor needs its own authorization receipt |
| 7 | `execution-recovery-attempt.ts` uncharged lane and
`agentDirectoryWaits` | Kept | Cap and accounting |
| 8 | `pre-dispatch-execution-evidence.ts` (`isPreDispatchResourceWait`,
`isAgentDirectoryPreparationWait`), used by
`legacy-execution-recovery.ts`, `cancelled-native-startup.ts`,
`explicit-native-continuation.ts` | Kept | Describe the cancelled
pre-provider parent run, not the retry's wake |
| 9 | `explicit-native-continuation.ts`
`undeliveredLegacyUserCommentIds`, `run-dispatch/adapters/postgres.ts`
`unstartedUserTurnOrigin` | Kept | The cancelled parent did not consume
the user's comment |
| 10 | `execution-projection.ts` label and saved accounting,
`task-interruption-presentation.ts`, UI toast and chat suppression |
Kept | Presentation of the wait |
| 11 | `run-dispatch/domain/policy.ts` AI-wait lock requirement;
AI-subscription lock recheck; AI wait detection
(`execution-configuration.ts`, `provider-limit-wait.ts`, `heartbeat.ts`,
`ai-connections.ts`, `ai-connection-runtime.ts`, `local-ai-login.ts`) |
Not applicable | AI-subscription specific |
| 12 | `workspace-busy-wait.ts`, `environment-selection.ts`,
`workspace-preparation.ts`, UI workspace-holder banners | Not applicable
| Workspace holder parking; the directory wait is a timed retry |
| 13 | `cancelled-native-startup.ts` paused workspace-retry repair;
`failureRetriesBefore*Wait` | Not applicable | Historical workspace and
AI records only; every directory wait saves its ledger |
| 14 | `failedRunRetryBudgetSpent`; storage-wait exhaustion escalation;
`status-relevance.ts` `isAdmissionWait` | Not applicable | A directory
wait never fails the run, stops at its own bound, and its run has
started |
## Verification
- Original wake kinds after a directory wait
(`heartbeat-agent-directory-busy.test.ts`, real heartbeat): an
assignment runs; a plain user comment on a dependency-blocked task is
answered with `dependencyBlockedInteraction`; an accepted interaction
continuation keeps its parked-task and reassignment gates; a user
message continuation runs with its comment; an accepted non-assignee
mention runs; and a reassignment during preparation does not let the
previous assignee restart. Each retry keeps the original `wakeReason`
and records `preparationWait`. With `preserveWakeContext` off, five of
these fail. Restart recovery keeps its attempt credit
(`native-runner-restart-recovery.integration.test.ts`).
- Non-assignee wake: an accepted mention for an agent other than the
assignee meets a busy directory; its retry records
`preparationWaitDeferredWhileAssignee: false`, survives promotion, runs
and succeeds once the directory is free. Without the shared clause in
`isNonAssigneeWorkspaceBusyRetry`, promotion cancels it.
- Interaction continuations: `heartbeat-agent-directory-busy.test.ts`
runs an accepted interaction continuation into a busy directory. The
retry keeps `mutation`, `interactionId`, `interactionKind` and
`interactionStatus` and is still classified as a continuation. If the
task is moved to todo, or reassigned, while the retry waits, the retry
does not run and the task stays parked or with its new owner. With the
old explicit storage/Runner lists, both cases fail.
- Red/green, new run:
`server/src/__tests__/heartbeat-agent-directory-busy.test.ts` runs a
real heartbeat for a new assignment whose first instruction preparation
meets a busy lock. Without the preparation wrapper, the run ends failed
(red). With it, the first run is cancelled as a wait, an
`agent_directory_busy` retry is scheduled with the task still locked to
it, the retry runs and succeeds, and no run of the task is failed
(green).
- Red/green, same-run: with the deferral as a no-op, the same-run
deferral tests failed (`expected false to be true`); they now pass.
- `native-controller-recovery.integration.test.ts` covers repeated
claim-and-defer cycles of a resumed run: each cycle keeps the attempt
count, cannot be claimed before the delay, stores the count on the run,
and the fourth deferral past a bound of 3 is refused. It also covers
Stop after the busy error, a foreign lease owner and a moved task (each
refused, nothing changed), a run before admission, and that a raw busy
error (as from collection) is never deferred.
- Focused suites: 232/232 passed (`agent-directory-working-copies`,
`instruction-working-copy`, `heartbeat-agent-directory-busy`,
`native-controller-recovery.integration`, `heartbeat-retry-scheduling`).
`tsc --noEmit` for `server` reports no errors in the changed files.
- Staging evidence (read-only): the failed run had a graceful restart at
07:25Z, then same-run resumes. Its third attempt failed at 07:31:08Z.
Another run of the same agent recorded a warm checkpoint receipt at
07:31:37Z. A second run of the same agent failed the same way at
07:59:46Z.
## Risks
- A deferred run waits 5 to 15 seconds per deferral, up to 30 times. If
the holder is still busy after that, the run fails as before with a
plain message.
- Before admission, deferral cancels the run and creates a retry run,
like the existing workspace-busy wait.
## Model Used
- Claude Opus 5.5 (claude-opus-5-5), Anthropic, with tool use and code
execution in Claude Code.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
394 lines
27 KiB
TypeScript
394 lines
27 KiB
TypeScript
import { posix } from "node:path";
|
|
import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
|
|
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, nativeRunnerCheckpoints, type Db } from "@paperclipai/db";
|
|
import type { AdapterExecutionTarget } from "@paperclipai/adapter-utils/execution-target";
|
|
import { captureLegacyDurablePrpCheckpoint, parseNativeExecutionInput, type DurablePrpCheckpoint, type PersistedNativeSession } from "../../vendor/paperclip-runner/index.js";
|
|
import { notifyNativeFinalizationWork } from "../native-finalization-notifications.js";
|
|
import { readRunCancellation } from "../run-cancellation.js";
|
|
import { nativeSha256 } from "./canonical.js";
|
|
import { findNativeSessionResumeRun, isNativeSessionId, isUnusedNativeSessionBootstrap, nativeSessionProviderEvidence } from "./native-session-resume.js";
|
|
import { insertMigratedNativeRunnerCheckpoint, loadNativeRunnerCheckpoint } from "./native-runner-checkpoint.js";
|
|
import { offlineMigrationExecutionWhere } from "../offline-migration-execution.js";
|
|
import { redactRunnerApiPayloads } from "./runner-api-log-redaction.js";
|
|
|
|
export const NATIVE_CONTROLLER_RECOVERY_MAX_BYTES = 4 * 1024;
|
|
|
|
/** Private run-profile metadata. No credentials or controller-local paths. */
|
|
export interface NativeControllerRecovery {
|
|
schema: "paperclip.native-controller-recovery.v1";
|
|
snapshotSha256: string;
|
|
configDigest: string;
|
|
processConfigDigest: string | null;
|
|
checkpoint: {
|
|
runId: string;
|
|
runnerInstanceId: string;
|
|
normalizedSessionId: string;
|
|
} | null;
|
|
}
|
|
|
|
export type NativeControllerCompatibility = Pick<NativeControllerRecovery, "configDigest" | "processConfigDigest">;
|
|
|
|
export function createNativeControllerRecovery(
|
|
snapshot: PersistedNativeSession,
|
|
compatibility: NativeControllerCompatibility,
|
|
checkpoint: NativeControllerRecovery["checkpoint"],
|
|
): NativeControllerRecovery {
|
|
const value: NativeControllerRecovery = {
|
|
schema: "paperclip.native-controller-recovery.v1",
|
|
snapshotSha256: nativeSha256(snapshot),
|
|
configDigest: compatibility.configDigest,
|
|
processConfigDigest: compatibility.processConfigDigest,
|
|
checkpoint,
|
|
};
|
|
if (Buffer.byteLength(JSON.stringify(value)) > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES) {
|
|
throw new Error("native_controller_recovery_metadata_too_large");
|
|
}
|
|
return value;
|
|
}
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return value !== null && typeof value === "object" && !Array.isArray(value) ? value as Record<string, unknown> : {};
|
|
}
|
|
|
|
export function readNativeControllerRecovery(value: unknown, snapshot: unknown): NativeControllerRecovery | null {
|
|
const envelope = record(value), checkpoint = record(envelope.checkpoint);
|
|
if (Buffer.byteLength(JSON.stringify(value) ?? "null") > NATIVE_CONTROLLER_RECOVERY_MAX_BYTES
|
|
|| envelope.schema !== "paperclip.native-controller-recovery.v1"
|
|
|| typeof envelope.snapshotSha256 !== "string" || envelope.snapshotSha256 !== nativeSha256(snapshot)
|
|
|| typeof envelope.configDigest !== "string" || !/^sha256:[a-f0-9]{64}$/.test(envelope.configDigest)
|
|
|| !(envelope.processConfigDigest === null || (typeof envelope.processConfigDigest === "string" && /^[a-f0-9]{64}$/.test(envelope.processConfigDigest)))
|
|
|| ![checkpoint.runId, checkpoint.runnerInstanceId, checkpoint.normalizedSessionId].every(v => typeof v === "string" && v.length > 0)) return null;
|
|
return envelope as unknown as NativeControllerRecovery;
|
|
}
|
|
|
|
export type NativeControllerRecoveryOutcome =
|
|
| { kind: "recoverable"; snapshot: PersistedNativeSession; metadata: NativeControllerRecovery; checkpoint: DurablePrpCheckpoint }
|
|
| { kind: "waiting_for_owner"; deadline: Date }
|
|
| { kind: "missing_evidence" | "unsupported" | "dependency_unavailable"; reason: string; cause?: unknown }
|
|
| { kind: "no_prior_execution" };
|
|
|
|
export const NATIVE_RETAINED_OWNER_WAIT = "native_controller_recovery_waiting_for_owner";
|
|
|
|
export class NativeControllerRecoveryWaitingError extends Error {
|
|
constructor(readonly retryAt: Date) {
|
|
super(NATIVE_RETAINED_OWNER_WAIT);
|
|
this.name = "NativeControllerRecoveryWaitingError";
|
|
}
|
|
}
|
|
|
|
/** Admission has not crossed the provider boundary. Retain this same business
|
|
* run on the existing retry owner; never turn an ownership wait into a failure
|
|
* or invent a new provider attempt. The usual resumption gates recheck Stop,
|
|
* permissions and task ownership when this deadline arrives. */
|
|
export async function deferNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; runId: string;
|
|
leaseOwner?: string; retryAt: Date;
|
|
}): Promise<void> {
|
|
await db.transaction(async tx => {
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("update");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId))).for("update");
|
|
const [authority] = await tx.select({ runId: nativeRunnerCheckpoints.runId }).from(nativeRunnerCheckpoints)
|
|
.where(and(eq(nativeRunnerCheckpoints.companyId, input.companyId), eq(nativeRunnerCheckpoints.runId, input.runId))).limit(1);
|
|
const progress = await nativeSessionProviderEvidence(tx as unknown as Db, [input.runId]);
|
|
const cancellation = record(run?.resultJson?.nativeCancellation);
|
|
const stop = readRunCancellation(run?.resultJson);
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|
|
|| !owner || owner.phase !== "observed" || owner.resultId || owner.attempt !== 0
|
|
|| owner.controllerGeneration !== 0 || owner.leaseOwner !== (input.leaseOwner ?? null)
|
|
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|
|
|| run.processPid || run.processGroupId || run.processStartedAt || authority || progress.size
|
|
|| !record(run.runnerProfileJson).nativeExecutionInput || run.resultJson?.startupCancellation
|
|
|| (stop?.source === "operator" && stop.expected)
|
|
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))) {
|
|
throw new Error("native_controller_recovery_wait_authority_changed");
|
|
}
|
|
await notifyNativeFinalizationWork(tx, input);
|
|
await tx.update(nativeRunFinalizations).set({
|
|
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
|
|
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_RETAINED_OWNER_WAIT,
|
|
failureDetail: { originalFailureCode: NATIVE_RETAINED_OWNER_WAIT,
|
|
message: "Waiting for the retained native controller owner to release its lease.",
|
|
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
|
|
providerSessionEstablished: false, providerEventsExist: false },
|
|
updatedAt: new Date(),
|
|
}).where(eq(nativeRunFinalizations.runId, input.runId));
|
|
});
|
|
}
|
|
|
|
export const NATIVE_AGENT_DIRECTORY_BUSY_WAIT = "agent_directory_operation_busy";
|
|
|
|
export async function deferNativeRunForBusyAgentDirectory(db: Db, input: {
|
|
companyId: string; runId: string; leaseOwner?: string; retryAt: Date; maxDeferrals: number;
|
|
}): Promise<"deferred" | "pre_admission" | "refused"> {
|
|
return db.transaction(async tx => {
|
|
const [current] = await tx.select({ issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
|
|
.where(and(eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId)));
|
|
const [existing] = await tx.select({ runId: nativeRunFinalizations.runId }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.runId, input.runId)));
|
|
if (!current?.issueId || !existing) return "pre_admission";
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, current.issueId))).for("update");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, current.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.runId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.nativeIssueId, current.issueId))).for("update");
|
|
const cancellation = record(run?.resultJson?.nativeCancellation);
|
|
const stop = readRunCancellation(run?.resultJson);
|
|
const context = record(run?.contextSnapshot);
|
|
const deferrals = typeof context.agentDirectoryBusyDeferrals === "number" ? context.agentDirectoryBusyDeferrals : 0;
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
if (!run || run.runtimeMode !== "native" || run.status !== "running" || task?.executionRunId !== input.runId
|
|
|| !owner || owner.resultId || owner.phase !== "observed"
|
|
|| owner.leaseOwner !== (input.leaseOwner ?? null)
|
|
|| (input.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt <= new Date(clock.now)))
|
|
|| run.resultJson?.startupCancellation || stop
|
|
|| (["run", "issue"].includes(String(cancellation.scope)) && ["pending", "acknowledged"].includes(String(cancellation.dispatchState)))
|
|
|| deferrals >= input.maxDeferrals) return "refused";
|
|
await notifyNativeFinalizationWork(tx, { companyId: input.companyId, issueId: current.issueId, runId: input.runId });
|
|
await tx.update(heartbeatRuns).set({ contextSnapshot: { ...context, agentDirectoryBusyDeferrals: deferrals + 1 }, updatedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.runId));
|
|
const detail = record(owner.failureDetail);
|
|
await tx.update(nativeRunFinalizations).set({
|
|
phase: "retryable_failure", leaseOwner: null, leaseExpiresAt: null, controlDeadlineAt: null,
|
|
nextAttemptAt: input.retryAt, recoveryState: "resuming_session", failureCode: NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
|
|
failureDetail: {
|
|
recoveryMode: record(run.runnerProfileJson).sessionCheckpoint ? "same_session_resume" : "bootstrap_retry",
|
|
providerSessionEstablished: false, providerEventsExist: false,
|
|
...detail,
|
|
originalFailureCode: detail.originalFailureCode ?? NATIVE_AGENT_DIRECTORY_BUSY_WAIT,
|
|
message: "Waiting for another run of this agent to finish with its files.",
|
|
},
|
|
updatedAt: new Date(),
|
|
}).where(eq(nativeRunFinalizations.runId, input.runId));
|
|
return "deferred";
|
|
});
|
|
}
|
|
|
|
/** The retained Runner keeps its state under the directory its own lease
|
|
* recorded. A workspace can move to a new allocation of the SAME sandbox: its
|
|
* prior epoch was abandoned, so the last save was restored into a fresh root
|
|
* named for the new lease. Both roots are canonical siblings under the
|
|
* sandbox's `.paperclip-workspaces` and carry the same execution-workspace ID.
|
|
* Anything else (another workspace, another parent, a non-canonical path) is
|
|
* not the retained allocation and stays refused. */
|
|
function sameSandboxWorkspaceAllocation(recorded: string, current: string): boolean {
|
|
if (recorded === current) return true;
|
|
const canonical = (path: string) => posix.isAbsolute(path) && posix.normalize(path) === path && !path.endsWith("/");
|
|
if (!canonical(recorded) || !canonical(current)) return false;
|
|
const parent = posix.dirname(recorded);
|
|
if (posix.basename(parent) !== ".paperclip-workspaces" || posix.dirname(current) !== parent) return false;
|
|
const workspace = WORKSPACE_ALLOCATION.exec(posix.basename(recorded))?.[1];
|
|
return Boolean(workspace && WORKSPACE_ALLOCATION.exec(posix.basename(current))?.[1] === workspace);
|
|
}
|
|
|
|
/** The published allocation names (remote-workspaces publishedWorkspaceDirectory):
|
|
* exactly `<workspaceId>`, or `<workspaceId>-<epoch>-<leaseId>`. */
|
|
const UUID = "[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}";
|
|
const WORKSPACE_ALLOCATION = new RegExp(`^(${UUID})(?:-[1-9][0-9]*-${UUID})?$`);
|
|
|
|
/** Bind a read of retained Runner lifecycle to the surviving sandbox and
|
|
* return the directory that Runner wrote under. The ordinary admission owner
|
|
* still checks current permissions before execution. */
|
|
export async function assertNativeRetainedSandboxBinding(db: Db, input: {
|
|
companyId: string; issueId: string; runId: string;
|
|
target: Extract<AdapterExecutionTarget, { kind: "remote" }>;
|
|
}): Promise<{ remoteCwd: string }> {
|
|
const { target } = input;
|
|
if (target.transport !== "sandbox" || target.sandboxLeaseAcquisition?.outcome !== "resumed"
|
|
|| !target.providerKey || !target.environmentId || !target.leaseId) {
|
|
throw new Error("native_controller_recovery_dependency_unavailable:retained_sandbox");
|
|
}
|
|
const [owner] = await db.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId)));
|
|
const leases = await db.select({ metadata: environmentLeases.metadata, status: environmentLeases.status }).from(environmentLeases).where(and(
|
|
eq(environmentLeases.companyId, input.companyId), eq(environmentLeases.issueId, input.issueId),
|
|
eq(environmentLeases.heartbeatRunId, input.runId), eq(environmentLeases.environmentId, target.environmentId),
|
|
eq(environmentLeases.provider, target.providerKey), eq(environmentLeases.providerLeaseId, target.sandboxLeaseAcquisition.providerLeaseId)));
|
|
const recordedCwd = leases.length === 1 ? leases[0].metadata?.remoteCwd : undefined;
|
|
if (!owner || !owner.resultId || owner.recoveryState || typeof recordedCwd !== "string"
|
|
|| !sameSandboxWorkspaceAllocation(recordedCwd, target.remoteCwd)
|
|
|| !["active", "released", "expired"].includes(leases[0].status)) {
|
|
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
|
|
}
|
|
if (owner.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt > new Date()) {
|
|
throw new NativeControllerRecoveryWaitingError(owner.leaseExpiresAt);
|
|
}
|
|
if (owner.leaseOwner || owner.phase !== "committed") {
|
|
throw new Error("native_controller_recovery_missing_evidence:retained_sandbox_binding");
|
|
}
|
|
return { remoteCwd: recordedCwd };
|
|
}
|
|
|
|
/** Only the existing bounded selector chooses history. Never search backwards
|
|
* again when its selected record is incomplete or incompatible. This operation
|
|
* reads evidence; it grants neither command nor acknowledgement authority. */
|
|
export async function resolveNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; normalizedSessionId: string; currentRunId: string;
|
|
}): Promise<NativeControllerRecoveryOutcome> {
|
|
if (!isNativeSessionId(input.normalizedSessionId)) return { kind: "missing_evidence", reason: "session_binding" };
|
|
const [current] = await db.select({ createdAt: heartbeatRuns.createdAt, processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId, processStartedAt: heartbeatRuns.processStartedAt,
|
|
runnerProfileJson: heartbeatRuns.runnerProfileJson }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)));
|
|
if (!current) return { kind: "missing_evidence", reason: "current_binding" };
|
|
const source = await findNativeSessionResumeRun(db, { ...input, beforeCreatedAt: current.createdAt, includeProgressBarrier: true });
|
|
if (!source) {
|
|
// An exhausted candidate window is unknown history, not fresh-bootstrap
|
|
// proof. Read only IDs, and retain the selector's existing 32-row boundary.
|
|
const prior = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
ne(heartbeatRuns.id, input.currentRunId), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).limit(33);
|
|
if (prior.length > 32) return { kind: "missing_evidence", reason: "candidate_window_exhausted" };
|
|
const progress = await nativeSessionProviderEvidence(db, [input.currentRunId]);
|
|
if (!isUnusedNativeSessionBootstrap(current, progress.has(input.currentRunId))) {
|
|
return { kind: "missing_evidence", reason: "current_execution_progress" };
|
|
}
|
|
// The normal admission and remote untouched-session claim still execute.
|
|
return { kind: "no_prior_execution" };
|
|
}
|
|
const [owner] = await db.select({ leaseOwner: nativeRunFinalizations.leaseOwner, leaseExpiresAt: nativeRunFinalizations.leaseExpiresAt }).from(nativeRunFinalizations)
|
|
.where(and(eq(nativeRunFinalizations.runId, source.id), eq(nativeRunFinalizations.companyId, input.companyId),
|
|
eq(nativeRunFinalizations.issueId, input.issueId)));
|
|
if (owner?.leaseOwner && owner.leaseExpiresAt && owner.leaseExpiresAt.getTime() > Date.now()) {
|
|
return { kind: "waiting_for_owner", deadline: owner.leaseExpiresAt };
|
|
}
|
|
if (source.runtimeMode !== "native" || !["succeeded", "failed", "cancelled", "interrupted", "timed_out"].includes(source.status)
|
|
|| source.createdAt > current.createdAt) return { kind: "missing_evidence", reason: "newer_progress_barrier" };
|
|
const profile = record(source.runnerProfileJson), snapshot = record(profile.sessionCheckpoint);
|
|
const identity = record(snapshot.identity);
|
|
if (identity.companyId !== input.companyId || identity.agentId !== input.agentId
|
|
|| identity.issueId !== input.issueId || identity.runId !== source.id || identity.sessionId !== input.normalizedSessionId) {
|
|
return { kind: "missing_evidence", reason: "snapshot_binding" };
|
|
}
|
|
if (record(profile.nativeControllerRecovery).schema !== "paperclip.native-controller-recovery.v1") {
|
|
return { kind: "unsupported", reason: "metadata_format" };
|
|
}
|
|
const metadata = readNativeControllerRecovery(profile.nativeControllerRecovery, snapshot);
|
|
if (!metadata?.checkpoint || metadata.checkpoint.runId !== source.id
|
|
|| metadata.checkpoint.normalizedSessionId !== input.normalizedSessionId
|
|
|| metadata.checkpoint.runnerInstanceId !== source.runnerInstanceId) {
|
|
return { kind: "missing_evidence", reason: "snapshot_digest_or_checkpoint_binding" };
|
|
}
|
|
let checkpoint: DurablePrpCheckpoint | null;
|
|
try {
|
|
checkpoint = await loadNativeRunnerCheckpoint(db, { companyId: input.companyId, issueId: input.issueId, runId: source.id });
|
|
} catch (cause) {
|
|
return { kind: "dependency_unavailable", reason: "checkpoint_authority_unreadable", cause };
|
|
}
|
|
if (!checkpoint || checkpoint.recoveryEvidence.runnerInstanceId !== metadata.checkpoint.runnerInstanceId
|
|
|| checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId) {
|
|
return { kind: "missing_evidence", reason: "checkpoint_binding" };
|
|
}
|
|
return { kind: "recoverable", snapshot: snapshot as unknown as PersistedNativeSession, metadata, checkpoint };
|
|
}
|
|
|
|
/** Bounded internal migration of ONE retained source while its disk is available.
|
|
* The migration owner supplies compatibility computed from the admitted input.
|
|
* This never repairs/replaces the saved snapshot or claims a live execution. */
|
|
export async function importNativeControllerRecovery(db: Db, input: {
|
|
companyId: string; agentId: string; issueId: string; runId: string; normalizedSessionId: string;
|
|
expectedConfigDigest: string;
|
|
supervisor?: unknown;
|
|
offline?: { writersStopped: true; legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
|
|
legacyCheckpoint?: { legacyControllerState?: string; legacyRunnerArtifact?: { version: string; digest: string } };
|
|
}): Promise<{ kind: "imported" | "unchanged" } | Exclude<NativeControllerRecoveryOutcome, { kind: "recoverable" } | { kind: "no_prior_execution" }>> {
|
|
if (input.offline && input.legacyCheckpoint) throw new Error("Conflicting native controller migration authority");
|
|
const legacy = input.offline ?? input.legacyCheckpoint;
|
|
const supervisor = record(input.supervisor);
|
|
if (input.supervisor !== undefined && (supervisor.schema !== "paperclip.native-session-supervisor.v1" || supervisor.configDigest !== input.expectedConfigDigest)) {
|
|
return { kind: "unsupported", reason: "supervisor_compatibility" };
|
|
}
|
|
return db.transaction(async tx => {
|
|
const [task] = await tx.select({ executionRunId: issues.executionRunId, checkoutRunId: issues.checkoutRunId }).from(issues)
|
|
.where(and(eq(issues.companyId, input.companyId), eq(issues.id, input.issueId))).for("share");
|
|
const [owner] = await tx.select().from(nativeRunFinalizations).where(and(
|
|
eq(nativeRunFinalizations.companyId, input.companyId), eq(nativeRunFinalizations.issueId, input.issueId),
|
|
eq(nativeRunFinalizations.runId, input.runId))).for("update");
|
|
if (!task || !owner) return { kind: "missing_evidence" as const, reason: "owner_binding" };
|
|
if (input.offline) {
|
|
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(offlineMigrationExecutionWhere()).limit(1);
|
|
if (!input.offline.writersStopped || active) return { kind: "missing_evidence" as const, reason: "offline_writers_required" };
|
|
}
|
|
const [clock] = await tx.execute<{ now: Date }>(sql`select clock_timestamp() as now`);
|
|
const liveOwner = owner.leaseOwner && (!owner.leaseExpiresAt || owner.leaseExpiresAt > new Date(clock.now));
|
|
if (input.offline ? liveOwner : owner.leaseOwner || task.executionRunId) {
|
|
return owner.leaseExpiresAt && owner.leaseExpiresAt > new Date(clock.now)
|
|
? { kind: "waiting_for_owner" as const, deadline: owner.leaseExpiresAt }
|
|
: { kind: "missing_evidence" as const, reason: "normal_ownership_recovery_required" };
|
|
}
|
|
if (!input.offline && (!(owner.phase === "committed" && owner.resultId) || owner.recoveryState)) {
|
|
return { kind: "missing_evidence" as const, reason: "pending_finalization" };
|
|
}
|
|
if (input.legacyCheckpoint) {
|
|
const [active] = await tx.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(
|
|
eq(heartbeatRuns.companyId, input.companyId), inArray(heartbeatRuns.status, ["queued", "running"]),
|
|
or(eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.issueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId)))).limit(1);
|
|
if (task.checkoutRunId || active) return { kind: "missing_evidence" as const, reason: "legacy_source_active" };
|
|
}
|
|
const [run] = await tx.select().from(heartbeatRuns).where(and(eq(heartbeatRuns.id, input.runId),
|
|
eq(heartbeatRuns.companyId, input.companyId), eq(heartbeatRuns.agentId, input.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, input.issueId), eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId))).for("update");
|
|
const profile = record(run?.runnerProfileJson), snapshot = record(profile.sessionCheckpoint), identity = record(snapshot.identity);
|
|
if (run?.runtimeMode !== "native" || !(run.status === "succeeded" || input.legacyCheckpoint && run.status === "failed"
|
|
|| input.offline && ["failed", "cancelled", "interrupted", "timed_out"].includes(run.status)) || !run.finishedAt
|
|
|| identity.runId !== input.runId || identity.companyId !== input.companyId || identity.issueId !== input.issueId
|
|
|| identity.agentId !== input.agentId || identity.sessionId !== input.normalizedSessionId
|
|
|| input.supervisor !== undefined && nativeSha256(snapshot) !== nativeSha256(redactRunnerApiPayloads(supervisor.snapshot))) {
|
|
return { kind: "missing_evidence" as const, reason: "stale_supervisor_snapshot" };
|
|
}
|
|
let checkpoint = await loadNativeRunnerCheckpoint(tx, input);
|
|
if (input.supervisor === undefined) {
|
|
try {
|
|
const admitted = parseNativeExecutionInput(profile.nativeExecutionInput);
|
|
if (!legacy || (!checkpoint && admitted.session.lifecyclePolicy.mode !== "per_turn")
|
|
|| admitted.session.normalizedSessionId !== input.normalizedSessionId
|
|
|| admitted.binding.runId !== input.runId || admitted.binding.companyId !== input.companyId
|
|
|| admitted.binding.agentId !== input.agentId || admitted.binding.issueId !== input.issueId) {
|
|
return { kind: "missing_evidence" as const, reason: "admitted_per_turn_binding" };
|
|
}
|
|
} catch { return { kind: "unsupported" as const, reason: "admitted_input_format" }; }
|
|
}
|
|
const importing = !checkpoint && legacy?.legacyControllerState !== undefined;
|
|
if (importing) {
|
|
if (!run.runnerInstanceId) return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
|
|
checkpoint = captureLegacyDurablePrpCheckpoint(legacy!.legacyControllerState!, {
|
|
runId: input.runId, runnerInstanceId: run.runnerInstanceId, normalizedSessionId: input.normalizedSessionId,
|
|
}, legacy!.legacyRunnerArtifact);
|
|
}
|
|
if (!checkpoint || checkpoint.recoveryEvidence.normalizedSessionId !== input.normalizedSessionId
|
|
|| checkpoint.recoveryEvidence.runnerInstanceId !== run.runnerInstanceId) {
|
|
return { kind: "missing_evidence" as const, reason: "checkpoint_binding" };
|
|
}
|
|
const metadata = createNativeControllerRecovery(snapshot as unknown as PersistedNativeSession, {
|
|
configDigest: input.expectedConfigDigest,
|
|
processConfigDigest: typeof supervisor.processConfigDigest === "string" ? supervisor.processConfigDigest : null,
|
|
}, { runId: input.runId, runnerInstanceId: checkpoint.recoveryEvidence.runnerInstanceId, normalizedSessionId: input.normalizedSessionId });
|
|
if (!readNativeControllerRecovery(metadata, snapshot)) return { kind: "unsupported" as const, reason: "metadata_format" };
|
|
if (profile.nativeControllerRecovery) {
|
|
if (nativeSha256(profile.nativeControllerRecovery) !== nativeSha256(metadata)) {
|
|
return { kind: "missing_evidence" as const, reason: "existing_metadata_conflict" };
|
|
}
|
|
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
|
|
return { kind: importing ? "imported" as const : "unchanged" as const };
|
|
}
|
|
if (importing) await insertMigratedNativeRunnerCheckpoint(tx, input, checkpoint);
|
|
await tx.update(heartbeatRuns).set({ runnerProfileJson: { ...profile, nativeControllerRecovery: metadata }, updatedAt: new Date() })
|
|
.where(eq(heartbeatRuns.id, input.runId));
|
|
return { kind: "imported" as const };
|
|
});
|
|
}
|