mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-15 10:21:41 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a per-turn remote sandbox saves a provider state backup after its result is accepted. The sandbox is then released, so the backup is the only copy of the provider session. > - When that backup fails to save (for example, #20772: storage throttling), the run keeps its accepted result. Nothing records that its checkpoint can no longer be resumed. > - The next run on the session rebinds that checkpoint and asks the Runner to resume. The Runner has no state and refuses (#20757). With the normal policy, the run opens a replacement session after that failed attempt. With a governed-wait or unfinished-goal checkpoint, the run fails with `native_provider_state_incompatible`, and only Retry or the next message starts fresh. > - The server already knew the state was gone when the backup failed. The failed resume is wasted work. > - This pull request marks the accepted run's checkpoint as not resumable at the moment the backup fails. The next run then starts a fresh provider session directly. > - The benefit is that a lost backup no longer costs a failed run or a failed resume attempt before the task continues. ## Linked Issues or Issue Description Refs #20757 (the missing-state refusal that the next run reaches today). Refs #20772 (the throttled backup upload that loses the backup). Refs #20809 (a related fix for governed-wait follow-ups that reach the missing state; this PR stops most of them from reaching it). **What happened?** 1. A native run in a per-turn sandbox finished, and its result was accepted. 2. `harness_state.backup.persist` failed. The run recorded `native.post_result_failure`. Its run row kept a normal `sessionCheckpoint`. 3. The sandbox was released. 4. The next run on the same session selected that run as its resume source, and `rebindNativeSessionCheckpoint` returned the checkpoint. 5. The Runner found no saved state and refused with `PRP provider resume state is unavailable`. The run then opened a replacement session after the failed attempt, or (governed-wait or unfinished-goal checkpoint) failed with `native_provider_state_incompatible`. Retry or the next message then started fresh. **Expected behavior** The next run starts a fresh provider session at once, because the server already knows the saved state is not there. The accepted result and the task outcome stay as they are. **Steps to reproduce** 1. Give a run row an accepted result and a normal `sessionCheckpoint`. 2. Make its close checkpoint fail in a per-turn sandbox (`reusableLeaseConfigured: false`). 3. Select the next run's resume source with `findNativeSessionResumeRun` and call `rebindNativeSessionCheckpoint`. 4. On `master` it returns the checkpoint, so the next run tries to resume state that does not exist. **Paperclip version or commit** `master` at `556b249897`. ## What Changed - `server/src/services/native-runtime/native-provider-state-unsaved.ts` (new): `markAcceptedRunProviderStateUnsaved` adds `nativeProviderStateUnsaved: { code: "native_provider_state_incompatible", reason: "harness_backup_not_saved" }` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it only when the run's coordinator holds a result with `schema_status = 'accepted'`. `clearRunProviderStateUnsaved` removes it. - `server/src/services/native-runtime/native-session-executor.ts`: the remote close checkpoint (`checkpointRemoteRunner`) now wraps the existing body. When the checkpoint throws in a sandbox without a reusable lease, it calls the marker, writes one run-log line, and throws the original error. A marker or log failure cannot replace that error. If a later checkpoint of the same run saves, it clears the mark. - `server/src/services/native-runtime/native-session-resume.ts`: `rebindNativeSessionCheckpoint` returns `null` for a source with the mark, as it already does for `errorCode = native_provider_state_incompatible`. `findNativeSessionResumeRun` projects the new key. - `doc/execution-semantics.md`: one additive paragraph. - Tests in `native-session-resume.test.ts` and `native-session-executor.test.ts`. Guard conditions. The mark is written only when all of these are true: - The close checkpoint threw before the durable backup was stored (after the upload's own bounded retries). A later local publication, stamp or cleanup failure does not mark the run. - The target is a remote `sandbox` with `reusableLeaseConfigured !== true`. A retained sandbox or an SSH host keeps the live state, so its resume path is unchanged. - The run holds an accepted result. A failure before acceptance changes nothing. Saved state from the previous release: run rows without the key keep today's resume path. An older server that reads a row with the key ignores it, and so it behaves as it does today. No schema change. User-visible: the next run's transcript shows "Paperclip session started" instead of "resumed", with a new session id. The source run's log shows `provider state was not saved after the accepted result; the next run starts a fresh provider session`. The fresh session gets the full task context, as for other fresh starts. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-resume.test.ts -t "not resumable only after|provider state was not saved"`: 9 pass (6 pure, 3 embedded Postgres). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "marks the provider state unsaved|uses the bundled Linux image authority for remote opencode-new-durable"`: 8 pass. They include a durable store failure (marked) and a local publication failure after the store (not marked). - Red on `master` (source files reverted, tests kept): the 3 "starts the next run in a fresh provider session directly" cases fail (`expected {…} to be null`), the DB "accepted" case fails the same way, and the executor per-turn case fails (`mark` called 0 times). - Guard removal checks: - Sandbox guard replaced with `true`: the "retained sandbox" case fails (`mark` called once). - `!checkpointBackupStored` removed: the "publish-failed" case fails. - `schema_status = 'accepted'` removed from the marker: the "rejected" case fails (`expected true to be false`). - Negative cases that pass: no result, rejected result, retained sandbox, records without the key or with another code (all three recovery policies), and a later save that clears the mark. - `cd server && npx tsc --noEmit -p .`: clean. - Not related to this change and failing on `master` in this worktree too: `runtime-selection.test.ts` connector skill cases (missing `.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts` "archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required CI is the authority for those suites. ## Risks - Low. The change only skips a resume that cannot succeed. The fresh session uses the same workspace and full task context, like a Retry after `native_provider_state_incompatible`. - If the backup failed but the provider state were still somewhere recoverable, the next run would lose provider context. For a per-turn sandbox without a reusable lease, the restore path uses only the backup, so this does not apply. Retained sandboxes and SSH hosts are excluded. - One extra conditional `UPDATE` runs only on the failure path. The normal path before `turn.start` has no new waits or work. - `doc/execution-semantics.md` is also edited by #20809. The paragraphs are at different places; whichever merges second rebases. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
746 lines
30 KiB
TypeScript
746 lines
30 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { isDeepStrictEqual } from "node:util";
|
|
import { and, desc, eq, exists, inArray, isNotNull, ne, not, notInArray, or, sql } from "drizzle-orm";
|
|
import {
|
|
heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunnerCheckpoints, nativeRunResults, type Db,
|
|
} from "@paperclipai/db";
|
|
import type {
|
|
NativeExecutionInput,
|
|
PersistedNativeSession,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
export type NativeToolExecutionTargetKind = "local" | "remote";
|
|
|
|
/** Fence live ACPX owners when shared runtime semantics change, even if the
|
|
* underlying native executable/profile did not change. Undefined preserves
|
|
* the existing configuration digest for other provider transports.
|
|
*/
|
|
export function nativeRuntimeContractForProvider(
|
|
provider: NativeExecutionInput["provider"],
|
|
): string | undefined {
|
|
return provider.kind === "acpx" ? "paperclip.acpx-runtime-contract.v1" : undefined;
|
|
}
|
|
|
|
/**
|
|
* Persisted provider threads retain their dynamic-tool declarations. This
|
|
* fingerprint is part of checkpoint compatibility and must change whenever
|
|
* the server-authorized native tool definitions or advertisement policy
|
|
* changes. The execution target remains part of compatibility because local and
|
|
* remote files use different server-bound readers.
|
|
*/
|
|
export function nativeToolContractFingerprintForTarget(
|
|
executionTargetKind: NativeToolExecutionTargetKind,
|
|
providerKind?: NativeExecutionInput["provider"]["kind"],
|
|
): string {
|
|
return `sha256:${createHash("sha256")
|
|
.update(
|
|
JSON.stringify({
|
|
schema: "paperclip.native-tool-contract.v19",
|
|
executionTargetKind,
|
|
advertisementPolicy: {
|
|
// Direct provider threads retain declarations from thread/start.
|
|
// Keep this explicit so changing a tool from conditional to stable
|
|
// advertisement rotates checkpoints even when its schema is unchanged.
|
|
readCurrentWakeComments: "always_advertised_binding_gated.v2",
|
|
historicalChatAttachments:
|
|
"always_advertised_conversation_binding_gated.v1",
|
|
registerDeliverable: "verified_local_or_remote_workspace.v2",
|
|
// Advertised only to Codex runs admitted on an OpenAI API-key
|
|
// connection; the key stays in the server.
|
|
generateImage: "codex_openai_api_key_connection_server_key.v1",
|
|
readChatAttachment: "always_advertised_run_scope_local_staging.v2",
|
|
structuredHumanInput:
|
|
"always_advertised_run_issue_agent_binding_gated_current_task_description.v2",
|
|
semanticCompletion: "finish_accessible_deliverable_evidence.v4",
|
|
...(providerKind === "opencode"
|
|
? { opencodeAssignedMcpTools: "server_gateway_prp_on_demand_exact_names.v2" }
|
|
: {}),
|
|
connectorTools: "assigned_resources_and_pinned_skill_bundle.v1",
|
|
chatTaskTools: "durable_task_binding_and_accepted_origin.v1",
|
|
githubResponses: "bound_working_comment_progress_and_final.v1",
|
|
githubInstructionSkills: "trusted_configuration_native_selection.v1",
|
|
...(executionTargetKind === "remote"
|
|
? { assignedMcpTools: "codex_server_gateway_prp_relay.v1" }
|
|
: {}),
|
|
},
|
|
tools: [
|
|
{ name: "register_deliverable", version: 2 },
|
|
{ name: "generate_image", semanticContract: "paperclip.server-openai-image-generation.v1", version: 1 },
|
|
{
|
|
name: "read_current_wake_comments",
|
|
semanticContract: "paperclip.server-current-wake-comments.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "list_chat_attachments",
|
|
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "reuse_chat_attachment",
|
|
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "read_chat_attachment",
|
|
semanticContract: "paperclip.server-chat-attachment-read.v1",
|
|
version: 1,
|
|
},
|
|
{ name: "request_human_input", version: 1 },
|
|
],
|
|
}),
|
|
)
|
|
.digest("hex")}`;
|
|
}
|
|
|
|
/** Default local-target fingerprint retained for callers and test fixtures. */
|
|
export const NATIVE_TOOL_CONTRACT_FINGERPRINT =
|
|
nativeToolContractFingerprintForTarget("local");
|
|
|
|
/** The runner proved that this run's saved provider session cannot be
|
|
* restored by the current runner build. Its checkpoint is not resumable. */
|
|
export const NATIVE_PROVIDER_STATE_INCOMPATIBLE = "native_provider_state_incompatible";
|
|
/**
|
|
* `runnerProfileJson` key on a run whose result was accepted but whose
|
|
* provider state backup was not saved. Its per-turn sandbox is released after
|
|
* the run, so that backup was the only copy of the provider session. The next
|
|
* run on the session starts a fresh provider session instead of resuming.
|
|
* Records without the key, including every record written before it existed,
|
|
* keep the ordinary resume path.
|
|
*/
|
|
export const NATIVE_PROVIDER_STATE_UNSAVED_KEY = "nativeProviderStateUnsaved";
|
|
|
|
export function isNativeProviderStateUnsaved(profile: unknown): boolean {
|
|
return record(record(profile)[NATIVE_PROVIDER_STATE_UNSAVED_KEY]).code === NATIVE_PROVIDER_STATE_INCOMPATIBLE;
|
|
}
|
|
/** The saved session owned a side-effecting call without a delivered result.
|
|
* Recovery requires an explicit reconciliation of that action. */
|
|
export const NATIVE_PROVIDER_SIDE_EFFECT_PENDING = "native_provider_side_effect_pending";
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
export function isNativeSessionId(value: unknown): value is string {
|
|
return (
|
|
typeof value === "string" &&
|
|
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(
|
|
value,
|
|
)
|
|
);
|
|
}
|
|
|
|
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
|
|
"succeeded",
|
|
"interrupted",
|
|
"failed",
|
|
"cancelled",
|
|
"timed_out",
|
|
]);
|
|
|
|
export type NativeSessionBootstrapState = {
|
|
processPid: number | null;
|
|
processGroupId: number | null;
|
|
processStartedAt: Date | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
|
|
export function isUnusedNativeSessionBootstrap(
|
|
run: NativeSessionBootstrapState,
|
|
hasProviderEvents: boolean,
|
|
): boolean {
|
|
return (
|
|
run.processPid === null &&
|
|
run.processGroupId === null &&
|
|
run.processStartedAt === null &&
|
|
record(run.runnerProfileJson).sessionCheckpoint == null &&
|
|
!hasProviderEvents
|
|
);
|
|
}
|
|
|
|
/** Newest-first rows for one exact task/session; never skip provider progress. */
|
|
export function selectNativeSessionResumeRun<
|
|
T extends NativeSessionBootstrapState & {
|
|
id: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
nativeIssueId: string | null;
|
|
nativeSessionId: string | null;
|
|
runtimeMode: string | null;
|
|
status: string;
|
|
createdAt: Date;
|
|
},
|
|
>(input: {
|
|
runs: readonly T[];
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
normalizedSessionId: string;
|
|
currentRunId: string;
|
|
beforeCreatedAt: Date;
|
|
providerEvidenceRunIds: ReadonlySet<string>;
|
|
}): T | null {
|
|
for (const run of input.runs) {
|
|
if (
|
|
run.id === input.currentRunId ||
|
|
run.companyId !== input.companyId ||
|
|
run.agentId !== input.agentId ||
|
|
run.nativeIssueId !== input.issueId ||
|
|
run.nativeSessionId !== input.normalizedSessionId
|
|
)
|
|
continue;
|
|
if (
|
|
run.runtimeMode !== "native" ||
|
|
!LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(run.status)
|
|
)
|
|
return null;
|
|
// Even an incompatible checkpoint is a progress barrier. The caller must
|
|
// reject it or start a fresh session, not fall back to an older checkpoint.
|
|
if (record(run.runnerProfileJson).sessionCheckpoint != null)
|
|
return run.createdAt > input.beforeCreatedAt ? null : run;
|
|
if (
|
|
!isUnusedNativeSessionBootstrap(
|
|
run,
|
|
input.providerEvidenceRunIds.has(run.id),
|
|
)
|
|
)
|
|
return null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
|
|
"harness.ready",
|
|
"session.started",
|
|
"session.resumed",
|
|
"session.updated",
|
|
"turn.started",
|
|
"provider.event",
|
|
"provider.rpc_result",
|
|
];
|
|
|
|
export async function nativeSessionProviderEvidence(
|
|
db: Pick<Db, "select">,
|
|
runIds: string[],
|
|
): Promise<Set<string>> {
|
|
if (!runIds.length) return new Set();
|
|
const rows = await db
|
|
.select({ runId: heartbeatRunEvents.runId })
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
inArray(heartbeatRunEvents.runId, runIds),
|
|
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
|
|
),
|
|
)
|
|
.groupBy(heartbeatRunEvents.runId);
|
|
return new Set(rows.map((row) => row.runId));
|
|
}
|
|
|
|
type NativeSessionResumeSelection = {
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
normalizedSessionId: string;
|
|
currentRunId?: string;
|
|
beforeCreatedAt: Date;
|
|
/** Recovery discovery must retain an ineligible newest progress barrier. */
|
|
includeProgressBarrier?: boolean;
|
|
};
|
|
|
|
const NATIVE_RESUME_CANDIDATE_WINDOW = 32;
|
|
const INHERITED_CHECKPOINT_SOURCE_FIELDS = ["backendKind", "driverKind", "sessionId", "providerSessionId"] as const;
|
|
|
|
export async function findNativeSessionResumeRun(db: Db, input: NativeSessionResumeSelection) {
|
|
if (!isNativeSessionId(input.normalizedSessionId)) return null;
|
|
const barrier = await selectNativeSessionResumeBarrier(db, input, []);
|
|
// A follow-up that failed before any provider or controller work holds only
|
|
// the checkpoint it inherited. Select its source instead, but only when
|
|
// every skipped run proves that and inherited exactly that source session.
|
|
const skipped: NonNullable<typeof barrier>[] = [];
|
|
let selected = barrier;
|
|
while (selected && skipped.length < NATIVE_RESUME_CANDIDATE_WINDOW &&
|
|
await isUnstartedInheritedFollowUp(db, input, selected)) {
|
|
skipped.push(selected);
|
|
selected = await selectNativeSessionResumeBarrier(db, input, skipped.map(run => run.id));
|
|
}
|
|
if (!skipped.length) return selected;
|
|
const source = record(selected?.runnerProfileJson.sessionCheckpoint);
|
|
const inherited = selected !== null && typeof source.sessionId === "string" && skipped.every(run => {
|
|
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
|
|
return INHERITED_CHECKPOINT_SOURCE_FIELDS.every(field => checkpoint[field] === source[field]);
|
|
});
|
|
return inherited ? selected : barrier;
|
|
}
|
|
|
|
/** The run row, its coordinator, and the absence of controller checkpoints,
|
|
* results and provider events together prove no provider work started. */
|
|
async function isUnstartedInheritedFollowUp(
|
|
db: Db,
|
|
input: NativeSessionResumeSelection,
|
|
run: { id: string; runtimeMode: string | null; status: string; processPid: number | null;
|
|
processGroupId: number | null; processStartedAt: Date | null; runnerProfileJson: { sessionCheckpoint: unknown; nativeControllerRecovery: unknown } },
|
|
): Promise<boolean> {
|
|
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
|
|
const identity = record(checkpoint.identity);
|
|
if (run.runtimeMode !== "native" || run.status !== "failed" || run.processPid !== null ||
|
|
run.processGroupId !== null || run.processStartedAt !== null || run.runnerProfileJson.nativeControllerRecovery != null ||
|
|
identity.runId !== run.id || identity.sessionId !== input.normalizedSessionId || checkpoint.cursor !== null) return false;
|
|
const [evidence] = await db.select({
|
|
executionRecovery: sql<unknown>`${heartbeatRuns.resultJson}->'executionRecovery'`,
|
|
phase: nativeRunFinalizations.phase,
|
|
resultId: nativeRunFinalizations.resultId,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner,
|
|
controllerPid: nativeRunFinalizations.controllerPid,
|
|
controllerCheckpoint: sql<boolean>`exists (select 1 from ${nativeRunnerCheckpoints}
|
|
where ${nativeRunnerCheckpoints.companyId} = ${heartbeatRuns.companyId} and ${nativeRunnerCheckpoints.runId} = ${heartbeatRuns.id})`,
|
|
result: sql<boolean>`exists (select 1 from ${nativeRunResults}
|
|
where ${nativeRunResults.companyId} = ${heartbeatRuns.companyId} and ${nativeRunResults.runId} = ${heartbeatRuns.id})`,
|
|
}).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and(
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.issueId, input.issueId),
|
|
)).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId)));
|
|
const recovery = record(evidence?.executionRecovery);
|
|
return Boolean(evidence && recovery.kind === "bootstrap" && recovery.providerWorkStarted === false &&
|
|
evidence.phase === "terminal_failure" && evidence.resultId === null && evidence.leaseOwner === null &&
|
|
evidence.controllerPid === null && !evidence.controllerCheckpoint && !evidence.result &&
|
|
!(await nativeSessionProviderEvidence(db, [run.id])).has(run.id));
|
|
}
|
|
|
|
async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResumeSelection, skippedRunIds: string[]) {
|
|
const candidates = db.$with("native_resume_candidates").as(db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt,
|
|
hasCheckpoint: sql<boolean>`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' <> 'null'::jsonb, false)`.as("has_checkpoint"),
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
input.currentRunId ? ne(heartbeatRuns.id, input.currentRunId) : undefined,
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id))
|
|
.limit(NATIVE_RESUME_CANDIDATE_WINDOW));
|
|
const terminalStatuses = [...LEGACY_RETRY_SOURCE_TERMINAL_STATUSES];
|
|
const barrier = db.$with("native_resume_barrier").as(db
|
|
.select()
|
|
.from(candidates)
|
|
.where(and(skippedRunIds.length ? notInArray(candidates.id, skippedRunIds) : undefined, or(
|
|
ne(candidates.runtimeMode, "native"),
|
|
not(inArray(candidates.status, terminalStatuses)),
|
|
candidates.hasCheckpoint,
|
|
isNotNull(candidates.processPid),
|
|
isNotNull(candidates.processGroupId),
|
|
isNotNull(candidates.processStartedAt),
|
|
exists(db.select({ runId: heartbeatRunEvents.runId })
|
|
.from(heartbeatRunEvents)
|
|
.where(and(
|
|
eq(heartbeatRunEvents.runId, candidates.id),
|
|
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
|
|
))),
|
|
)))
|
|
.orderBy(desc(candidates.createdAt), desc(candidates.id))
|
|
.limit(1));
|
|
const [run] = await db.with(candidates, barrier)
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt,
|
|
runnerProfileJson: sql<{
|
|
nativeToolContractFingerprint: unknown;
|
|
nativeExecutionInput: unknown;
|
|
sessionCheckpoint: unknown;
|
|
nativeControllerRecovery: unknown;
|
|
nativeProviderStateUnsaved?: unknown;
|
|
}>`jsonb_build_object(
|
|
'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint',
|
|
'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput',
|
|
'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint',
|
|
'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery',
|
|
'${sql.raw(NATIVE_PROVIDER_STATE_UNSAVED_KEY)}', ${heartbeatRuns.runnerProfileJson}->'${sql.raw(NATIVE_PROVIDER_STATE_UNSAVED_KEY)}'
|
|
)`,
|
|
contextSnapshot: sql<{
|
|
paperclipWake: unknown;
|
|
paperclipIssue: unknown;
|
|
}>`jsonb_build_object(
|
|
'paperclipWake', ${heartbeatRuns.contextSnapshot}->'paperclipWake',
|
|
'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object'
|
|
then (select coalesce(jsonb_object_agg(brief.key, brief.value), '{}'::jsonb)
|
|
from jsonb_each(${heartbeatRuns.contextSnapshot}->'paperclipIssue') as brief
|
|
where brief.key in ('id', 'title', 'description')) else ${heartbeatRuns.contextSnapshot}->'paperclipIssue' end
|
|
)`,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.innerJoin(barrier, eq(heartbeatRuns.id, barrier.id))
|
|
.where(and(
|
|
...(input.includeProgressBarrier ? [] : [
|
|
eq(barrier.runtimeMode, "native"),
|
|
inArray(barrier.status, terminalStatuses),
|
|
eq(barrier.hasCheckpoint, true),
|
|
sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`,
|
|
]),
|
|
sql`${heartbeatRuns.companyId}::text = ${input.companyId}`,
|
|
sql`${heartbeatRuns.agentId}::text = ${input.agentId}`,
|
|
sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`,
|
|
sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`,
|
|
));
|
|
return run ?? null;
|
|
}
|
|
|
|
/** A preassigned session id may rotate only before immutable native admission. */
|
|
export function nativeSessionIdForBootstrapPersistence(input: {
|
|
run: NativeSessionBootstrapState & { nativeSessionId: string | null };
|
|
selectedSessionId: string;
|
|
hasProviderEvents: boolean;
|
|
}): string {
|
|
if (input.run.nativeSessionId === input.selectedSessionId)
|
|
return input.selectedSessionId;
|
|
if (
|
|
record(input.run.runnerProfileJson).nativeExecutionInput !== undefined ||
|
|
!isUnusedNativeSessionBootstrap(input.run, input.hasProviderEvents)
|
|
) {
|
|
throw new Error("native_session_bootstrap_identity_conflict");
|
|
}
|
|
return input.selectedSessionId;
|
|
}
|
|
|
|
/** Caller holds the run row lock; recheck authority after waiting for that lock. */
|
|
export async function prepareNativeSessionBootstrapPersistence(
|
|
db: Pick<Db, "select">,
|
|
input: {
|
|
run: NativeSessionBootstrapState & {
|
|
id: string;
|
|
nativeSessionId: string | null;
|
|
};
|
|
selectedSessionId: string;
|
|
execution: NativeExecutionInput;
|
|
restoringCheckpoint: boolean;
|
|
},
|
|
) {
|
|
const hasProviderEvents = (
|
|
await nativeSessionProviderEvidence(db, [input.run.id])
|
|
).has(input.run.id);
|
|
const profile = record(input.run.runnerProfileJson);
|
|
if (
|
|
profile.nativeExecutionInput !== undefined &&
|
|
!isDeepStrictEqual(
|
|
parseNativeExecutionInput(profile.nativeExecutionInput),
|
|
input.execution,
|
|
)
|
|
) {
|
|
throw new Error("native_execution_input_persisted_binding_mismatch");
|
|
}
|
|
if (
|
|
input.restoringCheckpoint &&
|
|
!isUnusedNativeSessionBootstrap(input.run, hasProviderEvents)
|
|
) {
|
|
throw new Error("native_session_bootstrap_identity_conflict");
|
|
}
|
|
return nativeSessionIdForBootstrapPersistence({
|
|
...input,
|
|
hasProviderEvents,
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Legacy compatibility is deliberately narrower than ordinary task-session
|
|
* continuation: only a replacement row that never acquired any native process
|
|
* or provider authority may be rebound to a terminal native source. The exact
|
|
* checkpoint/session/workspace/provider binding is validated separately by
|
|
* rebindNativeSessionCheckpoint.
|
|
*/
|
|
export function isUnusedLegacyNativeRetryReplacement(input: {
|
|
replacement: {
|
|
processPid: number | null;
|
|
processGroupId: number | null;
|
|
processStartedAt: Date | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
source: {
|
|
runtimeMode: string | null;
|
|
status: string;
|
|
nativeSessionId: string | null;
|
|
} | null;
|
|
hasProviderEvents: boolean;
|
|
}): boolean {
|
|
return Boolean(
|
|
input.source?.runtimeMode === "native" &&
|
|
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
|
|
isNativeSessionId(input.source.nativeSessionId) &&
|
|
isUnusedNativeSessionBootstrap(input.replacement, input.hasProviderEvents),
|
|
);
|
|
}
|
|
|
|
function sameProvider(
|
|
previous: NativeExecutionInput["provider"],
|
|
current: NativeExecutionInput["provider"],
|
|
): boolean {
|
|
return JSON.stringify(previous) === JSON.stringify(current);
|
|
}
|
|
|
|
function nativeCheckpointWorkspaceScope(
|
|
execution: NativeExecutionInput,
|
|
owningRunId: string,
|
|
) {
|
|
// Projectless work uses each heartbeat run id as a placeholder workspace id.
|
|
// Continuity is safe only when both sides independently prove that shape and
|
|
// their complete immutable workspace descriptors match. A real managed
|
|
// workspace instead binds continuity to its durable workspace row id.
|
|
return execution.binding.executionWorkspaceId === owningRunId
|
|
? {
|
|
kind: "transient" as const,
|
|
cwd: execution.workspace.cwd,
|
|
repoUrl: execution.workspace.repoUrl,
|
|
repoRef: execution.workspace.repoRef,
|
|
branchName: execution.workspace.branchName,
|
|
}
|
|
: {
|
|
kind: "managed" as const,
|
|
id: execution.binding.executionWorkspaceId,
|
|
};
|
|
}
|
|
|
|
function sameWorkspaceScope(input: {
|
|
previousExecution: NativeExecutionInput;
|
|
previousRunId: string;
|
|
currentExecution: NativeExecutionInput;
|
|
}): boolean {
|
|
return (
|
|
JSON.stringify(
|
|
nativeCheckpointWorkspaceScope(
|
|
input.previousExecution,
|
|
input.previousRunId,
|
|
),
|
|
) ===
|
|
JSON.stringify(
|
|
nativeCheckpointWorkspaceScope(
|
|
input.currentExecution,
|
|
input.currentExecution.binding.runId,
|
|
),
|
|
)
|
|
);
|
|
}
|
|
|
|
/** A resume delta is valid only if the provider checkpoint really can be used. */
|
|
export function buildNativeExecutionWithCheckpoint(input: {
|
|
previousRun:
|
|
Parameters<typeof rebindNativeSessionCheckpoint>[0]["previousRun"] | null;
|
|
normalizedSessionId: string;
|
|
executionTargetKind?: NativeToolExecutionTargetKind;
|
|
toolRefreshOnResume?: boolean;
|
|
refreshTools?: boolean;
|
|
buildExecution: (options: {
|
|
normalizedSessionId: string;
|
|
resumedSession: boolean;
|
|
}) => NativeExecutionInput;
|
|
}): {
|
|
execution: NativeExecutionInput;
|
|
checkpoint: PersistedNativeSession | null;
|
|
normalizedSessionId: string;
|
|
} {
|
|
const execution = input.buildExecution({
|
|
normalizedSessionId: input.normalizedSessionId,
|
|
resumedSession: input.previousRun !== null,
|
|
});
|
|
// A presentation-only upgrade must not rotate a healthy provider session or
|
|
// lose a durable goal. Keep its v4/v5 format until that session naturally ends.
|
|
// All ordinary identity, workspace, provider, tool and recovery checks still apply.
|
|
if (input.previousRun && (execution.schema === "paperclip.native-execution-input.v4" || execution.schema === "paperclip.native-execution-input.v5")) {
|
|
const previousSchema = record(record(input.previousRun.runnerProfileJson).nativeExecutionInput).schema;
|
|
if ((previousSchema === "paperclip.native-execution-input.v4" || previousSchema === "paperclip.native-execution-input.v5") && previousSchema !== execution.schema) {
|
|
const { completionSources: _sources, ...common } = execution as typeof execution & { completionSources?: unknown };
|
|
const retainedFormat = parseNativeExecutionInput({ ...common, schema: previousSchema });
|
|
const retainedCheckpoint = rebindNativeSessionCheckpoint({
|
|
previousRun: input.previousRun,
|
|
currentExecution: retainedFormat,
|
|
executionTargetKind: input.executionTargetKind,
|
|
toolRefreshOnResume: input.toolRefreshOnResume,
|
|
refreshTools: input.refreshTools,
|
|
});
|
|
if (retainedCheckpoint) return { execution: retainedFormat, checkpoint: retainedCheckpoint, normalizedSessionId: input.normalizedSessionId };
|
|
}
|
|
}
|
|
const checkpoint = input.previousRun
|
|
? rebindNativeSessionCheckpoint({
|
|
previousRun: input.previousRun,
|
|
currentExecution: execution,
|
|
executionTargetKind: input.executionTargetKind,
|
|
toolRefreshOnResume: input.toolRefreshOnResume,
|
|
refreshTools: input.refreshTools,
|
|
})
|
|
: null;
|
|
if (checkpoint)
|
|
return {
|
|
execution,
|
|
checkpoint,
|
|
normalizedSessionId: input.normalizedSessionId,
|
|
};
|
|
// Rebuild both task context and wake instructions. Merely rotating the ID
|
|
// leaves a fresh provider with a compact delta and missing task context.
|
|
const normalizedSessionId = randomUUID();
|
|
return {
|
|
execution: input.buildExecution({
|
|
normalizedSessionId,
|
|
resumedSession: false,
|
|
}),
|
|
checkpoint: null,
|
|
normalizedSessionId,
|
|
};
|
|
}
|
|
|
|
/**
|
|
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
|
|
* The provider/driver session identity is retained, while every per-turn and
|
|
* per-event field is reset so the new run starts one clean turn via resume.
|
|
*/
|
|
export function rebindNativeSessionCheckpoint(input: {
|
|
previousRun: {
|
|
id: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
nativeSessionId: string | null;
|
|
errorCode?: string | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
currentExecution: NativeExecutionInput;
|
|
executionTargetKind?: NativeToolExecutionTargetKind;
|
|
toolRefreshOnResume?: boolean;
|
|
refreshTools?: boolean;
|
|
}): PersistedNativeSession | null {
|
|
if (input.refreshTools && input.toolRefreshOnResume !== true) return null;
|
|
// The source is still the progress barrier. Its provider session cannot be
|
|
// restored, so the next run starts a fresh session in the same workspace.
|
|
if (input.previousRun.errorCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE) return null;
|
|
// The saved session still holds the unsettled call and refuses attachment.
|
|
// Generic Retry is blocked until an operator reconciles that action; the
|
|
// reconciled continuation then starts a fresh session in the same workspace.
|
|
if (input.previousRun.errorCode === NATIVE_PROVIDER_SIDE_EFFECT_PENDING) return null;
|
|
const previousProfile = record(input.previousRun.runnerProfileJson);
|
|
// The source's result was accepted, but its provider state was never saved
|
|
// and its sandbox is gone. Resuming would only fail, so start fresh now.
|
|
if (isNativeProviderStateUnsaved(previousProfile)) return null;
|
|
if (
|
|
previousProfile.nativeToolContractFingerprint !==
|
|
nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local", input.currentExecution.provider.kind)
|
|
) {
|
|
return null;
|
|
}
|
|
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
|
|
const checkpointIdentity = record(rawCheckpoint.identity);
|
|
const current = input.currentExecution;
|
|
const normalizedSessionId = current.session.normalizedSessionId;
|
|
if (
|
|
!isNativeSessionId(normalizedSessionId) ||
|
|
input.previousRun.companyId !== current.binding.companyId ||
|
|
input.previousRun.agentId !== current.binding.agentId ||
|
|
input.previousRun.nativeSessionId !== normalizedSessionId ||
|
|
typeof rawCheckpoint.sessionId !== "string" ||
|
|
checkpointIdentity.runId !== input.previousRun.id ||
|
|
checkpointIdentity.companyId !== current.binding.companyId ||
|
|
checkpointIdentity.issueId !== current.binding.issueId ||
|
|
checkpointIdentity.agentId !== current.binding.agentId ||
|
|
checkpointIdentity.sessionId !== normalizedSessionId
|
|
)
|
|
return null;
|
|
|
|
let previousExecution: NativeExecutionInput;
|
|
try {
|
|
previousExecution = parseNativeExecutionInput(
|
|
previousProfile.nativeExecutionInput,
|
|
);
|
|
} catch {
|
|
return null;
|
|
}
|
|
if (
|
|
previousExecution.binding.runId !== input.previousRun.id ||
|
|
previousExecution.binding.companyId !== current.binding.companyId ||
|
|
previousExecution.binding.issueId !== current.binding.issueId ||
|
|
previousExecution.binding.agentId !== current.binding.agentId ||
|
|
previousExecution.session.normalizedSessionId !== normalizedSessionId ||
|
|
previousExecution.session.driverKind !== current.session.driverKind ||
|
|
!sameWorkspaceScope({
|
|
previousExecution,
|
|
previousRunId: input.previousRun.id,
|
|
currentExecution: current,
|
|
}) ||
|
|
previousExecution.task.workMode !== current.task.workMode ||
|
|
("executionMode" in previousExecution
|
|
? previousExecution.executionMode
|
|
: "default") !==
|
|
("executionMode" in current ? current.executionMode : "default") ||
|
|
!sameProvider(previousExecution.provider, current.provider) ||
|
|
previousExecution.schema !== current.schema ||
|
|
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
|
|
("runtimeContext" in previousExecution &&
|
|
"runtimeContext" in current &&
|
|
previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest &&
|
|
!(input.toolRefreshOnResume === true &&
|
|
canonicalNativeRuntimeContextDigest({
|
|
...previousExecution.runtimeContext,
|
|
mcp: current.runtimeContext.mcp,
|
|
}) === current.runtimeContext.aggregateDigest))
|
|
)
|
|
return null;
|
|
|
|
const priorSemanticResult = record(rawCheckpoint.semanticResult);
|
|
const priorContinuation = record(priorSemanticResult.continuation);
|
|
const rawGoal = rawCheckpoint.goal;
|
|
const hasUnfinishedGoal =
|
|
rawGoal !== null &&
|
|
rawGoal !== undefined &&
|
|
record(rawGoal).status !== "complete";
|
|
const providerRecoveryPolicy = hasUnfinishedGoal
|
|
? ("same_session_only" as const)
|
|
: !input.refreshTools && priorSemanticResult.reportedWorkDisposition === "yielded" &&
|
|
priorContinuation.kind === "response_wake"
|
|
? ("allow_replacement_after_governed_wait" as const)
|
|
: ("allow_replacement_after_resume_failure" as const);
|
|
|
|
return {
|
|
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
|
|
identity: {
|
|
runId: current.binding.runId,
|
|
sessionId: normalizedSessionId,
|
|
companyId: current.binding.companyId,
|
|
issueId: current.binding.issueId,
|
|
agentId: current.binding.agentId,
|
|
},
|
|
cursor: null,
|
|
semanticResult: null,
|
|
terminal: null,
|
|
activeTurnId: null,
|
|
terminalTurns: [],
|
|
pendingRuntimeRequests: [],
|
|
dispositionOnlyRecoveryConsumed: false,
|
|
dispositionOnlyRecoveryTurnId: null,
|
|
// Settlement evidence belongs to the prior heartbeat run, even when the
|
|
// provider session is reused for the user's response.
|
|
governedWait: undefined,
|
|
providerRecoveryPolicy,
|
|
};
|
|
}
|