Files
PaperClipAI/server/src/services/native-runtime/native-session-resume.ts
T
Devin FoleyandPaperclip b0c4e34526 fix(native): resume from the owner after an unstarted failed follow-up (#18860)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner sessions keep one conversation across task turns.
> - A follow-up turn resumes from the newest earlier run that holds a
provider checkpoint.
> - A follow-up can fail during admission, before provider work, and
still hold the checkpoint that it inherited.
> - A later Retry then resumes from that failed run. Controller recovery
finds no controller checkpoint there and fails.
> - This pull request makes resume source selection skip a failed
follow-up that proves that it never started provider work.
> - The benefit is that a board Retry resumes from the last committed
owner, and the task can continue.

## Linked Issues or Issue Description

Refs #18846, which lets a follow-up start after a recovered per-turn
owner. Refs #18854 (same-run retry of failures before provider work) and
#18718 (controller projection refresh).

**What happened?**

Run 1 was a per-turn remote sandbox run. Recovery committed its result,
but it saved no Runner suspension receipt. Run 2 was a follow-up that
was admitted before #18846. Admission failed with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`, before any provider work. Run
2 has these properties:

- Status `failed`, and the result has `executionRecovery: { kind:
"bootstrap", providerWorkStarted: false }`.
- The coordinator is `terminal_failure`, with no result, no lease owner,
and no controller process.
- No controller checkpoint rows and no controller recovery pointer.
- A `sessionCheckpoint` bound to run 2 with a null cursor. The provider
session in it is copied from run 1.

A board Retry then started run 3. Resume source selection treats any
checkpoint as a progress barrier, so it selected run 2. Controller
recovery then failed with
`native_controller_recovery_unsupported:metadata_format`, because run 2
has no controller recovery pointer. The task stayed blocked. Every later
Retry selects the newest failed follow-up and fails the same way.

**Expected behavior**

A Retry must resume from the last committed owner (run 1) when the newer
follow-up provably never started provider work. If the follow-up might
have started provider work, or holds its own controller evidence, it
must stay the source.

**Steps to reproduce**

1. Commit a per-turn remote owner through recovery, without a suspension
receipt.
2. Admit a follow-up, and make its admission fail with
`prior_owner_terminal_state_indeterminate`.
3. Settle that follow-up as a failed bootstrap: `providerWorkStarted:
false`, `terminal_failure`, and no result.
4. Admit a Retry. It selects the failed follow-up and fails with
`native_controller_recovery_unsupported:metadata_format`.

The new regression test runs these steps against PostgreSQL.

**Paperclip version or commit**

Reproduced on `909d86b59a`.

**Deployment mode**

Native Runner with a per-turn lifecycle on the built-in remote sandbox
transport. Source selection is shared by all native providers.

## What Changed

- `findNativeSessionResumeRun` skips a newest barrier that is an
unstarted inherited follow-up and selects again without it. The selector
repeats this inside the existing 32-run window.
- A run is skipped only when all of these are true:
- It is a native run with status `failed` and a `bootstrap` receipt with
`providerWorkStarted: false`.
- Its coordinator is `terminal_failure`, with no result, lease owner, or
controller process.
- It has no process identity, provider event, controller checkpoint row,
run result row, or controller recovery pointer.
- Its checkpoint is bound to this run and session, and it has no cursor.
- After the skips, every skipped run must name the same backend, driver,
and provider session as the selected source. Otherwise the selector
returns the original barrier.
- Runtime selection and controller recovery both call this selector.
Both now use the committed owner. Admission then reaches the #18846
path, which retires the old authority and starts a new Runner.
- Add four cases to `native-recovered-owner-followup.test.ts`, with one
negative control for each guard.
- Document the rule in `doc/execution-semantics.md` and the test in
`doc/testing-native-remote-continuity.md`.

### Why source selection

- Source selection made the wrong choice. The failed follow-up has no
provider progress. Its checkpoint is a copy, and the provider never saw
its turn.
- One shared selector feeds runtime selection (checkpoint rebind,
previous turn, runner instance) and controller recovery. A fix in only
one of them would let them disagree about the source.
- Controller recovery keeps its rule: it never searches backward when
the selected run is incomplete. The selector skips a run only with
positive proof, so an incomplete run with any evidence of work still
stops recovery.
- A fix in the executor could not help. The executor sees only the
source that the selector gives it.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-recovered-owner-followup.test.ts`: 22
tests pass.
- Before the change, the two new retry tests failed with
`native_controller_recovery_unsupported:metadata_format`.
- After the change, the retry selects the committed owner and is
admitted. It is also admitted after two failed follow-ups.
- Eight controls keep the failed follow-up as the source: process
identity, provider event, controller checkpoint row, a receipt with
`providerWorkStarted: true`, a coordinator with a controller process, a
run result row, a different provider session, and a non-null cursor.
- I disabled each guard in turn. Each change made its matching control
fail.
- These suites pass: native session resume, resume projection,
controller recovery (unit and integration), native startup retry,
runtime selection, execution workspace service, native status corpus,
task workspace binding, and all 1,088 native executor tests. The
container and Runner binary cases skip locally.
- Server typecheck, `node scripts/typecheck-low-memory.mjs`, `pnpm
check:module-boundaries`, and `git diff --check` pass.
- I did not run the full `pnpm test:run`, `pnpm build`, or the opt-in
Linux Runner container cases locally. CI runs them.
- The tests start at the selector and executor admission. They do not
exercise the Retry route, the task retry queue, or a live sandbox
provider.

## Risks

- A wrong skip could resume from an older checkpoint. Each guard
requires positive evidence of no provider work. A skipped run must also
name the same provider session as the selected source. If any check
fails, the selector returns the original barrier.
- Only `failed` runs qualify. Cancelled, interrupted, and timed-out runs
keep their existing handling.
- The skip adds one proof query per candidate. It runs only when the
newest barrier is a failed native run whose checkpoint is bound to
itself and has no cursor.
- The rebind now uses the owner's tool-contract fingerprint. If the tool
contract changed since the owner ran, the Retry starts a new provider
session. The provider history in this failure mode is already
unavailable.
- No migration or configuration change is required.

## Model Used

- Anthropic Claude Opus 5.5 (model ID `claude-opus-5-5`) in Claude Code,
with extended thinking, repository tools, shell and code execution, and
local test runs. The runtime does not expose the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 14:02:36 -08:00

706 lines
28 KiB
TypeScript

import { createHash, randomUUID } from "node:crypto";
import { isDeepStrictEqual } from "node:util";
import { and, desc, eq, exists, inArray, isNotNull, ne, not, notInArray, or, sql } from "drizzle-orm";
import {
heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunnerCheckpoints, nativeRunResults, type Db,
} from "@paperclipai/db";
import type {
NativeExecutionInput,
PersistedNativeSession,
} from "../../vendor/paperclip-runner/index.js";
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
export type NativeToolExecutionTargetKind = "local" | "remote";
/** Fence live ACPX owners when shared runtime semantics change, even if the
* underlying native executable/profile did not change. Undefined preserves
* the existing configuration digest for other provider transports.
*/
export function nativeRuntimeContractForProvider(
provider: NativeExecutionInput["provider"],
): string | undefined {
return provider.kind === "acpx" ? "paperclip.acpx-runtime-contract.v1" : undefined;
}
/**
* Persisted provider threads retain their dynamic-tool declarations. This
* fingerprint is part of checkpoint compatibility and must change whenever
* the server-authorized native tool definitions or advertisement policy
* changes. The execution target remains part of compatibility because local and
* remote files use different server-bound readers.
*/
export function nativeToolContractFingerprintForTarget(
executionTargetKind: NativeToolExecutionTargetKind,
providerKind?: NativeExecutionInput["provider"]["kind"],
): string {
return `sha256:${createHash("sha256")
.update(
JSON.stringify({
schema: "paperclip.native-tool-contract.v18",
executionTargetKind,
advertisementPolicy: {
// Direct provider threads retain declarations from thread/start.
// Keep this explicit so changing a tool from conditional to stable
// advertisement rotates checkpoints even when its schema is unchanged.
readCurrentWakeComments: "always_advertised_binding_gated.v2",
historicalChatAttachments:
"always_advertised_conversation_binding_gated.v1",
registerDeliverable: "verified_local_or_remote_workspace.v2",
readChatAttachment: "always_advertised_run_scope_local_staging.v2",
structuredHumanInput:
"always_advertised_run_issue_agent_binding_gated_current_task_description.v2",
semanticCompletion: "finish_accessible_deliverable_evidence.v4",
...(providerKind === "opencode"
? { opencodeAssignedMcpTools: "server_gateway_prp_on_demand_exact_names.v2" }
: {}),
connectorTools: "assigned_resources_and_pinned_skill_bundle.v1",
chatTaskTools: "durable_task_binding_and_accepted_origin.v1",
githubResponses: "bound_working_comment_progress_and_final.v1",
githubInstructionSkills: "trusted_configuration_native_selection.v1",
...(executionTargetKind === "remote"
? { assignedMcpTools: "codex_server_gateway_prp_relay.v1" }
: {}),
},
tools: [
{ name: "register_deliverable", version: 2 },
{
name: "read_current_wake_comments",
semanticContract: "paperclip.server-current-wake-comments.v1",
version: 1,
},
{
name: "list_chat_attachments",
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
version: 1,
},
{
name: "reuse_chat_attachment",
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
version: 1,
},
{
name: "read_chat_attachment",
semanticContract: "paperclip.server-chat-attachment-read.v1",
version: 1,
},
{ name: "request_human_input", version: 1 },
],
}),
)
.digest("hex")}`;
}
/** Default local-target fingerprint retained for callers and test fixtures. */
export const NATIVE_TOOL_CONTRACT_FINGERPRINT =
nativeToolContractFingerprintForTarget("local");
function record(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
export function isNativeSessionId(value: unknown): value is string {
return (
typeof value === "string" &&
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(
value,
)
);
}
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
"succeeded",
"interrupted",
"failed",
"cancelled",
"timed_out",
]);
export type NativeSessionBootstrapState = {
processPid: number | null;
processGroupId: number | null;
processStartedAt: Date | null;
runnerProfileJson: unknown;
};
export function isUnusedNativeSessionBootstrap(
run: NativeSessionBootstrapState,
hasProviderEvents: boolean,
): boolean {
return (
run.processPid === null &&
run.processGroupId === null &&
run.processStartedAt === null &&
record(run.runnerProfileJson).sessionCheckpoint == null &&
!hasProviderEvents
);
}
/** Newest-first rows for one exact task/session; never skip provider progress. */
export function selectNativeSessionResumeRun<
T extends NativeSessionBootstrapState & {
id: string;
companyId: string;
agentId: string;
nativeIssueId: string | null;
nativeSessionId: string | null;
runtimeMode: string | null;
status: string;
createdAt: Date;
},
>(input: {
runs: readonly T[];
companyId: string;
agentId: string;
issueId: string;
normalizedSessionId: string;
currentRunId: string;
beforeCreatedAt: Date;
providerEvidenceRunIds: ReadonlySet<string>;
}): T | null {
for (const run of input.runs) {
if (
run.id === input.currentRunId ||
run.companyId !== input.companyId ||
run.agentId !== input.agentId ||
run.nativeIssueId !== input.issueId ||
run.nativeSessionId !== input.normalizedSessionId
)
continue;
if (
run.runtimeMode !== "native" ||
!LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(run.status)
)
return null;
// Even an incompatible checkpoint is a progress barrier. The caller must
// reject it or start a fresh session, not fall back to an older checkpoint.
if (record(run.runnerProfileJson).sessionCheckpoint != null)
return run.createdAt > input.beforeCreatedAt ? null : run;
if (
!isUnusedNativeSessionBootstrap(
run,
input.providerEvidenceRunIds.has(run.id),
)
)
return null;
}
return null;
}
const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"turn.started",
"provider.event",
"provider.rpc_result",
];
export async function nativeSessionProviderEvidence(
db: Pick<Db, "select">,
runIds: string[],
): Promise<Set<string>> {
if (!runIds.length) return new Set();
const rows = await db
.select({ runId: heartbeatRunEvents.runId })
.from(heartbeatRunEvents)
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
),
)
.groupBy(heartbeatRunEvents.runId);
return new Set(rows.map((row) => row.runId));
}
type NativeSessionResumeSelection = {
companyId: string;
agentId: string;
issueId: string;
normalizedSessionId: string;
currentRunId: string;
beforeCreatedAt: Date;
/** Recovery discovery must retain an ineligible newest progress barrier. */
includeProgressBarrier?: boolean;
};
const NATIVE_RESUME_CANDIDATE_WINDOW = 32;
const INHERITED_CHECKPOINT_SOURCE_FIELDS = ["backendKind", "driverKind", "sessionId", "providerSessionId"] as const;
export async function findNativeSessionResumeRun(db: Db, input: NativeSessionResumeSelection) {
if (!isNativeSessionId(input.normalizedSessionId)) return null;
const barrier = await selectNativeSessionResumeBarrier(db, input, []);
// A follow-up that failed before any provider or controller work holds only
// the checkpoint it inherited. Select its source instead, but only when
// every skipped run proves that and inherited exactly that source session.
const skipped: NonNullable<typeof barrier>[] = [];
let selected = barrier;
while (selected && skipped.length < NATIVE_RESUME_CANDIDATE_WINDOW &&
await isUnstartedInheritedFollowUp(db, input, selected)) {
skipped.push(selected);
selected = await selectNativeSessionResumeBarrier(db, input, skipped.map(run => run.id));
}
if (!skipped.length) return selected;
const source = record(selected?.runnerProfileJson.sessionCheckpoint);
const inherited = selected !== null && typeof source.sessionId === "string" && skipped.every(run => {
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
return INHERITED_CHECKPOINT_SOURCE_FIELDS.every(field => checkpoint[field] === source[field]);
});
return inherited ? selected : barrier;
}
/** The run row, its coordinator, and the absence of controller checkpoints,
* results and provider events together prove no provider work started. */
async function isUnstartedInheritedFollowUp(
db: Db,
input: NativeSessionResumeSelection,
run: { id: string; runtimeMode: string | null; status: string; processPid: number | null;
processGroupId: number | null; processStartedAt: Date | null; runnerProfileJson: { sessionCheckpoint: unknown; nativeControllerRecovery: unknown } },
): Promise<boolean> {
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
const identity = record(checkpoint.identity);
if (run.runtimeMode !== "native" || run.status !== "failed" || run.processPid !== null ||
run.processGroupId !== null || run.processStartedAt !== null || run.runnerProfileJson.nativeControllerRecovery != null ||
identity.runId !== run.id || identity.sessionId !== input.normalizedSessionId || checkpoint.cursor !== null) return false;
const [evidence] = await db.select({
executionRecovery: sql<unknown>`${heartbeatRuns.resultJson}->'executionRecovery'`,
phase: nativeRunFinalizations.phase,
resultId: nativeRunFinalizations.resultId,
leaseOwner: nativeRunFinalizations.leaseOwner,
controllerPid: nativeRunFinalizations.controllerPid,
controllerCheckpoint: sql<boolean>`exists (select 1 from ${nativeRunnerCheckpoints}
where ${nativeRunnerCheckpoints.companyId} = ${heartbeatRuns.companyId} and ${nativeRunnerCheckpoints.runId} = ${heartbeatRuns.id})`,
result: sql<boolean>`exists (select 1 from ${nativeRunResults}
where ${nativeRunResults.companyId} = ${heartbeatRuns.companyId} and ${nativeRunResults.runId} = ${heartbeatRuns.id})`,
}).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
eq(nativeRunFinalizations.issueId, input.issueId),
)).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId)));
const recovery = record(evidence?.executionRecovery);
return Boolean(evidence && recovery.kind === "bootstrap" && recovery.providerWorkStarted === false &&
evidence.phase === "terminal_failure" && evidence.resultId === null && evidence.leaseOwner === null &&
evidence.controllerPid === null && !evidence.controllerCheckpoint && !evidence.result &&
!(await nativeSessionProviderEvidence(db, [run.id])).has(run.id));
}
async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResumeSelection, skippedRunIds: string[]) {
const candidates = db.$with("native_resume_candidates").as(db
.select({
id: heartbeatRuns.id,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
createdAt: heartbeatRuns.createdAt,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
processStartedAt: heartbeatRuns.processStartedAt,
hasCheckpoint: sql<boolean>`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' <> 'null'::jsonb, false)`.as("has_checkpoint"),
})
.from(heartbeatRuns)
.where(
and(
ne(heartbeatRuns.id, input.currentRunId),
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId),
),
)
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id))
.limit(NATIVE_RESUME_CANDIDATE_WINDOW));
const terminalStatuses = [...LEGACY_RETRY_SOURCE_TERMINAL_STATUSES];
const barrier = db.$with("native_resume_barrier").as(db
.select()
.from(candidates)
.where(and(skippedRunIds.length ? notInArray(candidates.id, skippedRunIds) : undefined, or(
ne(candidates.runtimeMode, "native"),
not(inArray(candidates.status, terminalStatuses)),
candidates.hasCheckpoint,
isNotNull(candidates.processPid),
isNotNull(candidates.processGroupId),
isNotNull(candidates.processStartedAt),
exists(db.select({ runId: heartbeatRunEvents.runId })
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.runId, candidates.id),
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
))),
)))
.orderBy(desc(candidates.createdAt), desc(candidates.id))
.limit(1));
const [run] = await db.with(candidates, barrier)
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
runnerInstanceId: heartbeatRuns.runnerInstanceId,
nativeSessionId: heartbeatRuns.nativeSessionId,
nativeIssueId: heartbeatRuns.nativeIssueId,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
createdAt: heartbeatRuns.createdAt,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
processStartedAt: heartbeatRuns.processStartedAt,
runnerProfileJson: sql<{
nativeToolContractFingerprint: unknown;
nativeExecutionInput: unknown;
sessionCheckpoint: unknown;
nativeControllerRecovery: unknown;
}>`jsonb_build_object(
'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint',
'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput',
'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint',
'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'
)`,
contextSnapshot: sql<{
paperclipWake: unknown;
paperclipIssue: unknown;
}>`jsonb_build_object(
'paperclipWake', ${heartbeatRuns.contextSnapshot}->'paperclipWake',
'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object'
then (select coalesce(jsonb_object_agg(brief.key, brief.value), '{}'::jsonb)
from jsonb_each(${heartbeatRuns.contextSnapshot}->'paperclipIssue') as brief
where brief.key in ('id', 'title', 'description')) else ${heartbeatRuns.contextSnapshot}->'paperclipIssue' end
)`,
})
.from(heartbeatRuns)
.innerJoin(barrier, eq(heartbeatRuns.id, barrier.id))
.where(and(
...(input.includeProgressBarrier ? [] : [
eq(barrier.runtimeMode, "native"),
inArray(barrier.status, terminalStatuses),
eq(barrier.hasCheckpoint, true),
sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`,
]),
sql`${heartbeatRuns.companyId}::text = ${input.companyId}`,
sql`${heartbeatRuns.agentId}::text = ${input.agentId}`,
sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`,
sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`,
));
return run ?? null;
}
/** A preassigned session id may rotate only before immutable native admission. */
export function nativeSessionIdForBootstrapPersistence(input: {
run: NativeSessionBootstrapState & { nativeSessionId: string | null };
selectedSessionId: string;
hasProviderEvents: boolean;
}): string {
if (input.run.nativeSessionId === input.selectedSessionId)
return input.selectedSessionId;
if (
record(input.run.runnerProfileJson).nativeExecutionInput !== undefined ||
!isUnusedNativeSessionBootstrap(input.run, input.hasProviderEvents)
) {
throw new Error("native_session_bootstrap_identity_conflict");
}
return input.selectedSessionId;
}
/** Caller holds the run row lock; recheck authority after waiting for that lock. */
export async function prepareNativeSessionBootstrapPersistence(
db: Pick<Db, "select">,
input: {
run: NativeSessionBootstrapState & {
id: string;
nativeSessionId: string | null;
};
selectedSessionId: string;
execution: NativeExecutionInput;
restoringCheckpoint: boolean;
},
) {
const hasProviderEvents = (
await nativeSessionProviderEvidence(db, [input.run.id])
).has(input.run.id);
const profile = record(input.run.runnerProfileJson);
if (
profile.nativeExecutionInput !== undefined &&
!isDeepStrictEqual(
parseNativeExecutionInput(profile.nativeExecutionInput),
input.execution,
)
) {
throw new Error("native_execution_input_persisted_binding_mismatch");
}
if (
input.restoringCheckpoint &&
!isUnusedNativeSessionBootstrap(input.run, hasProviderEvents)
) {
throw new Error("native_session_bootstrap_identity_conflict");
}
return nativeSessionIdForBootstrapPersistence({
...input,
hasProviderEvents,
});
}
/**
* Legacy compatibility is deliberately narrower than ordinary task-session
* continuation: only a replacement row that never acquired any native process
* or provider authority may be rebound to a terminal native source. The exact
* checkpoint/session/workspace/provider binding is validated separately by
* rebindNativeSessionCheckpoint.
*/
export function isUnusedLegacyNativeRetryReplacement(input: {
replacement: {
processPid: number | null;
processGroupId: number | null;
processStartedAt: Date | null;
runnerProfileJson: unknown;
};
source: {
runtimeMode: string | null;
status: string;
nativeSessionId: string | null;
} | null;
hasProviderEvents: boolean;
}): boolean {
return Boolean(
input.source?.runtimeMode === "native" &&
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
isNativeSessionId(input.source.nativeSessionId) &&
isUnusedNativeSessionBootstrap(input.replacement, input.hasProviderEvents),
);
}
function sameProvider(
previous: NativeExecutionInput["provider"],
current: NativeExecutionInput["provider"],
): boolean {
return JSON.stringify(previous) === JSON.stringify(current);
}
function nativeCheckpointWorkspaceScope(
execution: NativeExecutionInput,
owningRunId: string,
) {
// Projectless work uses each heartbeat run id as a placeholder workspace id.
// Continuity is safe only when both sides independently prove that shape and
// their complete immutable workspace descriptors match. A real managed
// workspace instead binds continuity to its durable workspace row id.
return execution.binding.executionWorkspaceId === owningRunId
? {
kind: "transient" as const,
cwd: execution.workspace.cwd,
repoUrl: execution.workspace.repoUrl,
repoRef: execution.workspace.repoRef,
branchName: execution.workspace.branchName,
}
: {
kind: "managed" as const,
id: execution.binding.executionWorkspaceId,
};
}
function sameWorkspaceScope(input: {
previousExecution: NativeExecutionInput;
previousRunId: string;
currentExecution: NativeExecutionInput;
}): boolean {
return (
JSON.stringify(
nativeCheckpointWorkspaceScope(
input.previousExecution,
input.previousRunId,
),
) ===
JSON.stringify(
nativeCheckpointWorkspaceScope(
input.currentExecution,
input.currentExecution.binding.runId,
),
)
);
}
/** A resume delta is valid only if the provider checkpoint really can be used. */
export function buildNativeExecutionWithCheckpoint(input: {
previousRun:
Parameters<typeof rebindNativeSessionCheckpoint>[0]["previousRun"] | null;
normalizedSessionId: string;
executionTargetKind?: NativeToolExecutionTargetKind;
toolRefreshOnResume?: boolean;
refreshTools?: boolean;
buildExecution: (options: {
normalizedSessionId: string;
resumedSession: boolean;
}) => NativeExecutionInput;
}): {
execution: NativeExecutionInput;
checkpoint: PersistedNativeSession | null;
normalizedSessionId: string;
} {
const execution = input.buildExecution({
normalizedSessionId: input.normalizedSessionId,
resumedSession: input.previousRun !== null,
});
// A presentation-only upgrade must not rotate a healthy provider session or
// lose a durable goal. Keep its v4/v5 format until that session naturally ends.
// All ordinary identity, workspace, provider, tool and recovery checks still apply.
if (input.previousRun && (execution.schema === "paperclip.native-execution-input.v4" || execution.schema === "paperclip.native-execution-input.v5")) {
const previousSchema = record(record(input.previousRun.runnerProfileJson).nativeExecutionInput).schema;
if ((previousSchema === "paperclip.native-execution-input.v4" || previousSchema === "paperclip.native-execution-input.v5") && previousSchema !== execution.schema) {
const { completionSources: _sources, ...common } = execution as typeof execution & { completionSources?: unknown };
const retainedFormat = parseNativeExecutionInput({ ...common, schema: previousSchema });
const retainedCheckpoint = rebindNativeSessionCheckpoint({
previousRun: input.previousRun,
currentExecution: retainedFormat,
executionTargetKind: input.executionTargetKind,
toolRefreshOnResume: input.toolRefreshOnResume,
refreshTools: input.refreshTools,
});
if (retainedCheckpoint) return { execution: retainedFormat, checkpoint: retainedCheckpoint, normalizedSessionId: input.normalizedSessionId };
}
}
const checkpoint = input.previousRun
? rebindNativeSessionCheckpoint({
previousRun: input.previousRun,
currentExecution: execution,
executionTargetKind: input.executionTargetKind,
toolRefreshOnResume: input.toolRefreshOnResume,
refreshTools: input.refreshTools,
})
: null;
if (checkpoint)
return {
execution,
checkpoint,
normalizedSessionId: input.normalizedSessionId,
};
// Rebuild both task context and wake instructions. Merely rotating the ID
// leaves a fresh provider with a compact delta and missing task context.
const normalizedSessionId = randomUUID();
return {
execution: input.buildExecution({
normalizedSessionId,
resumedSession: false,
}),
checkpoint: null,
normalizedSessionId,
};
}
/**
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
* The provider/driver session identity is retained, while every per-turn and
* per-event field is reset so the new run starts one clean turn via resume.
*/
export function rebindNativeSessionCheckpoint(input: {
previousRun: {
id: string;
companyId: string;
agentId: string;
nativeSessionId: string | null;
runnerProfileJson: unknown;
};
currentExecution: NativeExecutionInput;
executionTargetKind?: NativeToolExecutionTargetKind;
toolRefreshOnResume?: boolean;
refreshTools?: boolean;
}): PersistedNativeSession | null {
if (input.refreshTools && input.toolRefreshOnResume !== true) return null;
const previousProfile = record(input.previousRun.runnerProfileJson);
if (
previousProfile.nativeToolContractFingerprint !==
nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local", input.currentExecution.provider.kind)
) {
return null;
}
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
const checkpointIdentity = record(rawCheckpoint.identity);
const current = input.currentExecution;
const normalizedSessionId = current.session.normalizedSessionId;
if (
!isNativeSessionId(normalizedSessionId) ||
input.previousRun.companyId !== current.binding.companyId ||
input.previousRun.agentId !== current.binding.agentId ||
input.previousRun.nativeSessionId !== normalizedSessionId ||
typeof rawCheckpoint.sessionId !== "string" ||
checkpointIdentity.runId !== input.previousRun.id ||
checkpointIdentity.companyId !== current.binding.companyId ||
checkpointIdentity.issueId !== current.binding.issueId ||
checkpointIdentity.agentId !== current.binding.agentId ||
checkpointIdentity.sessionId !== normalizedSessionId
)
return null;
let previousExecution: NativeExecutionInput;
try {
previousExecution = parseNativeExecutionInput(
previousProfile.nativeExecutionInput,
);
} catch {
return null;
}
if (
previousExecution.binding.runId !== input.previousRun.id ||
previousExecution.binding.companyId !== current.binding.companyId ||
previousExecution.binding.issueId !== current.binding.issueId ||
previousExecution.binding.agentId !== current.binding.agentId ||
previousExecution.session.normalizedSessionId !== normalizedSessionId ||
previousExecution.session.driverKind !== current.session.driverKind ||
!sameWorkspaceScope({
previousExecution,
previousRunId: input.previousRun.id,
currentExecution: current,
}) ||
previousExecution.task.workMode !== current.task.workMode ||
("executionMode" in previousExecution
? previousExecution.executionMode
: "default") !==
("executionMode" in current ? current.executionMode : "default") ||
!sameProvider(previousExecution.provider, current.provider) ||
previousExecution.schema !== current.schema ||
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
("runtimeContext" in previousExecution &&
"runtimeContext" in current &&
previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest &&
!(input.toolRefreshOnResume === true &&
canonicalNativeRuntimeContextDigest({
...previousExecution.runtimeContext,
mcp: current.runtimeContext.mcp,
}) === current.runtimeContext.aggregateDigest))
)
return null;
const priorSemanticResult = record(rawCheckpoint.semanticResult);
const priorContinuation = record(priorSemanticResult.continuation);
const rawGoal = rawCheckpoint.goal;
const hasUnfinishedGoal =
rawGoal !== null &&
rawGoal !== undefined &&
record(rawGoal).status !== "complete";
const providerRecoveryPolicy = hasUnfinishedGoal
? ("same_session_only" as const)
: !input.refreshTools && priorSemanticResult.reportedWorkDisposition === "yielded" &&
priorContinuation.kind === "response_wake"
? ("allow_replacement_after_governed_wait" as const)
: ("allow_replacement_after_resume_failure" as const);
return {
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
identity: {
runId: current.binding.runId,
sessionId: normalizedSessionId,
companyId: current.binding.companyId,
issueId: current.binding.issueId,
agentId: current.binding.agentId,
},
cursor: null,
semanticResult: null,
terminal: null,
activeTurnId: null,
terminalTurns: [],
pendingRuntimeRequests: [],
// Settlement evidence belongs to the prior heartbeat run, even when the
// provider session is reused for the user's response.
governedWait: undefined,
providerRecoveryPolicy,
};
}