Files
PaperClipAI/server/src/services/native-runtime/native-session-resume.ts
T
DottaandClaude Opus 5.5 c910651115 fix(native): start each rebound run with a fresh restart-continuation marker (#20053)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the control plane restarts during an agent turn, the runner
restores the same conversation. A Codex, Claude (ACPX) or OpenCode turn
lost in the restart is marked `provider_turn_lost_on_restore`, and the
run continues it once in the same conversation
> - That one continuation is guarded by a disposition-only recovery
marker in the session checkpoint. It is run state: it names a turn of
the run that used it
> - A follow-up run (a user comment on the same task) reuses the
provider session by re-binding the previous run's checkpoint to the new
run. Re-binding clears the run-scoped turn, result and request state,
but it kept the marker
> - The chaos K13 storm (wave w-20261108T134237-s9401) hit this three
times (t24 and t45 on Claude, t77 on Codex). The task's first run had
already used its restart continuation. The follow-up run inherited a
spent marker naming a turn it never had, so when a restart lost its own
turn, the driver kept the session closed. The run failed, and the user
had to click Retry
> - This pull request resets the marker at both re-bind sites, as for
every other run-scoped field
> - The benefit is that a restart during a follow-up turn continues on
its own instead of putting a Retry click on the user

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan, restart continuation). Found by the
restart chaos campaign. In all three failed follow-up runs:

- After `resume_dead_runner`, the run emitted `turn.failed {code:
provider_turn_lost_on_restore, recoverable: true}` for its own active
turn.
- The checkpoint still carried `dispositionOnlyRecoveryConsumed: true`,
with `dispositionOnlyRecoveryTurnId` set to the previous run's
continuation turn.

## What Changed

- `native-session-resume.ts` (`rebindNativeSessionCheckpoint`) and
`native-session-executor.ts` (warm checkpoint rebind): a rebound run
starts with `dispositionOnlyRecoveryConsumed: false` and
`dispositionOnlyRecoveryTurnId: null`.
- Nothing changes inside a run: the driver's reopen rule
(`mapTerminalTurn`) and `nativeRestartInterruptedTurnId` are untouched.
The first commit changed the helper; review found the driver keeps its
own copy of the rule, and the second commit reverts the helper in favour
of this fix.

## Verification

- `native-session-resume.test.ts` "retains provider identity but clears
prior turn and event state" now seeds a spent marker on the previous
run. It fails on `master` and passes here.
- `native-session-executor.test.ts` "starts a new run's usage from the
prior run's total when a warm checkpoint resumes" (3 cases) seeds the
first run's checkpoint with a spent marker and asserts that the second
run starts with it reset. It fails on `master` and passes here.
- The other `native-session-resume.test.ts` tests pass. One test,
"archives a damaged prior epoch before completing guarded same-task
replacement with real runnerd", fails locally identically with and
without this change (ENOENT under `authority-epochs`, from the
real-runnerd environment). It's left to CI.
- Next: a targeted chaos rerun (Codex and Claude C1 under repeated K2)
on this branch. I'll add the result here.

## Risks

- **Previous-release story (upgrade-safe run state):** no persisted
shape changes. A run re-bound by this build gets a fresh marker. A run
re-bound before the deploy, still in flight with an inherited spent
marker, behaves as before: a lost turn falls back to Retry, once.
- Inside a run, nothing changes: the marker still bounds same-run
continuations, together with the provider-attempt restart budget.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 08:49:38 -06:00

728 lines
29 KiB
TypeScript

import { createHash, randomUUID } from "node:crypto";
import { isDeepStrictEqual } from "node:util";
import { and, desc, eq, exists, inArray, isNotNull, ne, not, notInArray, or, sql } from "drizzle-orm";
import {
heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunnerCheckpoints, nativeRunResults, type Db,
} from "@paperclipai/db";
import type {
NativeExecutionInput,
PersistedNativeSession,
} from "../../vendor/paperclip-runner/index.js";
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput } from "../../vendor/paperclip-runner/index.js";
export type NativeToolExecutionTargetKind = "local" | "remote";
/** Fence live ACPX owners when shared runtime semantics change, even if the
* underlying native executable/profile did not change. Undefined preserves
* the existing configuration digest for other provider transports.
*/
export function nativeRuntimeContractForProvider(
provider: NativeExecutionInput["provider"],
): string | undefined {
return provider.kind === "acpx" ? "paperclip.acpx-runtime-contract.v1" : undefined;
}
/**
* Persisted provider threads retain their dynamic-tool declarations. This
* fingerprint is part of checkpoint compatibility and must change whenever
* the server-authorized native tool definitions or advertisement policy
* changes. The execution target remains part of compatibility because local and
* remote files use different server-bound readers.
*/
export function nativeToolContractFingerprintForTarget(
executionTargetKind: NativeToolExecutionTargetKind,
providerKind?: NativeExecutionInput["provider"]["kind"],
): string {
return `sha256:${createHash("sha256")
.update(
JSON.stringify({
schema: "paperclip.native-tool-contract.v19",
executionTargetKind,
advertisementPolicy: {
// Direct provider threads retain declarations from thread/start.
// Keep this explicit so changing a tool from conditional to stable
// advertisement rotates checkpoints even when its schema is unchanged.
readCurrentWakeComments: "always_advertised_binding_gated.v2",
historicalChatAttachments:
"always_advertised_conversation_binding_gated.v1",
registerDeliverable: "verified_local_or_remote_workspace.v2",
// Advertised only to Codex runs admitted on an OpenAI API-key
// connection; the key stays in the server.
generateImage: "codex_openai_api_key_connection_server_key.v1",
readChatAttachment: "always_advertised_run_scope_local_staging.v2",
structuredHumanInput:
"always_advertised_run_issue_agent_binding_gated_current_task_description.v2",
semanticCompletion: "finish_accessible_deliverable_evidence.v4",
...(providerKind === "opencode"
? { opencodeAssignedMcpTools: "server_gateway_prp_on_demand_exact_names.v2" }
: {}),
connectorTools: "assigned_resources_and_pinned_skill_bundle.v1",
chatTaskTools: "durable_task_binding_and_accepted_origin.v1",
githubResponses: "bound_working_comment_progress_and_final.v1",
githubInstructionSkills: "trusted_configuration_native_selection.v1",
...(executionTargetKind === "remote"
? { assignedMcpTools: "codex_server_gateway_prp_relay.v1" }
: {}),
},
tools: [
{ name: "register_deliverable", version: 2 },
{ name: "generate_image", semanticContract: "paperclip.server-openai-image-generation.v1", version: 1 },
{
name: "read_current_wake_comments",
semanticContract: "paperclip.server-current-wake-comments.v1",
version: 1,
},
{
name: "list_chat_attachments",
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
version: 1,
},
{
name: "reuse_chat_attachment",
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
version: 1,
},
{
name: "read_chat_attachment",
semanticContract: "paperclip.server-chat-attachment-read.v1",
version: 1,
},
{ name: "request_human_input", version: 1 },
],
}),
)
.digest("hex")}`;
}
/** Default local-target fingerprint retained for callers and test fixtures. */
export const NATIVE_TOOL_CONTRACT_FINGERPRINT =
nativeToolContractFingerprintForTarget("local");
/** The runner proved that this run's saved provider session cannot be
* restored by the current runner build. Its checkpoint is not resumable. */
export const NATIVE_PROVIDER_STATE_INCOMPATIBLE = "native_provider_state_incompatible";
/** The saved session owned a side-effecting call without a delivered result.
* Recovery requires an explicit reconciliation of that action. */
export const NATIVE_PROVIDER_SIDE_EFFECT_PENDING = "native_provider_side_effect_pending";
function record(value: unknown): Record<string, unknown> {
return typeof value === "object" && value !== null && !Array.isArray(value)
? (value as Record<string, unknown>)
: {};
}
export function isNativeSessionId(value: unknown): value is string {
return (
typeof value === "string" &&
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(
value,
)
);
}
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
"succeeded",
"interrupted",
"failed",
"cancelled",
"timed_out",
]);
export type NativeSessionBootstrapState = {
processPid: number | null;
processGroupId: number | null;
processStartedAt: Date | null;
runnerProfileJson: unknown;
};
export function isUnusedNativeSessionBootstrap(
run: NativeSessionBootstrapState,
hasProviderEvents: boolean,
): boolean {
return (
run.processPid === null &&
run.processGroupId === null &&
run.processStartedAt === null &&
record(run.runnerProfileJson).sessionCheckpoint == null &&
!hasProviderEvents
);
}
/** Newest-first rows for one exact task/session; never skip provider progress. */
export function selectNativeSessionResumeRun<
T extends NativeSessionBootstrapState & {
id: string;
companyId: string;
agentId: string;
nativeIssueId: string | null;
nativeSessionId: string | null;
runtimeMode: string | null;
status: string;
createdAt: Date;
},
>(input: {
runs: readonly T[];
companyId: string;
agentId: string;
issueId: string;
normalizedSessionId: string;
currentRunId: string;
beforeCreatedAt: Date;
providerEvidenceRunIds: ReadonlySet<string>;
}): T | null {
for (const run of input.runs) {
if (
run.id === input.currentRunId ||
run.companyId !== input.companyId ||
run.agentId !== input.agentId ||
run.nativeIssueId !== input.issueId ||
run.nativeSessionId !== input.normalizedSessionId
)
continue;
if (
run.runtimeMode !== "native" ||
!LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(run.status)
)
return null;
// Even an incompatible checkpoint is a progress barrier. The caller must
// reject it or start a fresh session, not fall back to an older checkpoint.
if (record(run.runnerProfileJson).sessionCheckpoint != null)
return run.createdAt > input.beforeCreatedAt ? null : run;
if (
!isUnusedNativeSessionBootstrap(
run,
input.providerEvidenceRunIds.has(run.id),
)
)
return null;
}
return null;
}
const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
"harness.ready",
"session.started",
"session.resumed",
"session.updated",
"turn.started",
"provider.event",
"provider.rpc_result",
];
export async function nativeSessionProviderEvidence(
db: Pick<Db, "select">,
runIds: string[],
): Promise<Set<string>> {
if (!runIds.length) return new Set();
const rows = await db
.select({ runId: heartbeatRunEvents.runId })
.from(heartbeatRunEvents)
.where(
and(
inArray(heartbeatRunEvents.runId, runIds),
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
),
)
.groupBy(heartbeatRunEvents.runId);
return new Set(rows.map((row) => row.runId));
}
type NativeSessionResumeSelection = {
companyId: string;
agentId: string;
issueId: string;
normalizedSessionId: string;
currentRunId?: string;
beforeCreatedAt: Date;
/** Recovery discovery must retain an ineligible newest progress barrier. */
includeProgressBarrier?: boolean;
};
const NATIVE_RESUME_CANDIDATE_WINDOW = 32;
const INHERITED_CHECKPOINT_SOURCE_FIELDS = ["backendKind", "driverKind", "sessionId", "providerSessionId"] as const;
export async function findNativeSessionResumeRun(db: Db, input: NativeSessionResumeSelection) {
if (!isNativeSessionId(input.normalizedSessionId)) return null;
const barrier = await selectNativeSessionResumeBarrier(db, input, []);
// A follow-up that failed before any provider or controller work holds only
// the checkpoint it inherited. Select its source instead, but only when
// every skipped run proves that and inherited exactly that source session.
const skipped: NonNullable<typeof barrier>[] = [];
let selected = barrier;
while (selected && skipped.length < NATIVE_RESUME_CANDIDATE_WINDOW &&
await isUnstartedInheritedFollowUp(db, input, selected)) {
skipped.push(selected);
selected = await selectNativeSessionResumeBarrier(db, input, skipped.map(run => run.id));
}
if (!skipped.length) return selected;
const source = record(selected?.runnerProfileJson.sessionCheckpoint);
const inherited = selected !== null && typeof source.sessionId === "string" && skipped.every(run => {
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
return INHERITED_CHECKPOINT_SOURCE_FIELDS.every(field => checkpoint[field] === source[field]);
});
return inherited ? selected : barrier;
}
/** The run row, its coordinator, and the absence of controller checkpoints,
* results and provider events together prove no provider work started. */
async function isUnstartedInheritedFollowUp(
db: Db,
input: NativeSessionResumeSelection,
run: { id: string; runtimeMode: string | null; status: string; processPid: number | null;
processGroupId: number | null; processStartedAt: Date | null; runnerProfileJson: { sessionCheckpoint: unknown; nativeControllerRecovery: unknown } },
): Promise<boolean> {
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
const identity = record(checkpoint.identity);
if (run.runtimeMode !== "native" || run.status !== "failed" || run.processPid !== null ||
run.processGroupId !== null || run.processStartedAt !== null || run.runnerProfileJson.nativeControllerRecovery != null ||
identity.runId !== run.id || identity.sessionId !== input.normalizedSessionId || checkpoint.cursor !== null) return false;
const [evidence] = await db.select({
executionRecovery: sql<unknown>`${heartbeatRuns.resultJson}->'executionRecovery'`,
phase: nativeRunFinalizations.phase,
resultId: nativeRunFinalizations.resultId,
leaseOwner: nativeRunFinalizations.leaseOwner,
controllerPid: nativeRunFinalizations.controllerPid,
controllerCheckpoint: sql<boolean>`exists (select 1 from ${nativeRunnerCheckpoints}
where ${nativeRunnerCheckpoints.companyId} = ${heartbeatRuns.companyId} and ${nativeRunnerCheckpoints.runId} = ${heartbeatRuns.id})`,
result: sql<boolean>`exists (select 1 from ${nativeRunResults}
where ${nativeRunResults.companyId} = ${heartbeatRuns.companyId} and ${nativeRunResults.runId} = ${heartbeatRuns.id})`,
}).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and(
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
eq(nativeRunFinalizations.issueId, input.issueId),
)).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId)));
const recovery = record(evidence?.executionRecovery);
return Boolean(evidence && recovery.kind === "bootstrap" && recovery.providerWorkStarted === false &&
evidence.phase === "terminal_failure" && evidence.resultId === null && evidence.leaseOwner === null &&
evidence.controllerPid === null && !evidence.controllerCheckpoint && !evidence.result &&
!(await nativeSessionProviderEvidence(db, [run.id])).has(run.id));
}
async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResumeSelection, skippedRunIds: string[]) {
const candidates = db.$with("native_resume_candidates").as(db
.select({
id: heartbeatRuns.id,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
createdAt: heartbeatRuns.createdAt,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
processStartedAt: heartbeatRuns.processStartedAt,
hasCheckpoint: sql<boolean>`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' <> 'null'::jsonb, false)`.as("has_checkpoint"),
})
.from(heartbeatRuns)
.where(
and(
input.currentRunId ? ne(heartbeatRuns.id, input.currentRunId) : undefined,
eq(heartbeatRuns.companyId, input.companyId),
eq(heartbeatRuns.agentId, input.agentId),
eq(heartbeatRuns.nativeIssueId, input.issueId),
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId),
),
)
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id))
.limit(NATIVE_RESUME_CANDIDATE_WINDOW));
const terminalStatuses = [...LEGACY_RETRY_SOURCE_TERMINAL_STATUSES];
const barrier = db.$with("native_resume_barrier").as(db
.select()
.from(candidates)
.where(and(skippedRunIds.length ? notInArray(candidates.id, skippedRunIds) : undefined, or(
ne(candidates.runtimeMode, "native"),
not(inArray(candidates.status, terminalStatuses)),
candidates.hasCheckpoint,
isNotNull(candidates.processPid),
isNotNull(candidates.processGroupId),
isNotNull(candidates.processStartedAt),
exists(db.select({ runId: heartbeatRunEvents.runId })
.from(heartbeatRunEvents)
.where(and(
eq(heartbeatRunEvents.runId, candidates.id),
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
))),
)))
.orderBy(desc(candidates.createdAt), desc(candidates.id))
.limit(1));
const [run] = await db.with(candidates, barrier)
.select({
id: heartbeatRuns.id,
companyId: heartbeatRuns.companyId,
agentId: heartbeatRuns.agentId,
runnerInstanceId: heartbeatRuns.runnerInstanceId,
nativeSessionId: heartbeatRuns.nativeSessionId,
nativeIssueId: heartbeatRuns.nativeIssueId,
runtimeMode: heartbeatRuns.runtimeMode,
status: heartbeatRuns.status,
errorCode: heartbeatRuns.errorCode,
createdAt: heartbeatRuns.createdAt,
processPid: heartbeatRuns.processPid,
processGroupId: heartbeatRuns.processGroupId,
processStartedAt: heartbeatRuns.processStartedAt,
runnerProfileJson: sql<{
nativeToolContractFingerprint: unknown;
nativeExecutionInput: unknown;
sessionCheckpoint: unknown;
nativeControllerRecovery: unknown;
}>`jsonb_build_object(
'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint',
'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput',
'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint',
'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery'
)`,
contextSnapshot: sql<{
paperclipWake: unknown;
paperclipIssue: unknown;
}>`jsonb_build_object(
'paperclipWake', ${heartbeatRuns.contextSnapshot}->'paperclipWake',
'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object'
then (select coalesce(jsonb_object_agg(brief.key, brief.value), '{}'::jsonb)
from jsonb_each(${heartbeatRuns.contextSnapshot}->'paperclipIssue') as brief
where brief.key in ('id', 'title', 'description')) else ${heartbeatRuns.contextSnapshot}->'paperclipIssue' end
)`,
})
.from(heartbeatRuns)
.innerJoin(barrier, eq(heartbeatRuns.id, barrier.id))
.where(and(
...(input.includeProgressBarrier ? [] : [
eq(barrier.runtimeMode, "native"),
inArray(barrier.status, terminalStatuses),
eq(barrier.hasCheckpoint, true),
sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`,
]),
sql`${heartbeatRuns.companyId}::text = ${input.companyId}`,
sql`${heartbeatRuns.agentId}::text = ${input.agentId}`,
sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`,
sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`,
));
return run ?? null;
}
/** A preassigned session id may rotate only before immutable native admission. */
export function nativeSessionIdForBootstrapPersistence(input: {
run: NativeSessionBootstrapState & { nativeSessionId: string | null };
selectedSessionId: string;
hasProviderEvents: boolean;
}): string {
if (input.run.nativeSessionId === input.selectedSessionId)
return input.selectedSessionId;
if (
record(input.run.runnerProfileJson).nativeExecutionInput !== undefined ||
!isUnusedNativeSessionBootstrap(input.run, input.hasProviderEvents)
) {
throw new Error("native_session_bootstrap_identity_conflict");
}
return input.selectedSessionId;
}
/** Caller holds the run row lock; recheck authority after waiting for that lock. */
export async function prepareNativeSessionBootstrapPersistence(
db: Pick<Db, "select">,
input: {
run: NativeSessionBootstrapState & {
id: string;
nativeSessionId: string | null;
};
selectedSessionId: string;
execution: NativeExecutionInput;
restoringCheckpoint: boolean;
},
) {
const hasProviderEvents = (
await nativeSessionProviderEvidence(db, [input.run.id])
).has(input.run.id);
const profile = record(input.run.runnerProfileJson);
if (
profile.nativeExecutionInput !== undefined &&
!isDeepStrictEqual(
parseNativeExecutionInput(profile.nativeExecutionInput),
input.execution,
)
) {
throw new Error("native_execution_input_persisted_binding_mismatch");
}
if (
input.restoringCheckpoint &&
!isUnusedNativeSessionBootstrap(input.run, hasProviderEvents)
) {
throw new Error("native_session_bootstrap_identity_conflict");
}
return nativeSessionIdForBootstrapPersistence({
...input,
hasProviderEvents,
});
}
/**
* Legacy compatibility is deliberately narrower than ordinary task-session
* continuation: only a replacement row that never acquired any native process
* or provider authority may be rebound to a terminal native source. The exact
* checkpoint/session/workspace/provider binding is validated separately by
* rebindNativeSessionCheckpoint.
*/
export function isUnusedLegacyNativeRetryReplacement(input: {
replacement: {
processPid: number | null;
processGroupId: number | null;
processStartedAt: Date | null;
runnerProfileJson: unknown;
};
source: {
runtimeMode: string | null;
status: string;
nativeSessionId: string | null;
} | null;
hasProviderEvents: boolean;
}): boolean {
return Boolean(
input.source?.runtimeMode === "native" &&
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
isNativeSessionId(input.source.nativeSessionId) &&
isUnusedNativeSessionBootstrap(input.replacement, input.hasProviderEvents),
);
}
function sameProvider(
previous: NativeExecutionInput["provider"],
current: NativeExecutionInput["provider"],
): boolean {
return JSON.stringify(previous) === JSON.stringify(current);
}
function nativeCheckpointWorkspaceScope(
execution: NativeExecutionInput,
owningRunId: string,
) {
// Projectless work uses each heartbeat run id as a placeholder workspace id.
// Continuity is safe only when both sides independently prove that shape and
// their complete immutable workspace descriptors match. A real managed
// workspace instead binds continuity to its durable workspace row id.
return execution.binding.executionWorkspaceId === owningRunId
? {
kind: "transient" as const,
cwd: execution.workspace.cwd,
repoUrl: execution.workspace.repoUrl,
repoRef: execution.workspace.repoRef,
branchName: execution.workspace.branchName,
}
: {
kind: "managed" as const,
id: execution.binding.executionWorkspaceId,
};
}
function sameWorkspaceScope(input: {
previousExecution: NativeExecutionInput;
previousRunId: string;
currentExecution: NativeExecutionInput;
}): boolean {
return (
JSON.stringify(
nativeCheckpointWorkspaceScope(
input.previousExecution,
input.previousRunId,
),
) ===
JSON.stringify(
nativeCheckpointWorkspaceScope(
input.currentExecution,
input.currentExecution.binding.runId,
),
)
);
}
/** A resume delta is valid only if the provider checkpoint really can be used. */
export function buildNativeExecutionWithCheckpoint(input: {
previousRun:
Parameters<typeof rebindNativeSessionCheckpoint>[0]["previousRun"] | null;
normalizedSessionId: string;
executionTargetKind?: NativeToolExecutionTargetKind;
toolRefreshOnResume?: boolean;
refreshTools?: boolean;
buildExecution: (options: {
normalizedSessionId: string;
resumedSession: boolean;
}) => NativeExecutionInput;
}): {
execution: NativeExecutionInput;
checkpoint: PersistedNativeSession | null;
normalizedSessionId: string;
} {
const execution = input.buildExecution({
normalizedSessionId: input.normalizedSessionId,
resumedSession: input.previousRun !== null,
});
// A presentation-only upgrade must not rotate a healthy provider session or
// lose a durable goal. Keep its v4/v5 format until that session naturally ends.
// All ordinary identity, workspace, provider, tool and recovery checks still apply.
if (input.previousRun && (execution.schema === "paperclip.native-execution-input.v4" || execution.schema === "paperclip.native-execution-input.v5")) {
const previousSchema = record(record(input.previousRun.runnerProfileJson).nativeExecutionInput).schema;
if ((previousSchema === "paperclip.native-execution-input.v4" || previousSchema === "paperclip.native-execution-input.v5") && previousSchema !== execution.schema) {
const { completionSources: _sources, ...common } = execution as typeof execution & { completionSources?: unknown };
const retainedFormat = parseNativeExecutionInput({ ...common, schema: previousSchema });
const retainedCheckpoint = rebindNativeSessionCheckpoint({
previousRun: input.previousRun,
currentExecution: retainedFormat,
executionTargetKind: input.executionTargetKind,
toolRefreshOnResume: input.toolRefreshOnResume,
refreshTools: input.refreshTools,
});
if (retainedCheckpoint) return { execution: retainedFormat, checkpoint: retainedCheckpoint, normalizedSessionId: input.normalizedSessionId };
}
}
const checkpoint = input.previousRun
? rebindNativeSessionCheckpoint({
previousRun: input.previousRun,
currentExecution: execution,
executionTargetKind: input.executionTargetKind,
toolRefreshOnResume: input.toolRefreshOnResume,
refreshTools: input.refreshTools,
})
: null;
if (checkpoint)
return {
execution,
checkpoint,
normalizedSessionId: input.normalizedSessionId,
};
// Rebuild both task context and wake instructions. Merely rotating the ID
// leaves a fresh provider with a compact delta and missing task context.
const normalizedSessionId = randomUUID();
return {
execution: input.buildExecution({
normalizedSessionId,
resumedSession: false,
}),
checkpoint: null,
normalizedSessionId,
};
}
/**
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
* The provider/driver session identity is retained, while every per-turn and
* per-event field is reset so the new run starts one clean turn via resume.
*/
export function rebindNativeSessionCheckpoint(input: {
previousRun: {
id: string;
companyId: string;
agentId: string;
nativeSessionId: string | null;
errorCode?: string | null;
runnerProfileJson: unknown;
};
currentExecution: NativeExecutionInput;
executionTargetKind?: NativeToolExecutionTargetKind;
toolRefreshOnResume?: boolean;
refreshTools?: boolean;
}): PersistedNativeSession | null {
if (input.refreshTools && input.toolRefreshOnResume !== true) return null;
// The source is still the progress barrier. Its provider session cannot be
// restored, so the next run starts a fresh session in the same workspace.
if (input.previousRun.errorCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE) return null;
// The saved session still holds the unsettled call and refuses attachment.
// Generic Retry is blocked until an operator reconciles that action; the
// reconciled continuation then starts a fresh session in the same workspace.
if (input.previousRun.errorCode === NATIVE_PROVIDER_SIDE_EFFECT_PENDING) return null;
const previousProfile = record(input.previousRun.runnerProfileJson);
if (
previousProfile.nativeToolContractFingerprint !==
nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local", input.currentExecution.provider.kind)
) {
return null;
}
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
const checkpointIdentity = record(rawCheckpoint.identity);
const current = input.currentExecution;
const normalizedSessionId = current.session.normalizedSessionId;
if (
!isNativeSessionId(normalizedSessionId) ||
input.previousRun.companyId !== current.binding.companyId ||
input.previousRun.agentId !== current.binding.agentId ||
input.previousRun.nativeSessionId !== normalizedSessionId ||
typeof rawCheckpoint.sessionId !== "string" ||
checkpointIdentity.runId !== input.previousRun.id ||
checkpointIdentity.companyId !== current.binding.companyId ||
checkpointIdentity.issueId !== current.binding.issueId ||
checkpointIdentity.agentId !== current.binding.agentId ||
checkpointIdentity.sessionId !== normalizedSessionId
)
return null;
let previousExecution: NativeExecutionInput;
try {
previousExecution = parseNativeExecutionInput(
previousProfile.nativeExecutionInput,
);
} catch {
return null;
}
if (
previousExecution.binding.runId !== input.previousRun.id ||
previousExecution.binding.companyId !== current.binding.companyId ||
previousExecution.binding.issueId !== current.binding.issueId ||
previousExecution.binding.agentId !== current.binding.agentId ||
previousExecution.session.normalizedSessionId !== normalizedSessionId ||
previousExecution.session.driverKind !== current.session.driverKind ||
!sameWorkspaceScope({
previousExecution,
previousRunId: input.previousRun.id,
currentExecution: current,
}) ||
previousExecution.task.workMode !== current.task.workMode ||
("executionMode" in previousExecution
? previousExecution.executionMode
: "default") !==
("executionMode" in current ? current.executionMode : "default") ||
!sameProvider(previousExecution.provider, current.provider) ||
previousExecution.schema !== current.schema ||
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
("runtimeContext" in previousExecution &&
"runtimeContext" in current &&
previousExecution.runtimeContext.aggregateDigest !== current.runtimeContext.aggregateDigest &&
!(input.toolRefreshOnResume === true &&
canonicalNativeRuntimeContextDigest({
...previousExecution.runtimeContext,
mcp: current.runtimeContext.mcp,
}) === current.runtimeContext.aggregateDigest))
)
return null;
const priorSemanticResult = record(rawCheckpoint.semanticResult);
const priorContinuation = record(priorSemanticResult.continuation);
const rawGoal = rawCheckpoint.goal;
const hasUnfinishedGoal =
rawGoal !== null &&
rawGoal !== undefined &&
record(rawGoal).status !== "complete";
const providerRecoveryPolicy = hasUnfinishedGoal
? ("same_session_only" as const)
: !input.refreshTools && priorSemanticResult.reportedWorkDisposition === "yielded" &&
priorContinuation.kind === "response_wake"
? ("allow_replacement_after_governed_wait" as const)
: ("allow_replacement_after_resume_failure" as const);
return {
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
identity: {
runId: current.binding.runId,
sessionId: normalizedSessionId,
companyId: current.binding.companyId,
issueId: current.binding.issueId,
agentId: current.binding.agentId,
},
cursor: null,
semanticResult: null,
terminal: null,
activeTurnId: null,
terminalTurns: [],
pendingRuntimeRequests: [],
dispositionOnlyRecoveryConsumed: false,
dispositionOnlyRecoveryTurnId: null,
// Settlement evidence belongs to the prior heartbeat run, even when the
// provider session is reused for the user's response.
governedWait: undefined,
providerRecoveryPolicy,
};
}