mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - The server releases the process identity (`process_pid`,
`process_group_id`, `process_started_at`) of a finished native run when
it can prove that the runner process stopped (#17374)
> - A background queue (`settledNativeProcesses`) finds these runs and
checks them. A check that cannot finish ("skipped" or "pending") writes
nothing, so the run stays in the queue and every recovery hint checks it
again
> - A finished run that a replaced container recorded can never pass the
check, because only the host that recorded the process can prove that it
stopped. Each deploy adds more of these runs
> - On one hosted instance this loop ran about 10 checks per second and
about 55 SQL statements per second, and the server main thread was about
95% busy with only a few live runs
> - This pull request moves each "cannot make progress" reason into the
queue's SQL predicate, adds a growing backoff for "pending" runs, and
makes one check cost one locking statement
> - The benefit is that runs which cannot progress cost nothing until
something that can change the answer happens, and a run that becomes
releasable is still released at once
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
The settled native process recovery queue checks the same finished
native runs again on every recovery hint and every 30 seconds, forever.
With 300 such runs and one company-wide hint per second it ran 9 checks
per second and 84 SQL statements per second (9.4 statements per check,
2.2 ms of server CPU per check) on embedded PostgreSQL. A hosted
instance showed the same rates. The runs that never finish are: runs
recorded by a host that is gone (each redeploy starts a new container
with a new hostname), runs whose coordinator phase is not settled, runs
with no checkpoint and no provider evidence, and runs whose local
process is still alive (they defer 30 seconds with no limit). Startup
discovery (`discoverSettledNativeRunProcesses`) also opens one
transaction for every such run on every boot.
**Expected behavior**
A run that cannot make progress is not read again until something that
can change the answer happens. A run that becomes releasable (its phase
settles, provider evidence arrives, its own host checks it, its sandbox
receipt arrives, its process exits) is released promptly.
**Steps to reproduce**
1. Create a failed native run with a recorded local process identity
from another host (or an unsettled coordinator phase, or no checkpoint
and no provider evidence, or a live process).
2. Register the residual recovery work and send company-wide
`notifyLiveRunRecovery` hints.
3. Count the run's `for update` checks: on `master` every hint checks
the run again.
`server/src/__tests__/settled-native-process-rechecks.test.ts` automates
these steps.
**Paperclip version or commit**
`master` at 3c3e6c2d75.
**Deployment mode**
Any. The cost grows with every deploy of a hosted instance.
## What Changed
- `server/src/services/native-local-process-stop.ts`
- New `settledNativeProcessCandidates`: one fixed-text statement that
finds releasable runs. It reads only runs recorded in this server's PID
space (`process_host_hash`), sandbox runs (`remote`, with a non-local
lease), and rows saved before the hash (their latest identity event must
name this host). It excludes runs with an unsettled coordinator phase
and runs with no checkpoint and no provider evidence. Each PID-space
branch reads its own range of the new partial index.
- `releaseSettledNativeRunProcess` now reads the run, phase, evidence,
lease facts and identity match in one `for update` statement (it was 6
statements). It reads the lease rows again only for a sandbox run. The
new `inspectSettledNativeRunProcess` also returns whether a remote stop
acknowledgement can apply.
- Startup discovery uses the same candidate statement, so it opens no
transaction for an excluded run.
- `server/src/services/heartbeat/live-recovery-work.ts` (settled kind
only)
- The load runs the candidate statement and a separate indexed statement
for cancelled `cursor_cloud_api` runs whose Stop is still `requested`.
- The check uses the loaded entry. It no longer runs the load again for
each entry.
- A "pending" run backs off 30 s, 60 s, 120 s, up to 15 min. A
run-scoped hint (identity event, lease cleanup, stop receipt) resets the
backoff.
- A "skipped" run that the predicate cannot exclude (for example a birth
time mismatch on this host) waits for a run-scoped hint or a restart.
- `server/src/services/heartbeat/recovery.ts` (the settled handler
only): returns the outcome. It calls `acknowledgeRemoteStop` only when
the run has a non-local lease or is a cancelled `cursor_cloud_api` run.
For any other run that function reads two rows and writes nothing.
- `packages/db`: migration 0487 builds
`heartbeat_runs_settled_process_identity_idx` and
`heartbeat_runs_pending_api_stop_idx` with `CREATE INDEX CONCURRENTLY IF
NOT EXISTS`. The migration is `online_compatible` and is listed in
`MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`. `doc/DATABASE.md` describes both
indexes.
End state for each reason that a check cannot finish:
| Reason | Before | After | What re-admits the run |
| --- | --- | --- | --- |
| Recorded by another PID space | checked on every hint | not read |
only a server in that PID space reads it (a live older server in a
deploy overlap still releases its own runs) |
| Coordinator phase not settled | checked on every hint | not read | the
phase write; the finalizer sends a company hint in the same transaction
|
| No checkpoint and no provider evidence | checked on every hint | not
read | the next hint after a provider or turn event |
| Live local process, unreleased lease, missing sandbox receipt | every
hint and every 30 s | backoff 30 s to 15 min | a run-scoped hint resets
the backoff |
| Other same-host skip | checked on every hint | parked | a run-scoped
hint (an identity event sends one) or a restart |
## Verification
- `pnpm exec vitest run
src/__tests__/settled-native-process-rechecks.test.ts` (new, 12 tests,
real PostgreSQL, real queue and coordinator, fake clock):
- each exclusion: replaced container, unsettled phase, no evidence: 0
checks over 4 hints (red on `master`: 3 to 5 checks);
- a live older server and a stopped older server (different hash): the
new server never reads the run and writes no receipt; this server's own
run is released at start;
- re-admission: phase settles, late provider evidence, legacy rows with
no hash (this host released, another host not read), sandbox run
released from any server with its receipt;
- live process: at most 4 checks over 8 hints (red: 17), released on the
run-scoped hint after the process exits;
- one check is BEGIN, one `for update` statement and COMMIT (3
statements); the next hints do not check it; a run-scoped hint checks it
once more (red: 5 statements and a check on every hint);
- a restart costs one check for each pending run; startup discovery with
50 excluded runs sends 1 statement (red: 302).
- On `master` 11 of the 12 tests fail.
- Measurement on embedded PostgreSQL, 300 stuck runs, one company hint
per second for 300 s (fake clock), same harness before and after:
- before: 9.0 checks/s, 84.3 statements/s, 9.4 statements per check, 5.9
s CPU (2.2 ms per check);
- after: 0 checks/s, 3.2 statements/s for all seven residual queues
together, 0.16 s CPU.
- Load plan with 12,000 excluded finished runs (10% without a hash) in a
31,000-row `heartbeat_runs`: the candidate statement reads only index
entries for this server's hash, `remote` and `unknown`, and a bitmap
scan of the 1,200 legacy rows: 2.7 ms, 4,416 buffers. Without the index
the same exclusion read all 31,000 rows (31,127 buffers).
- Existing suites, one file at a time: all pass.
`live-recovery-work.test.ts` 51/51 (an earlier run under machine load
hit the `afterAll` hook timeout after all tests passed; the rerun passed
clean, and `native-safe-replacement.test.ts` hits the same hook timeout
on the base commit), `native-settled-process-release.test.ts` 31/31,
`native-local-process-stop.test.ts` 5/5, `heartbeat/recovery.test.ts`
107/107, `native-runner-restart-recovery.integration.test.ts` 231 passed
/ 14 skipped, `native-restart-recovery.test.ts` 49/49,
`native-controller-recovery.integration.test.ts` 30/30,
`retained-native-session-cleanup-work.test.ts` 29/29,
`native-provider-failure.integration.test.ts` 16/16,
`native-safe-replacement.test.ts` 70/70 (with the `afterAll` hook
timeout that also occurs on the base commit),
`native-remote-retry-admission.test.ts` 93/93.
- `pnpm check:module-boundaries`, `pnpm check:migrations` (packages/db),
server typecheck: pass.
- Not run: the full suite and browser suites.
## Risks
- Unresolved by design: a finished run whose server was replaced keeps
its process fields indefinitely and is never released, because only the
host that recorded a process can prove that it stopped, and the shared
prover refuses a controller-lease proof for a run with a process group.
This pull request only stops paying for it. These runs keep no stop
receipt (`native.local_process_stopped`), so what waits for that receipt
still waits: accounting close of the run's usage source, retained native
session cleanup, and instruction or directory working copies that wait
for proof that the run stopped. A follow-up must decide the ownership
rule for a replaced container.
- Rows saved before `process_host_hash` exist are read through the
legacy branch, which checks the latest identity event for each of them
on each load. On an instance with many such rows the load costs one
index lookup per row. The count can be measured with `select count(*)
from heartbeat_runs where process_host_hash is null and process_pid is
not null and status in
('failed','cancelled','timed_out','interrupted')`.
- A run that this server recorded but whose `process_host_hash` changed
(a new PID namespace on the same host, for example a container restart)
is no longer read here. Before, it was read and skipped, so no release
is lost.
- The backoff and parked state live in memory. A restart checks each
pending run once.
- The migration builds two partial indexes concurrently on
`heartbeat_runs`. It holds no long lock.
## Model Used
Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
and code execution.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
748 lines
30 KiB
TypeScript
748 lines
30 KiB
TypeScript
import { createHash, randomUUID } from "node:crypto";
|
|
import { isDeepStrictEqual } from "node:util";
|
|
import { and, desc, eq, exists, inArray, isNotNull, ne, not, notInArray, or, sql } from "drizzle-orm";
|
|
import {
|
|
heartbeatRunEvents, heartbeatRuns, nativeRunFinalizations, nativeRunnerCheckpoints, nativeRunResults, type Db,
|
|
} from "@paperclipai/db";
|
|
import type {
|
|
NativeExecutionInput,
|
|
PersistedNativeSession,
|
|
} from "../../vendor/paperclip-runner/index.js";
|
|
import { canonicalNativeRuntimeContextDigest, parseNativeExecutionInput, type NativeRuntimeContextSnapshot } from "../../vendor/paperclip-runner/index.js";
|
|
|
|
export type NativeToolExecutionTargetKind = "local" | "remote";
|
|
|
|
/** Fence live ACPX owners when shared runtime semantics change, even if the
|
|
* underlying native executable/profile did not change. Undefined preserves
|
|
* the existing configuration digest for other provider transports.
|
|
*/
|
|
export function nativeRuntimeContractForProvider(
|
|
provider: NativeExecutionInput["provider"],
|
|
): string | undefined {
|
|
return provider.kind === "acpx" ? "paperclip.acpx-runtime-contract.v1" : undefined;
|
|
}
|
|
|
|
/**
|
|
* Persisted provider threads retain their dynamic-tool declarations. This
|
|
* fingerprint is part of checkpoint compatibility and must change whenever
|
|
* the server-authorized native tool definitions or advertisement policy
|
|
* changes. The execution target remains part of compatibility because local and
|
|
* remote files use different server-bound readers.
|
|
*/
|
|
export function nativeToolContractFingerprintForTarget(
|
|
executionTargetKind: NativeToolExecutionTargetKind,
|
|
providerKind?: NativeExecutionInput["provider"]["kind"],
|
|
): string {
|
|
return `sha256:${createHash("sha256")
|
|
.update(
|
|
JSON.stringify({
|
|
schema: "paperclip.native-tool-contract.v19",
|
|
executionTargetKind,
|
|
advertisementPolicy: {
|
|
// Direct provider threads retain declarations from thread/start.
|
|
// Keep this explicit so changing a tool from conditional to stable
|
|
// advertisement rotates checkpoints even when its schema is unchanged.
|
|
readCurrentWakeComments: "always_advertised_binding_gated.v2",
|
|
historicalChatAttachments:
|
|
"always_advertised_conversation_binding_gated.v1",
|
|
registerDeliverable: "verified_local_or_remote_workspace.v2",
|
|
// Advertised only to Codex runs admitted on an OpenAI API-key
|
|
// connection; the key stays in the server.
|
|
generateImage: "codex_openai_api_key_connection_server_key.v1",
|
|
readChatAttachment: "always_advertised_run_scope_local_staging.v2",
|
|
structuredHumanInput:
|
|
"always_advertised_run_issue_agent_binding_gated_current_task_description.v2",
|
|
semanticCompletion: "finish_accessible_deliverable_evidence.v4",
|
|
...(providerKind === "opencode"
|
|
? { opencodeAssignedMcpTools: "server_gateway_prp_on_demand_exact_names.v2" }
|
|
: {}),
|
|
connectorTools: "assigned_resources_and_pinned_skill_bundle.v1",
|
|
chatTaskTools: "durable_task_binding_and_accepted_origin.v1",
|
|
githubResponses: "bound_working_comment_progress_and_final.v1",
|
|
githubInstructionSkills: "trusted_configuration_native_selection.v1",
|
|
assignedMcpTools: "codex_server_gateway_prp_relay.v1",
|
|
},
|
|
tools: [
|
|
{ name: "register_deliverable", version: 2 },
|
|
{ name: "generate_image", semanticContract: "paperclip.server-openai-image-generation.v1", version: 1 },
|
|
{
|
|
name: "read_current_wake_comments",
|
|
semanticContract: "paperclip.server-current-wake-comments.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "list_chat_attachments",
|
|
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "reuse_chat_attachment",
|
|
semanticContract: "paperclip.server-chat-attachment-reuse.v1",
|
|
version: 1,
|
|
},
|
|
{
|
|
name: "read_chat_attachment",
|
|
semanticContract: "paperclip.server-chat-attachment-read.v1",
|
|
version: 1,
|
|
},
|
|
{ name: "request_human_input", version: 1 },
|
|
],
|
|
}),
|
|
)
|
|
.digest("hex")}`;
|
|
}
|
|
|
|
/** Default local-target fingerprint retained for callers and test fixtures. */
|
|
export const NATIVE_TOOL_CONTRACT_FINGERPRINT =
|
|
nativeToolContractFingerprintForTarget("local");
|
|
|
|
/** The runner proved that this run's saved provider session cannot be
|
|
* restored by the current runner build. Its checkpoint is not resumable. */
|
|
export const NATIVE_PROVIDER_STATE_INCOMPATIBLE = "native_provider_state_incompatible";
|
|
/**
|
|
* `runnerProfileJson` key on a run whose result was accepted but whose
|
|
* provider state backup was not saved. Its per-turn sandbox is released after
|
|
* the run, so that backup was the only copy of the provider session. The next
|
|
* run on the session starts a fresh provider session instead of resuming.
|
|
* Records without the key, including every record written before it existed,
|
|
* keep the ordinary resume path.
|
|
*/
|
|
export const NATIVE_PROVIDER_STATE_UNSAVED_KEY = "nativeProviderStateUnsaved";
|
|
|
|
export function isNativeProviderStateUnsaved(profile: unknown): boolean {
|
|
return record(record(profile)[NATIVE_PROVIDER_STATE_UNSAVED_KEY]).code === NATIVE_PROVIDER_STATE_INCOMPATIBLE;
|
|
}
|
|
/** The saved session owned a side-effecting call without a delivered result.
|
|
* Recovery requires an explicit reconciliation of that action. */
|
|
export const NATIVE_PROVIDER_SIDE_EFFECT_PENDING = "native_provider_side_effect_pending";
|
|
|
|
function record(value: unknown): Record<string, unknown> {
|
|
return typeof value === "object" && value !== null && !Array.isArray(value)
|
|
? (value as Record<string, unknown>)
|
|
: {};
|
|
}
|
|
|
|
export function isNativeSessionId(value: unknown): value is string {
|
|
return (
|
|
typeof value === "string" &&
|
|
/^[0-9a-f]{8}-[0-9a-f]{4}-[1-5][0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$/i.test(
|
|
value,
|
|
)
|
|
);
|
|
}
|
|
|
|
const LEGACY_RETRY_SOURCE_TERMINAL_STATUSES = new Set([
|
|
"succeeded",
|
|
"interrupted",
|
|
"failed",
|
|
"cancelled",
|
|
"timed_out",
|
|
]);
|
|
|
|
export type NativeSessionBootstrapState = {
|
|
processPid: number | null;
|
|
processGroupId: number | null;
|
|
processStartedAt: Date | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
|
|
export function isUnusedNativeSessionBootstrap(
|
|
run: NativeSessionBootstrapState,
|
|
hasProviderEvents: boolean,
|
|
): boolean {
|
|
return (
|
|
run.processPid === null &&
|
|
run.processGroupId === null &&
|
|
run.processStartedAt === null &&
|
|
record(run.runnerProfileJson).sessionCheckpoint == null &&
|
|
!hasProviderEvents
|
|
);
|
|
}
|
|
|
|
/** Newest-first rows for one exact task/session; never skip provider progress. */
|
|
export function selectNativeSessionResumeRun<
|
|
T extends NativeSessionBootstrapState & {
|
|
id: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
nativeIssueId: string | null;
|
|
nativeSessionId: string | null;
|
|
runtimeMode: string | null;
|
|
status: string;
|
|
createdAt: Date;
|
|
},
|
|
>(input: {
|
|
runs: readonly T[];
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
normalizedSessionId: string;
|
|
currentRunId: string;
|
|
beforeCreatedAt: Date;
|
|
providerEvidenceRunIds: ReadonlySet<string>;
|
|
}): T | null {
|
|
for (const run of input.runs) {
|
|
if (
|
|
run.id === input.currentRunId ||
|
|
run.companyId !== input.companyId ||
|
|
run.agentId !== input.agentId ||
|
|
run.nativeIssueId !== input.issueId ||
|
|
run.nativeSessionId !== input.normalizedSessionId
|
|
)
|
|
continue;
|
|
if (
|
|
run.runtimeMode !== "native" ||
|
|
!LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(run.status)
|
|
)
|
|
return null;
|
|
// Even an incompatible checkpoint is a progress barrier. The caller must
|
|
// reject it or start a fresh session, not fall back to an older checkpoint.
|
|
if (record(run.runnerProfileJson).sessionCheckpoint != null)
|
|
return run.createdAt > input.beforeCreatedAt ? null : run;
|
|
if (
|
|
!isUnusedNativeSessionBootstrap(
|
|
run,
|
|
input.providerEvidenceRunIds.has(run.id),
|
|
)
|
|
)
|
|
return null;
|
|
}
|
|
return null;
|
|
}
|
|
|
|
export const NATIVE_SESSION_PROVIDER_EVENT_TYPES = [
|
|
"harness.ready",
|
|
"session.started",
|
|
"session.resumed",
|
|
"session.updated",
|
|
"turn.started",
|
|
"provider.event",
|
|
"provider.rpc_result",
|
|
];
|
|
|
|
export async function nativeSessionProviderEvidence(
|
|
db: Pick<Db, "select">,
|
|
runIds: string[],
|
|
): Promise<Set<string>> {
|
|
if (!runIds.length) return new Set();
|
|
const rows = await db
|
|
.select({ runId: heartbeatRunEvents.runId })
|
|
.from(heartbeatRunEvents)
|
|
.where(
|
|
and(
|
|
inArray(heartbeatRunEvents.runId, runIds),
|
|
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
|
|
),
|
|
)
|
|
.groupBy(heartbeatRunEvents.runId);
|
|
return new Set(rows.map((row) => row.runId));
|
|
}
|
|
|
|
type NativeSessionResumeSelection = {
|
|
companyId: string;
|
|
agentId: string;
|
|
issueId: string;
|
|
normalizedSessionId: string;
|
|
currentRunId?: string;
|
|
beforeCreatedAt: Date;
|
|
/** Recovery discovery must retain an ineligible newest progress barrier. */
|
|
includeProgressBarrier?: boolean;
|
|
};
|
|
|
|
const NATIVE_RESUME_CANDIDATE_WINDOW = 32;
|
|
const INHERITED_CHECKPOINT_SOURCE_FIELDS = ["backendKind", "driverKind", "sessionId", "providerSessionId"] as const;
|
|
|
|
export async function findNativeSessionResumeRun(db: Db, input: NativeSessionResumeSelection) {
|
|
if (!isNativeSessionId(input.normalizedSessionId)) return null;
|
|
const barrier = await selectNativeSessionResumeBarrier(db, input, []);
|
|
// A follow-up that failed before any provider or controller work holds only
|
|
// the checkpoint it inherited. Select its source instead, but only when
|
|
// every skipped run proves that and inherited exactly that source session.
|
|
const skipped: NonNullable<typeof barrier>[] = [];
|
|
let selected = barrier;
|
|
while (selected && skipped.length < NATIVE_RESUME_CANDIDATE_WINDOW &&
|
|
await isUnstartedInheritedFollowUp(db, input, selected)) {
|
|
skipped.push(selected);
|
|
selected = await selectNativeSessionResumeBarrier(db, input, skipped.map(run => run.id));
|
|
}
|
|
if (!skipped.length) return selected;
|
|
const source = record(selected?.runnerProfileJson.sessionCheckpoint);
|
|
const inherited = selected !== null && typeof source.sessionId === "string" && skipped.every(run => {
|
|
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
|
|
return INHERITED_CHECKPOINT_SOURCE_FIELDS.every(field => checkpoint[field] === source[field]);
|
|
});
|
|
return inherited ? selected : barrier;
|
|
}
|
|
|
|
/** The run row, its coordinator, and the absence of controller checkpoints,
|
|
* results and provider events together prove no provider work started. */
|
|
async function isUnstartedInheritedFollowUp(
|
|
db: Db,
|
|
input: NativeSessionResumeSelection,
|
|
run: { id: string; runtimeMode: string | null; status: string; processPid: number | null;
|
|
processGroupId: number | null; processStartedAt: Date | null; runnerProfileJson: { sessionCheckpoint: unknown; nativeControllerRecovery: unknown } },
|
|
): Promise<boolean> {
|
|
const checkpoint = record(run.runnerProfileJson.sessionCheckpoint);
|
|
const identity = record(checkpoint.identity);
|
|
if (run.runtimeMode !== "native" || run.status !== "failed" || run.processPid !== null ||
|
|
run.processGroupId !== null || run.processStartedAt !== null || run.runnerProfileJson.nativeControllerRecovery != null ||
|
|
identity.runId !== run.id || identity.sessionId !== input.normalizedSessionId || checkpoint.cursor !== null) return false;
|
|
const [evidence] = await db.select({
|
|
executionRecovery: sql<unknown>`${heartbeatRuns.resultJson}->'executionRecovery'`,
|
|
phase: nativeRunFinalizations.phase,
|
|
resultId: nativeRunFinalizations.resultId,
|
|
leaseOwner: nativeRunFinalizations.leaseOwner,
|
|
controllerPid: nativeRunFinalizations.controllerPid,
|
|
controllerCheckpoint: sql<boolean>`exists (select 1 from ${nativeRunnerCheckpoints}
|
|
where ${nativeRunnerCheckpoints.companyId} = ${heartbeatRuns.companyId} and ${nativeRunnerCheckpoints.runId} = ${heartbeatRuns.id})`,
|
|
result: sql<boolean>`exists (select 1 from ${nativeRunResults}
|
|
where ${nativeRunResults.companyId} = ${heartbeatRuns.companyId} and ${nativeRunResults.runId} = ${heartbeatRuns.id})`,
|
|
}).from(heartbeatRuns).innerJoin(nativeRunFinalizations, and(
|
|
eq(nativeRunFinalizations.runId, heartbeatRuns.id), eq(nativeRunFinalizations.companyId, heartbeatRuns.companyId),
|
|
eq(nativeRunFinalizations.issueId, input.issueId),
|
|
)).where(and(eq(heartbeatRuns.id, run.id), eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId), eq(heartbeatRuns.nativeIssueId, input.issueId)));
|
|
const recovery = record(evidence?.executionRecovery);
|
|
return Boolean(evidence && recovery.kind === "bootstrap" && recovery.providerWorkStarted === false &&
|
|
evidence.phase === "terminal_failure" && evidence.resultId === null && evidence.leaseOwner === null &&
|
|
evidence.controllerPid === null && !evidence.controllerCheckpoint && !evidence.result &&
|
|
!(await nativeSessionProviderEvidence(db, [run.id])).has(run.id));
|
|
}
|
|
|
|
async function selectNativeSessionResumeBarrier(db: Db, input: NativeSessionResumeSelection, skippedRunIds: string[]) {
|
|
const candidates = db.$with("native_resume_candidates").as(db
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt,
|
|
hasCheckpoint: sql<boolean>`coalesce(${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint' <> 'null'::jsonb, false)`.as("has_checkpoint"),
|
|
})
|
|
.from(heartbeatRuns)
|
|
.where(
|
|
and(
|
|
input.currentRunId ? ne(heartbeatRuns.id, input.currentRunId) : undefined,
|
|
eq(heartbeatRuns.companyId, input.companyId),
|
|
eq(heartbeatRuns.agentId, input.agentId),
|
|
eq(heartbeatRuns.nativeIssueId, input.issueId),
|
|
eq(heartbeatRuns.nativeSessionId, input.normalizedSessionId),
|
|
),
|
|
)
|
|
.orderBy(desc(heartbeatRuns.createdAt), desc(heartbeatRuns.id))
|
|
.limit(NATIVE_RESUME_CANDIDATE_WINDOW));
|
|
const terminalStatuses = [...LEGACY_RETRY_SOURCE_TERMINAL_STATUSES];
|
|
const barrier = db.$with("native_resume_barrier").as(db
|
|
.select()
|
|
.from(candidates)
|
|
.where(and(skippedRunIds.length ? notInArray(candidates.id, skippedRunIds) : undefined, or(
|
|
ne(candidates.runtimeMode, "native"),
|
|
not(inArray(candidates.status, terminalStatuses)),
|
|
candidates.hasCheckpoint,
|
|
isNotNull(candidates.processPid),
|
|
isNotNull(candidates.processGroupId),
|
|
isNotNull(candidates.processStartedAt),
|
|
exists(db.select({ runId: heartbeatRunEvents.runId })
|
|
.from(heartbeatRunEvents)
|
|
.where(and(
|
|
eq(heartbeatRunEvents.runId, candidates.id),
|
|
inArray(heartbeatRunEvents.eventType, NATIVE_SESSION_PROVIDER_EVENT_TYPES),
|
|
))),
|
|
)))
|
|
.orderBy(desc(candidates.createdAt), desc(candidates.id))
|
|
.limit(1));
|
|
const [run] = await db.with(candidates, barrier)
|
|
.select({
|
|
id: heartbeatRuns.id,
|
|
companyId: heartbeatRuns.companyId,
|
|
agentId: heartbeatRuns.agentId,
|
|
runnerInstanceId: heartbeatRuns.runnerInstanceId,
|
|
nativeSessionId: heartbeatRuns.nativeSessionId,
|
|
nativeIssueId: heartbeatRuns.nativeIssueId,
|
|
runtimeMode: heartbeatRuns.runtimeMode,
|
|
status: heartbeatRuns.status,
|
|
errorCode: heartbeatRuns.errorCode,
|
|
createdAt: heartbeatRuns.createdAt,
|
|
processPid: heartbeatRuns.processPid,
|
|
processGroupId: heartbeatRuns.processGroupId,
|
|
processStartedAt: heartbeatRuns.processStartedAt,
|
|
runnerProfileJson: sql<{
|
|
nativeToolContractFingerprint: unknown;
|
|
nativeExecutionInput: unknown;
|
|
sessionCheckpoint: unknown;
|
|
nativeControllerRecovery: unknown;
|
|
nativeProviderStateUnsaved?: unknown;
|
|
}>`jsonb_build_object(
|
|
'nativeToolContractFingerprint', ${heartbeatRuns.runnerProfileJson}->'nativeToolContractFingerprint',
|
|
'nativeExecutionInput', ${heartbeatRuns.runnerProfileJson}->'nativeExecutionInput',
|
|
'sessionCheckpoint', ${heartbeatRuns.runnerProfileJson}->'sessionCheckpoint',
|
|
'nativeControllerRecovery', ${heartbeatRuns.runnerProfileJson}->'nativeControllerRecovery',
|
|
'${sql.raw(NATIVE_PROVIDER_STATE_UNSAVED_KEY)}', ${heartbeatRuns.runnerProfileJson}->'${sql.raw(NATIVE_PROVIDER_STATE_UNSAVED_KEY)}'
|
|
)`,
|
|
contextSnapshot: sql<{
|
|
paperclipWake: unknown;
|
|
paperclipIssue: unknown;
|
|
}>`jsonb_build_object(
|
|
'paperclipWake', ${heartbeatRuns.contextSnapshot}->'paperclipWake',
|
|
'paperclipIssue', case when jsonb_typeof(${heartbeatRuns.contextSnapshot}->'paperclipIssue') = 'object'
|
|
then (select coalesce(jsonb_object_agg(brief.key, brief.value), '{}'::jsonb)
|
|
from jsonb_each(${heartbeatRuns.contextSnapshot}->'paperclipIssue') as brief
|
|
where brief.key in ('id', 'title', 'description')) else ${heartbeatRuns.contextSnapshot}->'paperclipIssue' end
|
|
)`,
|
|
})
|
|
.from(heartbeatRuns)
|
|
.innerJoin(barrier, eq(heartbeatRuns.id, barrier.id))
|
|
.where(and(
|
|
...(input.includeProgressBarrier ? [] : [
|
|
eq(barrier.runtimeMode, "native"),
|
|
inArray(barrier.status, terminalStatuses),
|
|
eq(barrier.hasCheckpoint, true),
|
|
sql`date_trunc('milliseconds', ${barrier.createdAt}) <= ${input.beforeCreatedAt.toISOString()}::timestamptz`,
|
|
]),
|
|
sql`${heartbeatRuns.companyId}::text = ${input.companyId}`,
|
|
sql`${heartbeatRuns.agentId}::text = ${input.agentId}`,
|
|
sql`${heartbeatRuns.nativeIssueId}::text = ${input.issueId}`,
|
|
sql`${heartbeatRuns.nativeSessionId}::text = ${input.normalizedSessionId}`,
|
|
));
|
|
return run ?? null;
|
|
}
|
|
|
|
/** A preassigned session id may rotate only before immutable native admission. */
|
|
export function nativeSessionIdForBootstrapPersistence(input: {
|
|
run: NativeSessionBootstrapState & { nativeSessionId: string | null };
|
|
selectedSessionId: string;
|
|
hasProviderEvents: boolean;
|
|
}): string {
|
|
if (input.run.nativeSessionId === input.selectedSessionId)
|
|
return input.selectedSessionId;
|
|
if (
|
|
record(input.run.runnerProfileJson).nativeExecutionInput !== undefined ||
|
|
!isUnusedNativeSessionBootstrap(input.run, input.hasProviderEvents)
|
|
) {
|
|
throw new Error("native_session_bootstrap_identity_conflict");
|
|
}
|
|
return input.selectedSessionId;
|
|
}
|
|
|
|
/** Caller holds the run row lock; recheck authority after waiting for that lock. */
|
|
export async function prepareNativeSessionBootstrapPersistence(
|
|
db: Pick<Db, "select">,
|
|
input: {
|
|
run: NativeSessionBootstrapState & {
|
|
id: string;
|
|
nativeSessionId: string | null;
|
|
};
|
|
selectedSessionId: string;
|
|
execution: NativeExecutionInput;
|
|
restoringCheckpoint: boolean;
|
|
},
|
|
) {
|
|
const hasProviderEvents = (
|
|
await nativeSessionProviderEvidence(db, [input.run.id])
|
|
).has(input.run.id);
|
|
const profile = record(input.run.runnerProfileJson);
|
|
if (
|
|
profile.nativeExecutionInput !== undefined &&
|
|
!isDeepStrictEqual(
|
|
parseNativeExecutionInput(profile.nativeExecutionInput),
|
|
input.execution,
|
|
)
|
|
) {
|
|
throw new Error("native_execution_input_persisted_binding_mismatch");
|
|
}
|
|
if (
|
|
input.restoringCheckpoint &&
|
|
!isUnusedNativeSessionBootstrap(input.run, hasProviderEvents)
|
|
) {
|
|
throw new Error("native_session_bootstrap_identity_conflict");
|
|
}
|
|
return nativeSessionIdForBootstrapPersistence({
|
|
...input,
|
|
hasProviderEvents,
|
|
});
|
|
}
|
|
|
|
/**
|
|
* Legacy compatibility is deliberately narrower than ordinary task-session
|
|
* continuation: only a replacement row that never acquired any native process
|
|
* or provider authority may be rebound to a terminal native source. The exact
|
|
* checkpoint/session/workspace/provider binding is validated separately by
|
|
* rebindNativeSessionCheckpoint.
|
|
*/
|
|
export function isUnusedLegacyNativeRetryReplacement(input: {
|
|
replacement: {
|
|
processPid: number | null;
|
|
processGroupId: number | null;
|
|
processStartedAt: Date | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
source: {
|
|
runtimeMode: string | null;
|
|
status: string;
|
|
nativeSessionId: string | null;
|
|
} | null;
|
|
hasProviderEvents: boolean;
|
|
}): boolean {
|
|
return Boolean(
|
|
input.source?.runtimeMode === "native" &&
|
|
LEGACY_RETRY_SOURCE_TERMINAL_STATUSES.has(input.source.status) &&
|
|
isNativeSessionId(input.source.nativeSessionId) &&
|
|
isUnusedNativeSessionBootstrap(input.replacement, input.hasProviderEvents),
|
|
);
|
|
}
|
|
|
|
function sameProvider(
|
|
previous: NativeExecutionInput["provider"],
|
|
current: NativeExecutionInput["provider"],
|
|
): boolean {
|
|
return JSON.stringify(previous) === JSON.stringify(current);
|
|
}
|
|
|
|
function nativeCheckpointWorkspaceScope(
|
|
execution: NativeExecutionInput,
|
|
owningRunId: string,
|
|
) {
|
|
// Projectless work uses each heartbeat run id as a placeholder workspace id.
|
|
// Continuity is safe only when both sides independently prove that shape and
|
|
// their complete immutable workspace descriptors match. A real managed
|
|
// workspace instead binds continuity to its durable workspace row id.
|
|
return execution.binding.executionWorkspaceId === owningRunId
|
|
? {
|
|
kind: "transient" as const,
|
|
cwd: execution.workspace.cwd,
|
|
repoUrl: execution.workspace.repoUrl,
|
|
repoRef: execution.workspace.repoRef,
|
|
branchName: execution.workspace.branchName,
|
|
}
|
|
: {
|
|
kind: "managed" as const,
|
|
id: execution.binding.executionWorkspaceId,
|
|
};
|
|
}
|
|
|
|
function sameWorkspaceScope(input: {
|
|
previousExecution: NativeExecutionInput;
|
|
previousRunId: string;
|
|
currentExecution: NativeExecutionInput;
|
|
}): boolean {
|
|
return (
|
|
JSON.stringify(
|
|
nativeCheckpointWorkspaceScope(
|
|
input.previousExecution,
|
|
input.previousRunId,
|
|
),
|
|
) ===
|
|
JSON.stringify(
|
|
nativeCheckpointWorkspaceScope(
|
|
input.currentExecution,
|
|
input.currentExecution.binding.runId,
|
|
),
|
|
)
|
|
);
|
|
}
|
|
|
|
/** A resume delta is valid only if the provider checkpoint really can be used. */
|
|
export function buildNativeExecutionWithCheckpoint(input: {
|
|
previousRun:
|
|
Parameters<typeof rebindNativeSessionCheckpoint>[0]["previousRun"] | null;
|
|
normalizedSessionId: string;
|
|
executionTargetKind?: NativeToolExecutionTargetKind;
|
|
toolRefreshOnResume?: boolean;
|
|
refreshTools?: boolean;
|
|
buildExecution: (options: {
|
|
normalizedSessionId: string;
|
|
resumedSession: boolean;
|
|
}) => NativeExecutionInput;
|
|
}): {
|
|
execution: NativeExecutionInput;
|
|
checkpoint: PersistedNativeSession | null;
|
|
normalizedSessionId: string;
|
|
} {
|
|
const execution = input.buildExecution({
|
|
normalizedSessionId: input.normalizedSessionId,
|
|
resumedSession: input.previousRun !== null,
|
|
});
|
|
// A presentation-only upgrade must not rotate a healthy provider session or
|
|
// lose a durable goal. Keep its v4/v5 format until that session naturally ends.
|
|
// All ordinary identity, workspace, provider, tool and recovery checks still apply.
|
|
if (input.previousRun && (execution.schema === "paperclip.native-execution-input.v4" || execution.schema === "paperclip.native-execution-input.v5")) {
|
|
const previousSchema = record(record(input.previousRun.runnerProfileJson).nativeExecutionInput).schema;
|
|
if ((previousSchema === "paperclip.native-execution-input.v4" || previousSchema === "paperclip.native-execution-input.v5") && previousSchema !== execution.schema) {
|
|
const { completionSources: _sources, ...common } = execution as typeof execution & { completionSources?: unknown };
|
|
const retainedFormat = parseNativeExecutionInput({ ...common, schema: previousSchema });
|
|
const retainedCheckpoint = rebindNativeSessionCheckpoint({
|
|
previousRun: input.previousRun,
|
|
currentExecution: retainedFormat,
|
|
executionTargetKind: input.executionTargetKind,
|
|
toolRefreshOnResume: input.toolRefreshOnResume,
|
|
refreshTools: input.refreshTools,
|
|
});
|
|
if (retainedCheckpoint) return { execution: retainedFormat, checkpoint: retainedCheckpoint, normalizedSessionId: input.normalizedSessionId };
|
|
}
|
|
}
|
|
const checkpoint = input.previousRun
|
|
? rebindNativeSessionCheckpoint({
|
|
previousRun: input.previousRun,
|
|
currentExecution: execution,
|
|
executionTargetKind: input.executionTargetKind,
|
|
toolRefreshOnResume: input.toolRefreshOnResume,
|
|
refreshTools: input.refreshTools,
|
|
})
|
|
: null;
|
|
if (checkpoint)
|
|
return {
|
|
execution,
|
|
checkpoint,
|
|
normalizedSessionId: input.normalizedSessionId,
|
|
};
|
|
// Rebuild both task context and wake instructions. Merely rotating the ID
|
|
// leaves a fresh provider with a compact delta and missing task context.
|
|
const normalizedSessionId = randomUUID();
|
|
return {
|
|
execution: input.buildExecution({
|
|
normalizedSessionId,
|
|
resumedSession: false,
|
|
}),
|
|
checkpoint: null,
|
|
normalizedSessionId,
|
|
};
|
|
}
|
|
|
|
export function nativeRuntimeContextsMatch(
|
|
previous: NativeRuntimeContextSnapshot,
|
|
current: NativeRuntimeContextSnapshot,
|
|
toolRefreshOnResume?: boolean,
|
|
): boolean {
|
|
return previous.aggregateDigest === current.aggregateDigest ||
|
|
(toolRefreshOnResume === true && canonicalNativeRuntimeContextDigest({ ...previous, mcp: current.mcp }) === current.aggregateDigest);
|
|
}
|
|
|
|
/**
|
|
* Rebind a completed prior run's provider checkpoint to a new heartbeat run.
|
|
* The provider/driver session identity is retained, while every per-turn and
|
|
* per-event field is reset so the new run starts one clean turn via resume.
|
|
*/
|
|
export function rebindNativeSessionCheckpoint(input: {
|
|
previousRun: {
|
|
id: string;
|
|
companyId: string;
|
|
agentId: string;
|
|
nativeSessionId: string | null;
|
|
errorCode?: string | null;
|
|
runnerProfileJson: unknown;
|
|
};
|
|
currentExecution: NativeExecutionInput;
|
|
executionTargetKind?: NativeToolExecutionTargetKind;
|
|
toolRefreshOnResume?: boolean;
|
|
refreshTools?: boolean;
|
|
}): PersistedNativeSession | null {
|
|
if (input.refreshTools && input.toolRefreshOnResume !== true) return null;
|
|
// The source is still the progress barrier. Its provider session cannot be
|
|
// restored, so the next run starts a fresh session in the same workspace.
|
|
if (input.previousRun.errorCode === NATIVE_PROVIDER_STATE_INCOMPATIBLE) return null;
|
|
// The saved session still holds the unsettled call and refuses attachment.
|
|
// Generic Retry is blocked until an operator reconciles that action; the
|
|
// reconciled continuation then starts a fresh session in the same workspace.
|
|
if (input.previousRun.errorCode === NATIVE_PROVIDER_SIDE_EFFECT_PENDING) return null;
|
|
const previousProfile = record(input.previousRun.runnerProfileJson);
|
|
// The source's result was accepted, but its provider state was never saved
|
|
// and its sandbox is gone. Resuming would only fail, so start fresh now.
|
|
if (isNativeProviderStateUnsaved(previousProfile)) return null;
|
|
if (
|
|
previousProfile.nativeToolContractFingerprint !==
|
|
nativeToolContractFingerprintForTarget(input.executionTargetKind ?? "local", input.currentExecution.provider.kind)
|
|
) {
|
|
return null;
|
|
}
|
|
const rawCheckpoint = record(previousProfile.sessionCheckpoint);
|
|
const checkpointIdentity = record(rawCheckpoint.identity);
|
|
const current = input.currentExecution;
|
|
const normalizedSessionId = current.session.normalizedSessionId;
|
|
if (
|
|
!isNativeSessionId(normalizedSessionId) ||
|
|
input.previousRun.companyId !== current.binding.companyId ||
|
|
input.previousRun.agentId !== current.binding.agentId ||
|
|
input.previousRun.nativeSessionId !== normalizedSessionId ||
|
|
typeof rawCheckpoint.sessionId !== "string" ||
|
|
checkpointIdentity.runId !== input.previousRun.id ||
|
|
checkpointIdentity.companyId !== current.binding.companyId ||
|
|
checkpointIdentity.issueId !== current.binding.issueId ||
|
|
checkpointIdentity.agentId !== current.binding.agentId ||
|
|
checkpointIdentity.sessionId !== normalizedSessionId
|
|
)
|
|
return null;
|
|
|
|
let previousExecution: NativeExecutionInput;
|
|
try {
|
|
previousExecution = parseNativeExecutionInput(
|
|
previousProfile.nativeExecutionInput,
|
|
);
|
|
} catch {
|
|
return null;
|
|
}
|
|
if (
|
|
previousExecution.binding.runId !== input.previousRun.id ||
|
|
previousExecution.binding.companyId !== current.binding.companyId ||
|
|
previousExecution.binding.issueId !== current.binding.issueId ||
|
|
previousExecution.binding.agentId !== current.binding.agentId ||
|
|
previousExecution.session.normalizedSessionId !== normalizedSessionId ||
|
|
previousExecution.session.driverKind !== current.session.driverKind ||
|
|
!sameWorkspaceScope({
|
|
previousExecution,
|
|
previousRunId: input.previousRun.id,
|
|
currentExecution: current,
|
|
}) ||
|
|
previousExecution.task.workMode !== current.task.workMode ||
|
|
("executionMode" in previousExecution
|
|
? previousExecution.executionMode
|
|
: "default") !==
|
|
("executionMode" in current ? current.executionMode : "default") ||
|
|
!sameProvider(previousExecution.provider, current.provider) ||
|
|
previousExecution.schema !== current.schema ||
|
|
("homeMode" in previousExecution ? previousExecution.homeMode : "isolated") !== ("homeMode" in current ? current.homeMode : "isolated") ||
|
|
("runtimeContext" in previousExecution &&
|
|
"runtimeContext" in current &&
|
|
!nativeRuntimeContextsMatch(previousExecution.runtimeContext, current.runtimeContext, input.toolRefreshOnResume))
|
|
)
|
|
return null;
|
|
|
|
const priorSemanticResult = record(rawCheckpoint.semanticResult);
|
|
const priorContinuation = record(priorSemanticResult.continuation);
|
|
const rawGoal = rawCheckpoint.goal;
|
|
const hasUnfinishedGoal =
|
|
rawGoal !== null &&
|
|
rawGoal !== undefined &&
|
|
record(rawGoal).status !== "complete";
|
|
const providerRecoveryPolicy = hasUnfinishedGoal
|
|
? ("same_session_only" as const)
|
|
: !input.refreshTools && priorSemanticResult.reportedWorkDisposition === "yielded" &&
|
|
priorContinuation.kind === "response_wake"
|
|
? ("allow_replacement_after_governed_wait" as const)
|
|
: ("allow_replacement_after_resume_failure" as const);
|
|
|
|
return {
|
|
...(structuredClone(rawCheckpoint) as unknown as PersistedNativeSession),
|
|
identity: {
|
|
runId: current.binding.runId,
|
|
sessionId: normalizedSessionId,
|
|
companyId: current.binding.companyId,
|
|
issueId: current.binding.issueId,
|
|
agentId: current.binding.agentId,
|
|
},
|
|
cursor: null,
|
|
semanticResult: null,
|
|
terminal: null,
|
|
activeTurnId: null,
|
|
terminalTurns: [],
|
|
pendingRuntimeRequests: [],
|
|
dispositionOnlyRecoveryConsumed: false,
|
|
dispositionOnlyRecoveryTurnId: null,
|
|
// Settlement evidence belongs to the prior heartbeat run, even when the
|
|
// provider session is reused for the user's response.
|
|
governedWait: undefined,
|
|
providerRecoveryPolicy,
|
|
};
|
|
}
|