Files
PaperClipAI/server/src/services/native-runtime/native-session-resume-projection.test.ts
T
DottaandClaude c8ae02e5ce fix(native): start a fresh session when an accepted run's provider state was not saved (#20811)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run in a per-turn remote sandbox saves a provider state
backup after its result is accepted. The sandbox is then released, so
the backup is the only copy of the provider session.
> - When that backup fails to save (for example, #20772: storage
throttling), the run keeps its accepted result. Nothing records that its
checkpoint can no longer be resumed.
> - The next run on the session rebinds that checkpoint and asks the
Runner to resume. The Runner has no state and refuses (#20757). With the
normal policy, the run opens a replacement session after that failed
attempt. With a governed-wait or unfinished-goal checkpoint, the run
fails with `native_provider_state_incompatible`, and only Retry or the
next message starts fresh.
> - The server already knew the state was gone when the backup failed.
The failed resume is wasted work.
> - This pull request marks the accepted run's checkpoint as not
resumable at the moment the backup fails. The next run then starts a
fresh provider session directly.
> - The benefit is that a lost backup no longer costs a failed run or a
failed resume attempt before the task continues.

## Linked Issues or Issue Description

Refs #20757 (the missing-state refusal that the next run reaches today).
Refs #20772 (the throttled backup upload that loses the backup). Refs
#20809 (a related fix for governed-wait follow-ups that reach the
missing state; this PR stops most of them from reaching it).

**What happened?**

1. A native run in a per-turn sandbox finished, and its result was
accepted.
2. `harness_state.backup.persist` failed. The run recorded
`native.post_result_failure`. Its run row kept a normal
`sessionCheckpoint`.
3. The sandbox was released.
4. The next run on the same session selected that run as its resume
source, and `rebindNativeSessionCheckpoint` returned the checkpoint.
5. The Runner found no saved state and refused with `PRP provider resume
state is unavailable`. The run then opened a replacement session after
the failed attempt, or (governed-wait or unfinished-goal checkpoint)
failed with `native_provider_state_incompatible`. Retry or the next
message then started fresh.

**Expected behavior**

The next run starts a fresh provider session at once, because the server
already knows the saved state is not there. The accepted result and the
task outcome stay as they are.

**Steps to reproduce**

1. Give a run row an accepted result and a normal `sessionCheckpoint`.
2. Make its close checkpoint fail in a per-turn sandbox
(`reusableLeaseConfigured: false`).
3. Select the next run's resume source with `findNativeSessionResumeRun`
and call `rebindNativeSessionCheckpoint`.
4. On `master` it returns the checkpoint, so the next run tries to
resume state that does not exist.

**Paperclip version or commit**

`master` at `556b249897`.

## What Changed

- `server/src/services/native-runtime/native-provider-state-unsaved.ts`
(new): `markAcceptedRunProviderStateUnsaved` adds
`nativeProviderStateUnsaved: { code:
"native_provider_state_incompatible", reason: "harness_backup_not_saved"
}` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it
only when the run's coordinator holds a result with `schema_status =
'accepted'`. `clearRunProviderStateUnsaved` removes it.
- `server/src/services/native-runtime/native-session-executor.ts`: the
remote close checkpoint (`checkpointRemoteRunner`) now wraps the
existing body. When the checkpoint throws in a sandbox without a
reusable lease, it calls the marker, writes one run-log line, and throws
the original error. A marker or log failure cannot replace that error.
If a later checkpoint of the same run saves, it clears the mark.
- `server/src/services/native-runtime/native-session-resume.ts`:
`rebindNativeSessionCheckpoint` returns `null` for a source with the
mark, as it already does for `errorCode =
native_provider_state_incompatible`. `findNativeSessionResumeRun`
projects the new key.
- `doc/execution-semantics.md`: one additive paragraph.
- Tests in `native-session-resume.test.ts` and
`native-session-executor.test.ts`.

Guard conditions. The mark is written only when all of these are true:

- The close checkpoint threw before the durable backup was stored (after
the upload's own bounded retries). A later local publication, stamp or
cleanup failure does not mark the run.
- The target is a remote `sandbox` with `reusableLeaseConfigured !==
true`. A retained sandbox or an SSH host keeps the live state, so its
resume path is unchanged.
- The run holds an accepted result. A failure before acceptance changes
nothing.

Saved state from the previous release: run rows without the key keep
today's resume path. An older server that reads a row with the key
ignores it, and so it behaves as it does today. No schema change.

User-visible: the next run's transcript shows "Paperclip session
started" instead of "resumed", with a new session id. The source run's
log shows `provider state was not saved after the accepted result; the
next run starts a fresh provider session`. The fresh session gets the
full task context, as for other fresh starts.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-session-resume.test.ts -t "not
resumable only after|provider state was not saved"`: 9 pass (6 pure, 3
embedded Postgres).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts -t "marks
the provider state unsaved|uses the bundled Linux image authority for
remote opencode-new-durable"`: 8 pass. They include a durable store
failure (marked) and a local publication failure after the store (not
marked).
- Red on `master` (source files reverted, tests kept): the 3 "starts the
next run in a fresh provider session directly" cases fail (`expected {…}
to be null`), the DB "accepted" case fails the same way, and the
executor per-turn case fails (`mark` called 0 times).
- Guard removal checks:
- Sandbox guard replaced with `true`: the "retained sandbox" case fails
(`mark` called once).
  - `!checkpointBackupStored` removed: the "publish-failed" case fails.
- `schema_status = 'accepted'` removed from the marker: the "rejected"
case fails (`expected true to be false`).
- Negative cases that pass: no result, rejected result, retained
sandbox, records without the key or with another code (all three
recovery policies), and a later save that clears the mark.
- `cd server && npx tsc --noEmit -p .`: clean.
- Not related to this change and failing on `master` in this worktree
too: `runtime-selection.test.ts` connector skill cases (missing
`.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts`
"archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required
CI is the authority for those suites.

## Risks

- Low. The change only skips a resume that cannot succeed. The fresh
session uses the same workspace and full task context, like a Retry
after `native_provider_state_incompatible`.
- If the backup failed but the provider state were still somewhere
recoverable, the next run would lose provider context. For a per-turn
sandbox without a reusable lease, the restore path uses only the backup,
so this does not apply. Retained sandboxes and SSH hosts are excluded.
- One extra conditional `UPDATE` runs only on the failure path. The
normal path before `turn.start` has no new waits or work.
- `doc/execution-semantics.md` is also edited by #20809. The paragraphs
are at different places; whichever merges second rebases.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with
tool use (shell, file edit, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-11-11 10:36:00 -06:00

217 lines
13 KiB
TypeScript

import { randomUUID } from "node:crypto";
import { and, eq, sql } from "drizzle-orm";
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
import { findNativeSessionResumeRun } from "./native-session-resume.js";
import { resolveNativeControllerRecovery } from "./native-controller-recovery.js";
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL");
const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport()).supported;
(supported ? describe : describe.skip)("native session resume SQL projection", () => {
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
let db: Db;
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), normalizedSessionId = randomUUID();
const currentRunId = randomUUID();
const at = (second: number) => new Date(Date.UTC(2026, 8, 8, 0, 0, second));
const input = { companyId, agentId, issueId, normalizedSessionId, currentRunId, beforeCreatedAt: at(30) };
const checkpoint = { sessionId: "provider-session", identity: { runId: "prior" }, complete: [false, 0, "", null] };
const nativeExecutionInput = { binding: { issueId }, task: { prompt: "Complete native input", description: "Description" }, runtimeContext: { nested: [1, 2] } };
const paperclipWake = { externalChatProvider: "slack", githubReviewConfigurations: [{ key: "configuration", nested: { retained: true } }], other: ["preserved"] };
const paperclipIssue = { id: issueId, title: "Saved canonical title", description: "Saved canonical description" };
const profile = { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", unrelated: "Omit" };
const run = (second: number, overrides: Partial<typeof heartbeatRuns.$inferInsert> = {}) => ({
id: randomUUID(), companyId, agentId, nativeIssueId: issueId, nativeSessionId: normalizedSessionId,
runtimeMode: "native", status: "failed", createdAt: at(second), runnerProfileJson: {}, ...overrides,
});
const select = () => findNativeSessionResumeRun(db, input);
const source = () => run(0, { runnerInstanceId: randomUUID(), processPid: 17, processGroupId: 19, processStartedAt: at(0),
errorCode: "native_session_interrupted",
runnerProfileJson: profile, contextSnapshot: { paperclipWake, paperclipIssue: { ...paperclipIssue, unrelated: "Omit" }, prompt: "Omit" } });
const writeProfile = (id: string, value: unknown) => db.execute(sql`update ${heartbeatRuns}
set runner_profile_json = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${id}`);
beforeAll(async () => {
database = externalUrl ? { connectionString: externalUrl, cleanup: async () => {} } : await startEmbeddedPostgresTestDatabase("native-resume-projection-");
db = createDb(database.connectionString);
await db.insert(companies).values({ id: companyId, name: "Resume projection", issuePrefix: `N${companyId.replaceAll("-", "")}` });
await db.insert(agents).values({ id: agentId, companyId, name: "Runner" });
await db.insert(issues).values({ id: issueId, companyId, title: "Resume task" });
});
afterEach(async () => {
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
});
afterAll(async () => {
if (!db) return;
try {
await db.delete(issues).where(eq(issues.companyId, companyId));
await db.delete(agents).where(eq(agents.companyId, companyId));
await db.delete(companies).where(eq(companies.id, companyId));
} finally {
try { await db.$client.end({ timeout: 0 }); }
finally { await database.cleanup(); }
}
});
it("retains full required input, checkpoint, wake, canonical brief and process identity only for the selected source", async () => {
const saved = source();
await db.insert(heartbeatRuns).values([saved, run(1), run(2)]);
const result = await select();
expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", nativeControllerRecovery: null,
nativeProviderStateUnsaved: null },
contextSnapshot: { paperclipWake, paperclipIssue } });
const stored = await db.select({ profile: heartbeatRuns.runnerProfileJson, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.id));
expect(stored[0].profile).toEqual(profile);
expect(stored[0].context).toEqual(saved.contextSnapshot);
for (const status of ["succeeded", "interrupted", "cancelled", "timed_out"]) {
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, saved.id));
expect((await select())?.id).toBe(saved.id);
}
});
it("treats every non-null checkpoint as a barrier, including invalid false, zero, empty string, array and object values", async () => {
const saved = source(), newer = run(1);
await db.insert(heartbeatRuns).values([saved, newer]);
for (const value of [false, 0, "", "invalid", [], {}]) {
await writeProfile(newer.id, { sessionCheckpoint: value });
const selected = await select();
expect(selected?.id).toBe(newer.id);
expect(selected?.runnerProfileJson.sessionCheckpoint).toEqual(value);
}
});
it("does not call missing or newer incompatible recovery material a fresh session", async () => {
const saved = source();
await db.insert(heartbeatRuns).values([saved, run(30, { id: currentRunId, status: "running" })]);
const { beforeCreatedAt: _before, ...binding } = input;
expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "snapshot_binding" });
// The newest progress barrier remains authoritative even without a snapshot.
await db.insert(heartbeatRuns).values(run(10, { runtimeMode: "legacy" }));
expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "newer_progress_barrier" });
});
it("skips only unused SQL-null, JSON-null, scalar, array and checkpoint-null profiles", async () => {
const saved = source(), newer = run(1, { runnerProfileJson: null });
await db.insert(heartbeatRuns).values([saved, newer]);
expect((await select())?.id).toBe(saved.id);
for (const value of [null, false, 0, "sessionCheckpoint", [], ["sessionCheckpoint"], {}, { sessionCheckpoint: null }]) {
await writeProfile(newer.id, value);
expect((await select())?.id).toBe(saved.id);
}
});
it("preserves malformed selected context values and missing brief fields", async () => {
const saved = source();
await db.insert(heartbeatRuns).values(saved);
for (const value of [null, false, 0, "invalid", [], {}, { paperclipWake: [], paperclipIssue: { id: issueId, title: false } },
{ paperclipIssue: { id: issueId, title: "Valid title without description" } },
{ paperclipIssue: { id: issueId, title: "Valid title", description: null } }]) {
await db.execute(sql`update ${heartbeatRuns} set context_snapshot = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${saved.id}`);
const result = await select();
const object = typeof value === "object" && value !== null && !Array.isArray(value) ? value : {};
expect(result?.contextSnapshot.paperclipWake).toEqual("paperclipWake" in object ? object.paperclipWake : null);
expect(result?.contextSnapshot.paperclipIssue).toEqual("paperclipIssue" in object ? object.paperclipIssue : null);
}
});
it("stops at wrong runtime, nonterminal status and each process authority field", async () => {
await db.insert(heartbeatRuns).values(source());
for (const override of [{ runtimeMode: "legacy" }, { status: "running" }, { processPid: 0 }, { processGroupId: 0 }, { processStartedAt: at(0) }]) {
const newer = run(1, override);
await db.insert(heartbeatRuns).values(newer);
expect(await select()).toBeNull();
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, newer.id));
}
});
it("checks every provider evidence event and ignores unrelated event kinds", async () => {
const saved = source(), newer = run(1);
await db.insert(heartbeatRuns).values([saved, newer]);
for (const eventType of ["harness.ready", "session.started", "session.resumed", "session.updated", "turn.started", "provider.event", "provider.rpc_result", "log"]) {
await db.insert(heartbeatRunEvents).values({ companyId, agentId, runId: newer.id, seq: 1, eventType });
if (eventType === "log") expect((await select())?.id).toBe(saved.id);
else expect(await select()).toBeNull();
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, newer.id));
}
});
it("excludes current and mismatched agent, company, task and session rows before applying barriers", async () => {
const saved = source(), otherCompany = randomUUID(), otherAgent = randomUUID(), otherIssue = randomUUID();
await db.insert(companies).values({ id: otherCompany, name: "Other", issuePrefix: `D${otherCompany.replaceAll("-", "")}` });
await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other" });
await db.insert(issues).values({ id: otherIssue, companyId, title: "Other" });
try {
await db.insert(heartbeatRuns).values([saved, run(1, { id: currentRunId, status: "running" }),
run(2, { companyId: otherCompany, status: "running" }), run(3, { agentId: otherAgent, status: "running" }),
run(4, { nativeIssueId: otherIssue, status: "running" }), run(5, { nativeSessionId: randomUUID(), status: "running" })]);
expect((await select())?.id).toBe(saved.id);
} finally {
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, otherCompany));
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.agentId, otherAgent));
await db.delete(issues).where(eq(issues.id, otherIssue));
await db.delete(agents).where(eq(agents.id, otherAgent));
await db.delete(companies).where(eq(companies.id, otherCompany));
}
expect(await findNativeSessionResumeRun(db, { ...input, normalizedSessionId: "invalid" })).toBeNull();
for (const key of ["companyId", "agentId", "issueId", "normalizedSessionId"] as const) {
expect(await findNativeSessionResumeRun(db, { ...input, [key]: input[key].toUpperCase() })).toBeNull();
}
});
it("retains the 32-row bound and deterministic created-at/id ordering", async () => {
const saved = source(), unused = Array.from({ length: 32 }, () => run(1));
await db.insert(heartbeatRuns).values([saved, ...unused]);
expect(await select()).toBeNull();
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, unused[0].id));
expect((await select())?.id).toBe(saved.id);
const tied = [run(2, { runnerProfileJson: { sessionCheckpoint: false } }), run(2, { status: "running" })].sort((a, b) => b.id.localeCompare(a.id));
await db.insert(heartbeatRuns).values(tied);
if (tied[0].status === "running") expect(await select()).toBeNull();
else expect((await select())?.id).toBe(tied[0].id);
});
it("does not skip future checkpoint barriers and preserves Date millisecond comparison", async () => {
const saved = source(), future = run(31);
await db.insert(heartbeatRuns).values([saved, future]);
expect((await select())?.id).toBe(saved.id);
await writeProfile(future.id, { sessionCheckpoint: false });
expect(await select()).toBeNull();
await db.execute(sql`update ${heartbeatRuns} set created_at = '2026-09-08T00:00:30.000999Z'::timestamptz where ${heartbeatRuns.id} = ${future.id}`);
expect((await select())?.id).toBe(future.id);
});
it("sees a newly committed barrier and changed source binding in its single selection snapshot", async () => {
const saved = source();
await db.insert(heartbeatRuns).values(saved);
let reads = 0;
const changing = new Proxy(db, { get(target, key) {
if (key !== "with") return Reflect.get(target, key);
return (...ctes: Parameters<Db["with"]>) => {
const builder = target.with(...ctes), original = builder.select.bind(builder);
builder.select = ((...fields: Parameters<typeof original>) => {
const selection = original(...fields), from = selection.from.bind(selection);
selection.from = ((...args: Parameters<typeof from>) => {
const query = from(...args), execute = query.execute.bind(query);
query.execute = async (...executeArgs: Parameters<typeof execute>) => {
reads++;
await db.insert(heartbeatRuns).values(run(1, { runnerProfileJson: { sessionCheckpoint: 0 } }));
await db.update(heartbeatRuns).set({ nativeSessionId: randomUUID() }).where(and(eq(heartbeatRuns.id, saved.id), eq(heartbeatRuns.companyId, companyId)));
return execute(...executeArgs);
};
return query;
}) as typeof selection.from;
return selection;
}) as typeof builder.select;
return builder;
};
}});
const selected = await findNativeSessionResumeRun(changing, input);
expect(reads).toBe(1);
expect(selected?.id).not.toBe(saved.id);
expect(selected?.runnerProfileJson.sessionCheckpoint).toBe(0);
});
});