mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-22 11:54:35 +01:00
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - A native run in a per-turn remote sandbox saves a provider state backup after its result is accepted. The sandbox is then released, so the backup is the only copy of the provider session. > - When that backup fails to save (for example, #20772: storage throttling), the run keeps its accepted result. Nothing records that its checkpoint can no longer be resumed. > - The next run on the session rebinds that checkpoint and asks the Runner to resume. The Runner has no state and refuses (#20757). With the normal policy, the run opens a replacement session after that failed attempt. With a governed-wait or unfinished-goal checkpoint, the run fails with `native_provider_state_incompatible`, and only Retry or the next message starts fresh. > - The server already knew the state was gone when the backup failed. The failed resume is wasted work. > - This pull request marks the accepted run's checkpoint as not resumable at the moment the backup fails. The next run then starts a fresh provider session directly. > - The benefit is that a lost backup no longer costs a failed run or a failed resume attempt before the task continues. ## Linked Issues or Issue Description Refs #20757 (the missing-state refusal that the next run reaches today). Refs #20772 (the throttled backup upload that loses the backup). Refs #20809 (a related fix for governed-wait follow-ups that reach the missing state; this PR stops most of them from reaching it). **What happened?** 1. A native run in a per-turn sandbox finished, and its result was accepted. 2. `harness_state.backup.persist` failed. The run recorded `native.post_result_failure`. Its run row kept a normal `sessionCheckpoint`. 3. The sandbox was released. 4. The next run on the same session selected that run as its resume source, and `rebindNativeSessionCheckpoint` returned the checkpoint. 5. The Runner found no saved state and refused with `PRP provider resume state is unavailable`. The run then opened a replacement session after the failed attempt, or (governed-wait or unfinished-goal checkpoint) failed with `native_provider_state_incompatible`. Retry or the next message then started fresh. **Expected behavior** The next run starts a fresh provider session at once, because the server already knows the saved state is not there. The accepted result and the task outcome stay as they are. **Steps to reproduce** 1. Give a run row an accepted result and a normal `sessionCheckpoint`. 2. Make its close checkpoint fail in a per-turn sandbox (`reusableLeaseConfigured: false`). 3. Select the next run's resume source with `findNativeSessionResumeRun` and call `rebindNativeSessionCheckpoint`. 4. On `master` it returns the checkpoint, so the next run tries to resume state that does not exist. **Paperclip version or commit** `master` at `556b249897`. ## What Changed - `server/src/services/native-runtime/native-provider-state-unsaved.ts` (new): `markAcceptedRunProviderStateUnsaved` adds `nativeProviderStateUnsaved: { code: "native_provider_state_incompatible", reason: "harness_backup_not_saved" }` to the run's `runnerProfileJson`. One conditional `UPDATE` writes it only when the run's coordinator holds a result with `schema_status = 'accepted'`. `clearRunProviderStateUnsaved` removes it. - `server/src/services/native-runtime/native-session-executor.ts`: the remote close checkpoint (`checkpointRemoteRunner`) now wraps the existing body. When the checkpoint throws in a sandbox without a reusable lease, it calls the marker, writes one run-log line, and throws the original error. A marker or log failure cannot replace that error. If a later checkpoint of the same run saves, it clears the mark. - `server/src/services/native-runtime/native-session-resume.ts`: `rebindNativeSessionCheckpoint` returns `null` for a source with the mark, as it already does for `errorCode = native_provider_state_incompatible`. `findNativeSessionResumeRun` projects the new key. - `doc/execution-semantics.md`: one additive paragraph. - Tests in `native-session-resume.test.ts` and `native-session-executor.test.ts`. Guard conditions. The mark is written only when all of these are true: - The close checkpoint threw before the durable backup was stored (after the upload's own bounded retries). A later local publication, stamp or cleanup failure does not mark the run. - The target is a remote `sandbox` with `reusableLeaseConfigured !== true`. A retained sandbox or an SSH host keeps the live state, so its resume path is unchanged. - The run holds an accepted result. A failure before acceptance changes nothing. Saved state from the previous release: run rows without the key keep today's resume path. An older server that reads a row with the key ignores it, and so it behaves as it does today. No schema change. User-visible: the next run's transcript shows "Paperclip session started" instead of "resumed", with a new session id. The source run's log shows `provider state was not saved after the accepted result; the next run starts a fresh provider session`. The fresh session gets the full task context, as for other fresh starts. ## Verification - `cd server && npx vitest run src/services/native-runtime/native-session-resume.test.ts -t "not resumable only after|provider state was not saved"`: 9 pass (6 pure, 3 embedded Postgres). - `cd server && npx vitest run src/services/native-runtime/native-session-executor.test.ts -t "marks the provider state unsaved|uses the bundled Linux image authority for remote opencode-new-durable"`: 8 pass. They include a durable store failure (marked) and a local publication failure after the store (not marked). - Red on `master` (source files reverted, tests kept): the 3 "starts the next run in a fresh provider session directly" cases fail (`expected {…} to be null`), the DB "accepted" case fails the same way, and the executor per-turn case fails (`mark` called 0 times). - Guard removal checks: - Sandbox guard replaced with `true`: the "retained sandbox" case fails (`mark` called once). - `!checkpointBackupStored` removed: the "publish-failed" case fails. - `schema_status = 'accepted'` removed from the marker: the "rejected" case fails (`expected true to be false`). - Negative cases that pass: no result, rejected result, retained sandbox, records without the key or with another code (all three recovery policies), and a later save that clears the mark. - `cd server && npx tsc --noEmit -p .`: clean. - Not related to this change and failing on `master` in this worktree too: `runtime-selection.test.ts` connector skill cases (missing `.claude/skills/*/SKILL.md`) and `native-session-resume.test.ts` "archives a damaged prior epoch" (`ENOENT … authority-epochs`). Required CI is the authority for those suites. ## Risks - Low. The change only skips a resume that cannot succeed. The fresh session uses the same workspace and full task context, like a Retry after `native_provider_state_incompatible`. - If the backup failed but the provider state were still somewhere recoverable, the next run would lose provider context. For a per-turn sandbox without a reusable lease, the restore path uses only the backup, so this does not apply. Retained sandboxes and SSH hosts are excluded. - One extra conditional `UPDATE` runs only on the failure path. The normal path before `turn.start` has no new waits or work. - `doc/execution-semantics.md` is also edited by #20809. The paragraphs are at different places; whichever merges second rebases. > For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and discuss it in `#dev` before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See `CONTRIBUTING.md`. ## Model Used - Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code, with tool use (shell, file edit, test execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>
217 lines
13 KiB
TypeScript
217 lines
13 KiB
TypeScript
import { randomUUID } from "node:crypto";
|
|
import { and, eq, sql } from "drizzle-orm";
|
|
import { agents, companies, createDb, heartbeatRunEvents, heartbeatRuns, issues, type Db } from "@paperclipai/db";
|
|
import { afterAll, afterEach, beforeAll, describe, expect, it } from "vitest";
|
|
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "../../__tests__/helpers/embedded-postgres.js";
|
|
import { resolveExternalTestDatabaseUrl } from "../../__tests__/helpers/external-test-database.js";
|
|
import { findNativeSessionResumeRun } from "./native-session-resume.js";
|
|
import { resolveNativeControllerRecovery } from "./native-controller-recovery.js";
|
|
|
|
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_TEST_DATABASE_URL");
|
|
const supported = Boolean(externalUrl) || (await getEmbeddedPostgresTestSupport()).supported;
|
|
|
|
(supported ? describe : describe.skip)("native session resume SQL projection", () => {
|
|
let database: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>>;
|
|
let db: Db;
|
|
const companyId = randomUUID(), agentId = randomUUID(), issueId = randomUUID(), normalizedSessionId = randomUUID();
|
|
const currentRunId = randomUUID();
|
|
const at = (second: number) => new Date(Date.UTC(2026, 8, 8, 0, 0, second));
|
|
const input = { companyId, agentId, issueId, normalizedSessionId, currentRunId, beforeCreatedAt: at(30) };
|
|
const checkpoint = { sessionId: "provider-session", identity: { runId: "prior" }, complete: [false, 0, "", null] };
|
|
const nativeExecutionInput = { binding: { issueId }, task: { prompt: "Complete native input", description: "Description" }, runtimeContext: { nested: [1, 2] } };
|
|
const paperclipWake = { externalChatProvider: "slack", githubReviewConfigurations: [{ key: "configuration", nested: { retained: true } }], other: ["preserved"] };
|
|
const paperclipIssue = { id: issueId, title: "Saved canonical title", description: "Saved canonical description" };
|
|
const profile = { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", unrelated: "Omit" };
|
|
const run = (second: number, overrides: Partial<typeof heartbeatRuns.$inferInsert> = {}) => ({
|
|
id: randomUUID(), companyId, agentId, nativeIssueId: issueId, nativeSessionId: normalizedSessionId,
|
|
runtimeMode: "native", status: "failed", createdAt: at(second), runnerProfileJson: {}, ...overrides,
|
|
});
|
|
const select = () => findNativeSessionResumeRun(db, input);
|
|
const source = () => run(0, { runnerInstanceId: randomUUID(), processPid: 17, processGroupId: 19, processStartedAt: at(0),
|
|
errorCode: "native_session_interrupted",
|
|
runnerProfileJson: profile, contextSnapshot: { paperclipWake, paperclipIssue: { ...paperclipIssue, unrelated: "Omit" }, prompt: "Omit" } });
|
|
const writeProfile = (id: string, value: unknown) => db.execute(sql`update ${heartbeatRuns}
|
|
set runner_profile_json = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${id}`);
|
|
|
|
beforeAll(async () => {
|
|
database = externalUrl ? { connectionString: externalUrl, cleanup: async () => {} } : await startEmbeddedPostgresTestDatabase("native-resume-projection-");
|
|
db = createDb(database.connectionString);
|
|
await db.insert(companies).values({ id: companyId, name: "Resume projection", issuePrefix: `N${companyId.replaceAll("-", "")}` });
|
|
await db.insert(agents).values({ id: agentId, companyId, name: "Runner" });
|
|
await db.insert(issues).values({ id: issueId, companyId, title: "Resume task" });
|
|
});
|
|
afterEach(async () => {
|
|
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.companyId, companyId));
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, companyId));
|
|
});
|
|
afterAll(async () => {
|
|
if (!db) return;
|
|
try {
|
|
await db.delete(issues).where(eq(issues.companyId, companyId));
|
|
await db.delete(agents).where(eq(agents.companyId, companyId));
|
|
await db.delete(companies).where(eq(companies.id, companyId));
|
|
} finally {
|
|
try { await db.$client.end({ timeout: 0 }); }
|
|
finally { await database.cleanup(); }
|
|
}
|
|
});
|
|
|
|
it("retains full required input, checkpoint, wake, canonical brief and process identity only for the selected source", async () => {
|
|
const saved = source();
|
|
await db.insert(heartbeatRuns).values([saved, run(1), run(2)]);
|
|
const result = await select();
|
|
expect(result).toEqual({ ...saved, runnerProfileJson: { nativeExecutionInput, sessionCheckpoint: checkpoint, nativeToolContractFingerprint: "fingerprint", nativeControllerRecovery: null,
|
|
nativeProviderStateUnsaved: null },
|
|
contextSnapshot: { paperclipWake, paperclipIssue } });
|
|
const stored = await db.select({ profile: heartbeatRuns.runnerProfileJson, context: heartbeatRuns.contextSnapshot }).from(heartbeatRuns).where(eq(heartbeatRuns.id, saved.id));
|
|
expect(stored[0].profile).toEqual(profile);
|
|
expect(stored[0].context).toEqual(saved.contextSnapshot);
|
|
for (const status of ["succeeded", "interrupted", "cancelled", "timed_out"]) {
|
|
await db.update(heartbeatRuns).set({ status }).where(eq(heartbeatRuns.id, saved.id));
|
|
expect((await select())?.id).toBe(saved.id);
|
|
}
|
|
});
|
|
|
|
it("treats every non-null checkpoint as a barrier, including invalid false, zero, empty string, array and object values", async () => {
|
|
const saved = source(), newer = run(1);
|
|
await db.insert(heartbeatRuns).values([saved, newer]);
|
|
for (const value of [false, 0, "", "invalid", [], {}]) {
|
|
await writeProfile(newer.id, { sessionCheckpoint: value });
|
|
const selected = await select();
|
|
expect(selected?.id).toBe(newer.id);
|
|
expect(selected?.runnerProfileJson.sessionCheckpoint).toEqual(value);
|
|
}
|
|
});
|
|
|
|
it("does not call missing or newer incompatible recovery material a fresh session", async () => {
|
|
const saved = source();
|
|
await db.insert(heartbeatRuns).values([saved, run(30, { id: currentRunId, status: "running" })]);
|
|
const { beforeCreatedAt: _before, ...binding } = input;
|
|
expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "snapshot_binding" });
|
|
// The newest progress barrier remains authoritative even without a snapshot.
|
|
await db.insert(heartbeatRuns).values(run(10, { runtimeMode: "legacy" }));
|
|
expect(await resolveNativeControllerRecovery(db, binding)).toEqual({ kind: "missing_evidence", reason: "newer_progress_barrier" });
|
|
});
|
|
|
|
it("skips only unused SQL-null, JSON-null, scalar, array and checkpoint-null profiles", async () => {
|
|
const saved = source(), newer = run(1, { runnerProfileJson: null });
|
|
await db.insert(heartbeatRuns).values([saved, newer]);
|
|
expect((await select())?.id).toBe(saved.id);
|
|
for (const value of [null, false, 0, "sessionCheckpoint", [], ["sessionCheckpoint"], {}, { sessionCheckpoint: null }]) {
|
|
await writeProfile(newer.id, value);
|
|
expect((await select())?.id).toBe(saved.id);
|
|
}
|
|
});
|
|
|
|
it("preserves malformed selected context values and missing brief fields", async () => {
|
|
const saved = source();
|
|
await db.insert(heartbeatRuns).values(saved);
|
|
for (const value of [null, false, 0, "invalid", [], {}, { paperclipWake: [], paperclipIssue: { id: issueId, title: false } },
|
|
{ paperclipIssue: { id: issueId, title: "Valid title without description" } },
|
|
{ paperclipIssue: { id: issueId, title: "Valid title", description: null } }]) {
|
|
await db.execute(sql`update ${heartbeatRuns} set context_snapshot = ${JSON.stringify(value)}::jsonb where ${heartbeatRuns.id} = ${saved.id}`);
|
|
const result = await select();
|
|
const object = typeof value === "object" && value !== null && !Array.isArray(value) ? value : {};
|
|
expect(result?.contextSnapshot.paperclipWake).toEqual("paperclipWake" in object ? object.paperclipWake : null);
|
|
expect(result?.contextSnapshot.paperclipIssue).toEqual("paperclipIssue" in object ? object.paperclipIssue : null);
|
|
}
|
|
});
|
|
|
|
it("stops at wrong runtime, nonterminal status and each process authority field", async () => {
|
|
await db.insert(heartbeatRuns).values(source());
|
|
for (const override of [{ runtimeMode: "legacy" }, { status: "running" }, { processPid: 0 }, { processGroupId: 0 }, { processStartedAt: at(0) }]) {
|
|
const newer = run(1, override);
|
|
await db.insert(heartbeatRuns).values(newer);
|
|
expect(await select()).toBeNull();
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, newer.id));
|
|
}
|
|
});
|
|
|
|
it("checks every provider evidence event and ignores unrelated event kinds", async () => {
|
|
const saved = source(), newer = run(1);
|
|
await db.insert(heartbeatRuns).values([saved, newer]);
|
|
for (const eventType of ["harness.ready", "session.started", "session.resumed", "session.updated", "turn.started", "provider.event", "provider.rpc_result", "log"]) {
|
|
await db.insert(heartbeatRunEvents).values({ companyId, agentId, runId: newer.id, seq: 1, eventType });
|
|
if (eventType === "log") expect((await select())?.id).toBe(saved.id);
|
|
else expect(await select()).toBeNull();
|
|
await db.delete(heartbeatRunEvents).where(eq(heartbeatRunEvents.runId, newer.id));
|
|
}
|
|
});
|
|
|
|
it("excludes current and mismatched agent, company, task and session rows before applying barriers", async () => {
|
|
const saved = source(), otherCompany = randomUUID(), otherAgent = randomUUID(), otherIssue = randomUUID();
|
|
await db.insert(companies).values({ id: otherCompany, name: "Other", issuePrefix: `D${otherCompany.replaceAll("-", "")}` });
|
|
await db.insert(agents).values({ id: otherAgent, companyId: otherCompany, name: "Other" });
|
|
await db.insert(issues).values({ id: otherIssue, companyId, title: "Other" });
|
|
try {
|
|
await db.insert(heartbeatRuns).values([saved, run(1, { id: currentRunId, status: "running" }),
|
|
run(2, { companyId: otherCompany, status: "running" }), run(3, { agentId: otherAgent, status: "running" }),
|
|
run(4, { nativeIssueId: otherIssue, status: "running" }), run(5, { nativeSessionId: randomUUID(), status: "running" })]);
|
|
expect((await select())?.id).toBe(saved.id);
|
|
} finally {
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.companyId, otherCompany));
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.agentId, otherAgent));
|
|
await db.delete(issues).where(eq(issues.id, otherIssue));
|
|
await db.delete(agents).where(eq(agents.id, otherAgent));
|
|
await db.delete(companies).where(eq(companies.id, otherCompany));
|
|
}
|
|
expect(await findNativeSessionResumeRun(db, { ...input, normalizedSessionId: "invalid" })).toBeNull();
|
|
for (const key of ["companyId", "agentId", "issueId", "normalizedSessionId"] as const) {
|
|
expect(await findNativeSessionResumeRun(db, { ...input, [key]: input[key].toUpperCase() })).toBeNull();
|
|
}
|
|
});
|
|
|
|
it("retains the 32-row bound and deterministic created-at/id ordering", async () => {
|
|
const saved = source(), unused = Array.from({ length: 32 }, () => run(1));
|
|
await db.insert(heartbeatRuns).values([saved, ...unused]);
|
|
expect(await select()).toBeNull();
|
|
await db.delete(heartbeatRuns).where(eq(heartbeatRuns.id, unused[0].id));
|
|
expect((await select())?.id).toBe(saved.id);
|
|
const tied = [run(2, { runnerProfileJson: { sessionCheckpoint: false } }), run(2, { status: "running" })].sort((a, b) => b.id.localeCompare(a.id));
|
|
await db.insert(heartbeatRuns).values(tied);
|
|
if (tied[0].status === "running") expect(await select()).toBeNull();
|
|
else expect((await select())?.id).toBe(tied[0].id);
|
|
});
|
|
|
|
it("does not skip future checkpoint barriers and preserves Date millisecond comparison", async () => {
|
|
const saved = source(), future = run(31);
|
|
await db.insert(heartbeatRuns).values([saved, future]);
|
|
expect((await select())?.id).toBe(saved.id);
|
|
await writeProfile(future.id, { sessionCheckpoint: false });
|
|
expect(await select()).toBeNull();
|
|
await db.execute(sql`update ${heartbeatRuns} set created_at = '2026-09-08T00:00:30.000999Z'::timestamptz where ${heartbeatRuns.id} = ${future.id}`);
|
|
expect((await select())?.id).toBe(future.id);
|
|
});
|
|
|
|
it("sees a newly committed barrier and changed source binding in its single selection snapshot", async () => {
|
|
const saved = source();
|
|
await db.insert(heartbeatRuns).values(saved);
|
|
let reads = 0;
|
|
const changing = new Proxy(db, { get(target, key) {
|
|
if (key !== "with") return Reflect.get(target, key);
|
|
return (...ctes: Parameters<Db["with"]>) => {
|
|
const builder = target.with(...ctes), original = builder.select.bind(builder);
|
|
builder.select = ((...fields: Parameters<typeof original>) => {
|
|
const selection = original(...fields), from = selection.from.bind(selection);
|
|
selection.from = ((...args: Parameters<typeof from>) => {
|
|
const query = from(...args), execute = query.execute.bind(query);
|
|
query.execute = async (...executeArgs: Parameters<typeof execute>) => {
|
|
reads++;
|
|
await db.insert(heartbeatRuns).values(run(1, { runnerProfileJson: { sessionCheckpoint: 0 } }));
|
|
await db.update(heartbeatRuns).set({ nativeSessionId: randomUUID() }).where(and(eq(heartbeatRuns.id, saved.id), eq(heartbeatRuns.companyId, companyId)));
|
|
return execute(...executeArgs);
|
|
};
|
|
return query;
|
|
}) as typeof selection.from;
|
|
return selection;
|
|
}) as typeof builder.select;
|
|
return builder;
|
|
};
|
|
}});
|
|
const selected = await findNativeSessionResumeRun(changing, input);
|
|
expect(reads).toBe(1);
|
|
expect(selected?.id).not.toBe(saved.id);
|
|
expect(selected?.runnerProfileJson.sessionCheckpoint).toBe(0);
|
|
});
|
|
});
|