mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-20 09:50:48 +01:00
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents that use OpenCode in a remote sandbox must have a durable,
verified copy of the provider launch files before their first turn (the
rule from #19622)
> - Today each new run copies these files out of the sandbox, uploads
them and verifies them again, even when the company already has a ready
artifact with the same files
> - The whole copy runs under one five-minute clock. When eight runs
start together on small sandboxes, the copies compete, and slow copies
fail the run before its first turn
> - This makes cold starts slow (p50 273 s from run start to turn
submitted) and makes half of the first runs fail in a burst
> - This pull request reuses a ready artifact with the same digests,
shares one in-flight copy per company and manifest, and bounds each
network step instead of the whole copy
> - The benefit is that first turns start in about one minute on a cold
burst and in about 20 s when the artifact is already ready, and a slow
copy no longer fails the run
## Linked Issues or Issue Description
Refs #19622
**What happened?**
When 8 OpenCode runs started together on 1 vCPU Daytona sandboxes, 4 of
8 first runs failed before their first turn. Each run copied, uploaded
and verified the same provider launch files. The single 300 s clock over
the whole copy expired. Run start to turn submitted was p50 273 s and
p95 314 s.
**Expected behavior**
A run starts its first turn without a new copy when a ready, verified
artifact with the same files exists. A slow copy does not fail the run.
**Steps to reproduce**
1. Use a company with remote OpenCode agents on Daytona sandboxes with
durable storage.
2. Start 8 runs at the same time on new sandboxes.
3. See the time before the first turn and the failed runs.
**Agent adapter(s) involved**
OpenCode (native runtime, remote sandbox).
## What Changed
- `retainReadyNativeProviderArtifacts` (`native-provider-artifacts.ts`):
reuse a ready, verified company artifact with the same paths and SHA-256
digests through a lookup only. No file is copied, uploaded or verified
again. The lookup (`readyNativeProviderArtifactCandidates`) matches
target, paths and digests in Postgres with jsonb containment on the
company's ready rows (existing partial index), newest first, limit 4. It
does not load or hash other retained versions. No migration.
- `shareNativeProviderArtifactCapture`
(`native-provider-artifact-capture.ts`): one in-flight capture per
company and manifest digest per server. Other runs wait for it, then
reuse its result.
- The single 300 s clock over the whole capture is replaced by a bound
for each network step (upload, verifying download): max(120 s, size at
256 KiB/s). A timeout has the typed cause
`native_provider_artifacts_transfer_timeout`. The capture makes up to 3
attempts from the local copy.
- Stop now interrupts the capture and the wait for a shared capture
(`ActiveNativeSession.admissionAbort`).
- `doc/opencode-cold-recovery.md` describes the new behavior.
- Tests: new `native-provider-artifact-gate.test.ts` (7 tests), 5 new
tests in `native-harness-backup-storage.integration.test.ts`, and new
executor variants `opencode-new-durable-slow-capture` and
`opencode-new-durable-ready` in `native-session-executor.test.ts`.
## Verification
Local (this branch, on master 9b613562b7):
- `npx vitest run
src/services/native-runtime/native-provider-artifact-gate.test.ts
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
3 files, 1511 passed, 7 skipped.
- `npx tsc --noEmit -p server`: pass.
- Lookup scale: the integration test "loads only the ready artifact
whose launch files match, however many other versions the company keeps"
adds 200 ready non-matching versions. It returns only the matching
artifact. Without the SQL filter it fails (4 rows returned).
- Red/green: with master's `native-session-executor.ts` and the new
tests, `opencode-new-durable-slow-capture` fails with `TimeoutError: The
operation was aborted due to timeout`, and `opencode-new-durable-ready`
fails because the ready artifact is not looked up. With this change,
both pass.
Daytona (same commit content, cherry-picked onto the baseline commit
84a1ee3e11 with #21262; image `sha256:9461cc75…`; 1 vCPU sandboxes;
OpenCode):
| Wave | Shape | First runs OK | Run start → turn submitted | Other |
|---|---|---|---|---|
| Baseline `w-20261113T011027-s1702` | 8 runs started together | 4/8 (4
failed) | p50 273 s / p95 314 s | "only a getter" TypeError ×8,
`bootstrap_not_proven_incomplete` ×8 |
| Fix `w-20261113T031045-s1702` | same seed and shape | 8/8 | p50 66.0 s
/ p95 85.7 s | 8/8 tasks hands-off, 0 Retry banners; first agent event
p50 72.9 s / p95 90.0 s; exactly 1 upload, 7 runs waited on it (~65 s
shared capture); all failure codes 0 |
| Fix `w-20261113T034033-s1702` | 16 tasks, cap 8 | 16/16 | p50 19.1 s /
p95 24.3 s | second batch (artifact already ready, cold sandboxes) gate
time 24, 25, 26, 27, 31, 31, 45, 119 ms; first agent event p50 25.9 s |
## Risks
- The rule from #19622 still holds: a durable, verified artifact exists
before the first turn. Reuse only accepts a ready, verified artifact of
the same company with the same paths and digests.
- The first run per company and manifest per server still pays the full
transfer.
- Two servers can still upload duplicate artifacts for the same company
and manifest. Sharing is per server.
- No test drives Stop through the executor during a capture. Signal
handling is unit-tested only.
- A capture that fails all 3 attempts still ends blocked with a working
Retry. A follow-up PR (pre-turn failures continue automatically) removes
that.
- The documented five-minute deadline per capture is replaced by bounds
per network step. A very large artifact on a slow link can take longer
than five minutes in total, but each step still has a bound.
> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.
## Model Used
- Claude (Anthropic), model ID `claude-opus-5-5`, used through Claude
Code with tool use (shell, file edit, test runs). Default context
window; extended reasoning on.
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1065 lines
88 KiB
TypeScript
1065 lines
88 KiB
TypeScript
import fs from "node:fs/promises";
|
|
import { createHash, randomUUID } from "node:crypto";
|
|
import { chmod, mkdir, readFile, rename, stat, symlink, readlink, writeFile } from "node:fs/promises";
|
|
import { join } from "node:path";
|
|
import { and, eq, inArray } from "drizzle-orm";
|
|
import { agentTaskSessions, environmentLeases, heartbeatRuns, issues, nativeHarnessBackups, nativeRunFinalizations, nativeRunResults,
|
|
nativeProviderArtifacts, nativeProviderArtifactReferences, nativeRunnerCheckpoints } from "@paperclipai/db";
|
|
import { afterEach, expect, it, vi } from "vitest";
|
|
import { describeEmbeddedPostgres, nativeRestartFixture } from "../../__tests__/helpers/native-restart.js";
|
|
import { createLocalDiskStorageProvider } from "../../storage/local-disk-provider.js";
|
|
import { createStorageService } from "../../storage/service.js";
|
|
import { nativeHarnessBackupOwner, persistNativeHarnessBackup, restoreDurableNativeHarnessBackup,
|
|
verifyDurableNativeHarnessBackup, deleteRetiredNativeHarnessBackup } from "./native-harness-backup-storage.js";
|
|
import { removeNativeHarnessBackup } from "./native-harness-backup-cleanup.js";
|
|
import { buildNativeHarnessBackupManifest } from "./native-session-executor.js";
|
|
import { createNativeHarnessBackupStamp, verifyNativeHarnessBackupStamp, verifyNativeHarnessBackupStampAsync } from "./native-harness-backup-stamp.js";
|
|
import { hasLocalNativeHarnessObligations, migrateNativeHarnessBackups, prestageNativeHarnessBackups } from "./native-harness-backup-migration.js";
|
|
import { nativeHarnessBackupExpirationService } from "./native-harness-backup-expiration.js";
|
|
import { legacyControllerBootId } from "../legacy-controller-lease.js";
|
|
import { DurablePrpControlPlane, captureLegacyDurablePrpCheckpoint } from "../../vendor/paperclip-runner/index.js";
|
|
import { loadNativeRunnerCheckpoint, nativeRunnerCheckpointWriter } from "./native-runner-checkpoint.js";
|
|
import { NativeProviderArtifactTransferTimeoutError, nativeProviderArtifactTransferBounds, restoreNativeProviderArtifacts,
|
|
readyNativeProviderArtifactCandidates, retainNativeProviderArtifacts, retainReadyNativeProviderArtifacts } from "./native-provider-artifacts.js";
|
|
import * as backupArchive from "./native-harness-backup-archive.js";
|
|
import { canonicalProviderPackJson } from "./remote-provider-pack-verification.js";
|
|
|
|
vi.mock("node:fs/promises", async importOriginal => {
|
|
const original = await importOriginal<typeof import("node:fs/promises")>();
|
|
const files = { ...original };
|
|
return { ...original, default: files, cp: (...args: Parameters<typeof original.cp>) => files.cp(...args) };
|
|
});
|
|
|
|
describeEmbeddedPostgres("durable native provider backups", () => {
|
|
const environment = nativeRestartFixture();
|
|
const ownedHomes: string[] = [];
|
|
function deferred() {
|
|
let resolve!: () => void;
|
|
const promise = new Promise<void>(done => { resolve = done; });
|
|
return { promise, resolve };
|
|
}
|
|
afterEach(async () => {
|
|
for (const home of ownedHomes.splice(0)) await removeNativeHarnessBackup(home);
|
|
vi.unstubAllEnvs();
|
|
});
|
|
async function fixture(provider = "codex") {
|
|
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "51".repeat(32));
|
|
const f = await environment.seedRun(randomUUID());
|
|
const { companyId, runtimeRoot, successor } = environment.current;
|
|
await f.db.update(nativeRunFinalizations).set({ controllerBootId: successor.bootId, controllerGeneration: 1,
|
|
leaseOwner: "backup-writer", leaseExpiresAt: new Date(Date.now() + 120_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId, issueId: f.issueId, runId: f.runId, bootId: successor.bootId });
|
|
const home = join(runtimeRoot, randomUUID()), root = join(home, "failover-backups", "current");
|
|
ownedHomes.push(home);
|
|
await mkdir(join(root, provider, "sessions"), { recursive: true });
|
|
await mkdir(join(root, "runner"));
|
|
const identity = { normalizedSessionId: randomUUID(), providerSessionId: randomUUID(), provider };
|
|
await writeFile(join(root, provider, "sessions", "history.json"), JSON.stringify(identity), { mode: 0o600 });
|
|
await writeFile(join(root, "runner", "runner-state.json"), JSON.stringify({ runId: f.runId, lifecycle: "suspended" }), { mode: 0o600 });
|
|
const manifest = JSON.stringify({ schema: "paperclip.native-harness-backup.v1", ...identity, directories: [provider, "runner"] });
|
|
await writeFile(join(root, "manifest.json"), manifest, { mode: 0o600 });
|
|
await chmod(join(root, provider, "sessions"), 0o500);
|
|
const sessionScopeId = randomUUID(), scopeDigest = createHash("sha256").update(sessionScopeId).digest("hex");
|
|
const storage = createStorageService(createLocalDiskStorageProvider(join(runtimeRoot, "durable-objects")));
|
|
return { ...f, home, root, companyId, owner, scopeDigest, sessionScopeId, storage, identity,
|
|
manifestSha256: `sha256:${createHash("sha256").update(manifest).digest("hex")}` };
|
|
}
|
|
async function offlineFixture(base: string) {
|
|
const f = await fixture("codex-home"), home = join(base, f.scopeDigest);
|
|
await rename(f.home, home);
|
|
const root = join(home, "failover-backups", "current"), runnerInstanceId = randomUUID(), providerLeaseId = randomUUID();
|
|
const execution = { provider: { kind: "codex" }, binding: { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, agentId: environment.current.agentId, executionWorkspaceId: randomUUID() },
|
|
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: f.identity.normalizedSessionId, driverKind: "codex_app_server",
|
|
protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as never;
|
|
const manifest = buildNativeHarnessBackupManifest({ backupRoot: root, execution, runnerInstanceId,
|
|
providerSessionIdentity: { providerSessionId: f.identity.providerSessionId }, sourceProviderLeaseId: providerLeaseId });
|
|
await writeFile(join(root, "manifest.json"), JSON.stringify(manifest));
|
|
const stamp = createNativeHarnessBackupStamp({ manifestPath: join(root, "manifest.json"), sessionScopeId: f.sessionScopeId,
|
|
authorizedProviderLeaseId: providerLeaseId, runnerInstanceId, normalizedSessionId: f.identity.normalizedSessionId, completedAt: manifest.completedAt });
|
|
await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, runnerInstanceId }).where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
|
|
providerLeaseId, status: "expired", metadata: { nativeHarnessBackup: stamp } });
|
|
return { ...f, home, root, runnerInstanceId, providerLeaseId };
|
|
}
|
|
async function offlineFixtures(count: number) {
|
|
const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base);
|
|
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base);
|
|
const fixtures = [];
|
|
for (let i = 0; i < count; i++) fixtures.push(await offlineFixture(base));
|
|
const runs = await fixtures[0].db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.companyId, fixtures[0].companyId));
|
|
for (const run of runs) await settle({ ...fixtures[0], runId: run.id, issueId: run.issueId! }, new Date());
|
|
return fixtures.sort((a, b) => a.scopeDigest.localeCompare(b.scopeDigest));
|
|
}
|
|
it("overlaps verified offline backups within a four-transfer bound and restores their bytes", async () => {
|
|
const fixtures = await offlineFixtures(6), f = fixtures[0], gate = deferred();
|
|
const put = f.storage.putFile.bind(f.storage); let active = 0, peak = 0, entered = 0;
|
|
vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
active++; entered++; peak = Math.max(peak, active);
|
|
try { await gate.promise; return await put(input); } finally { active--; }
|
|
});
|
|
const migration = migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) });
|
|
try { await vi.waitFor(() => expect(active).toBe(4), { timeout: 3000 }); }
|
|
finally { gate.resolve(); await migration.catch(() => undefined); }
|
|
expect(await migration).toBe(6); expect(peak).toBe(4); expect(entered).toBe(6); expect(active).toBe(0);
|
|
for (const source of fixtures) {
|
|
await removeNativeHarnessBackup(source.home);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...source, storage: f.storage, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", "codex-home", "sessions", "history.json"), "utf8"))).toEqual(source.identity);
|
|
}
|
|
});
|
|
it("waits for cancelled peer transfers before returning failure and permits a complete retry", async () => {
|
|
const fixtures = await offlineFixtures(2), f = fixtures[0], peerEntered = deferred(), release = deferred();
|
|
const failed = deferred(), put = f.storage.putFile.bind(f.storage); let entered = 0, active = 0, settled = false;
|
|
const spy = vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const ordinal = ++entered; active++;
|
|
try {
|
|
if (ordinal === 1) { await peerEntered.promise; failed.resolve(); throw new Error("storage unavailable"); }
|
|
peerEntered.resolve(); await release.promise; return await put(input);
|
|
} finally { active--; }
|
|
});
|
|
const migration = migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) });
|
|
const completed = migration.then(() => { settled = true; }, () => { settled = true; });
|
|
try {
|
|
await vi.waitFor(() => expect(entered).toBe(2), { timeout: 3000 }); await failed.promise;
|
|
await new Promise<void>(resolve => setImmediate(resolve)); expect(settled).toBe(false);
|
|
} finally { peerEntered.resolve(); release.resolve(); await completed; }
|
|
await expect(migration).rejects.toThrow("storage unavailable"); expect(active).toBe(0); spy.mockRestore();
|
|
for (const source of fixtures) expect(await readFile(join(source.root, "manifest.json"), "utf8")).toContain("paperclip.native-harness-backup.v1");
|
|
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) })).toBe(2);
|
|
for (const source of fixtures) {
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, source.scopeDigest));
|
|
expect(rows.filter(row => row.status === "current")).toHaveLength(1);
|
|
}
|
|
});
|
|
it("preserves unlisted files with an exclusive scratch reservation", async () => {
|
|
const fixtures = await offlineFixtures(3), f = fixtures[0], extra = fixtures[1];
|
|
await writeFile(join(extra.root, "retained-extra.txt"), "unlisted surviving bytes");
|
|
const put = f.storage.putFile.bind(f.storage); let active = 0, peak = 0;
|
|
vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
active++; peak = Math.max(peak, active);
|
|
try { await new Promise<void>(resolve => setImmediate(resolve)); return await put(input); } finally { active--; }
|
|
});
|
|
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) })).toBe(3);
|
|
expect(peak).toBe(1);
|
|
await removeNativeHarnessBackup(extra.home);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...extra, storage: f.storage, backupRoot: replacement })).toBe(true);
|
|
expect(await readFile(join(replacement, "current", "retained-extra.txt"), "utf8")).toBe("unlisted surviving bytes");
|
|
});
|
|
it("enforces the reserved archive size while streaming before publishing any object", async () => {
|
|
const f = await fixture(), put = vi.spyOn(f.storage, "putFile");
|
|
await expect(persistNativeHarnessBackup({ ...f, maxArchiveBytes: 512 })).rejects.toThrow("native_harness_archive_invalid");
|
|
expect(put).not.toHaveBeenCalled();
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0);
|
|
expect(JSON.parse(await readFile(join(f.root, "codex", "sessions", "history.json"), "utf8"))).toEqual(f.identity);
|
|
});
|
|
it.each(["codex", "claude", "opencode", "grok", "new-provider"])("restores %s session bytes after deleting the entire controller home", async provider => {
|
|
const f = await fixture(provider);
|
|
await persistNativeHarnessBackup(f);
|
|
const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(row.status).toBe("current");
|
|
const object = await f.storage.getObject(f.companyId, row.reference.objectKey as string);
|
|
const parts: Buffer[] = []; for await (const part of object.stream) parts.push(part);
|
|
expect(Buffer.concat(parts).includes(Buffer.from(f.identity.providerSessionId))).toBe(false);
|
|
await removeNativeHarnessBackup(f.home);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups");
|
|
ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", provider, "sessions", "history.json"), "utf8"))).toEqual(f.identity);
|
|
expect((await stat(join(replacement, "current", provider, "sessions"))).mode & 0o777).toBe(0o500);
|
|
expect(await verifyDurableNativeHarnessBackup(f)).toBe(true);
|
|
});
|
|
async function artifactFixture(artifactVersion = randomUUID()) {
|
|
const f = await fixture("opencode");
|
|
const sourceRoot = join(f.home, "qualified-provider-pack");
|
|
await mkdir(sourceRoot);
|
|
const original = { nodeCommand: "qualified Node executable", opencodeExecutable: "qualified OpenCode executable",
|
|
opencodeProxy: `qualified original proxy bytes ${artifactVersion}` };
|
|
const artifacts = Object.fromEntries(await Promise.all(Object.entries(original).map(async ([name, bytes]) => {
|
|
await writeFile(join(sourceRoot, name), bytes, { mode: 0o500 });
|
|
return [name, { path: name, sha256: `sha256:${createHash("sha256").update(bytes).digest("hex")}` }];
|
|
})));
|
|
const payload = { runnerSourceRevision: "a".repeat(40), target: { platform: "linux", architecture: "x64" },
|
|
pins: { opencode: "1.18.34" }, artifacts: { ...artifacts, opencodeCommand: artifacts.opencodeExecutable } };
|
|
const manifest = { schema: "paperclip-runner/remote-provider-pack/v1", payload,
|
|
digest: `sha256:${createHash("sha256").update(canonicalProviderPackJson(payload)).digest("hex")}` };
|
|
const launchRoot = `/home/daytona/.paperclip-runtime/paperclip-runner/provider-artifacts/${manifest.digest.slice(7)}`;
|
|
await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, driverKind: "opencode_server",
|
|
runnerProfileJson: { ...f.preparedProfile, nativeProviderPackAdmissionState: {
|
|
schema: "paperclip.native-provider-pack-admission-state.v1", state: "verified", admissionEpoch: randomUUID() },
|
|
nativeProviderPackAdmission: { schema: "paperclip.native-provider-pack-admission.v1", stage: "session_admitted",
|
|
authority: "core_verified_selection", companyId: f.companyId, agentId: environment.current.agentId,
|
|
runId: f.runId, sessionId: f.identity.normalizedSessionId, runnerInstanceId: "artifact-fixture-runner",
|
|
runnerEnvironmentLeaseId: "artifact-fixture-lease", providerPackDigest: manifest.digest,
|
|
opencodeProxySha256: artifacts.opencodeProxy.sha256, sourceRevision: payload.runnerSourceRevision } },
|
|
}).where(eq(heartbeatRuns.id, f.runId));
|
|
return { ...f, original, manifest, launchRoot, providerArtifacts: { sourceRoot, manifest, launchRoot, accountUid: 1000 } };
|
|
}
|
|
it("restores the admitted OpenCode launch bytes after the controller and sandbox image change", async () => {
|
|
const publication = await artifactFixture(), f = publication;
|
|
const { original, manifest, launchRoot } = publication;
|
|
await persistNativeHarnessBackup(publication);
|
|
await removeNativeHarnessBackup(f.home);
|
|
const newImage = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(newImage);
|
|
await mkdir(newImage);
|
|
await writeFile(join(newImage, "opencodeProxy"), "qualified newer proxy bytes");
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups"); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current/opencode/sessions/history.json"), "utf8"))).toEqual(f.identity);
|
|
for (const [name, bytes] of Object.entries(original)) {
|
|
expect(await readFile(join(replacement, "current/provider-artifacts", name), "utf8")).toBe(bytes);
|
|
}
|
|
const restored = JSON.parse(await readFile(join(replacement, "current/manifest.json"), "utf8"));
|
|
expect(restored.providerArtifacts).toMatchObject({ launchRoot, providerPackDigest: manifest.digest });
|
|
});
|
|
it("repairs a missing local launch cache without adding the binaries to each harness backup", async () => {
|
|
const f = await artifactFixture(); await persistNativeHarnessBackup(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
await removeNativeHarnessBackup(join(replacement, "current/provider-artifacts"));
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(await readFile(join(replacement, "current/provider-artifacts/opencodeProxy"), "utf8")).toBe(f.original.opencodeProxy);
|
|
await persistNativeHarnessBackup({ ...f, providerArtifacts: undefined, root: join(replacement, "current") });
|
|
const [after] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
|
|
eq(nativeHarnessBackups.status, "current")));
|
|
expect(after.reference.byteSize).toBe(before.reference.byteSize);
|
|
});
|
|
it("retries an interrupted artifact cache copy without publishing partial launch files", async () => {
|
|
const f = await artifactFixture(); await persistNativeHarnessBackup(f);
|
|
const receipt = JSON.parse(await readFile(join(f.root, "manifest.json"), "utf8")).providerArtifacts;
|
|
const home = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(home); await mkdir(home);
|
|
const targetRoot = join(home, "provider-artifacts"), copy = fs.cp.bind(fs);
|
|
const interrupted = vi.spyOn(fs, "cp").mockImplementation(async (source, destination, options) => {
|
|
if (String(destination).startsWith(targetRoot)) {
|
|
await mkdir(destination, { recursive: true });
|
|
await writeFile(join(String(destination), "opencodeProxy"), "partial copy");
|
|
throw Object.assign(new Error("controlled disk full during cache publication"), { code: "ENOSPC" });
|
|
}
|
|
return copy(source, destination, options);
|
|
});
|
|
try { await expect(restoreNativeProviderArtifacts({ ...f, receipt, targetRoot })).rejects.toThrow("controlled disk full"); }
|
|
finally { interrupted.mockRestore(); }
|
|
await expect(fs.stat(targetRoot)).rejects.toMatchObject({ code: "ENOENT" });
|
|
await expect(restoreNativeProviderArtifacts({ ...f, receipt, targetRoot })).resolves.toBeUndefined();
|
|
expect(await readFile(join(targetRoot, "opencodeProxy"), "utf8")).toBe(f.original.opencodeProxy);
|
|
expect(await fs.readdir(home)).toEqual(["provider-artifacts"]);
|
|
});
|
|
it("reuses a sequentially admitted closure and releases it only after the last stopped checkpoint retires", async () => {
|
|
const version = randomUUID(), first = await artifactFixture(version), other = await artifactFixture(version);
|
|
const second = { ...other, storage: first.storage };
|
|
const put = vi.spyOn(first.storage, "putFile");
|
|
await persistNativeHarnessBackup(first); await persistNativeHarnessBackup(second);
|
|
expect(put.mock.calls.filter(([input]) => input.namespace === "native-provider-artifacts")).toHaveLength(1);
|
|
const [firstBackup] = await first.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, first.scopeDigest));
|
|
const [secondBackup] = await first.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, second.scopeDigest));
|
|
const artifactId = (firstBackup.reference.providerArtifacts as { artifactId: string }).artifactId;
|
|
const [artifact] = await first.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, artifactId));
|
|
const stop = async (f: typeof first) => {
|
|
await settle(f, new Date(Date.now() - 10 * 86_400_000));
|
|
const leaseId = randomUUID(), providerLeaseId = randomUUID(), confirmedAt = new Date();
|
|
await f.db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, heartbeatRunId: f.runId,
|
|
issueId: f.issueId, provider: "daytona", providerLeaseId, status: "expired", cleanupStatus: "success", releasedAt: confirmedAt,
|
|
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
|
|
runId: f.runId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed", confirmedAt: confirmedAt.toISOString() } } });
|
|
};
|
|
await stop(first);
|
|
await first.db.update(nativeHarnessBackups).set({ status: "retired" }).where(eq(nativeHarnessBackups.id, firstBackup.id));
|
|
expect(await deleteRetiredNativeHarnessBackup(first.db, first.storage, firstBackup.id)).toBe(true);
|
|
expect(await first.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, artifact.id))).toHaveLength(1);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...second, backupRoot: replacement })).toBe(true);
|
|
await stop(second);
|
|
await second.db.update(nativeHarnessBackups).set({ status: "retired" }).where(eq(nativeHarnessBackups.id, secondBackup.id));
|
|
expect(await deleteRetiredNativeHarnessBackup(second.db, second.storage, secondBackup.id)).toBe(true);
|
|
expect(await first.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, artifact.id))).toHaveLength(1);
|
|
const runPins = await first.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.artifactId, artifact.id));
|
|
expect(runPins).toHaveLength(2); expect(runPins.every(pin => pin.backupId === null)).toBe(true);
|
|
await first.db.insert(nativeRunnerCheckpoints).values([first, second].map(f => ({ companyId: f.companyId,
|
|
issueId: f.issueId, runId: f.runId, part: "header", material: { retained: "canonical checkpoint expiry fixture" } })));
|
|
expect((await nativeHarnessBackupExpirationService(first.db, { storage: first.storage }).tickCheckpoints()).expiredCheckpoints).toBe(2);
|
|
expect(await first.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, artifact.id))).toHaveLength(0);
|
|
expect(await first.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.artifactId, artifact.id))).toHaveLength(0);
|
|
});
|
|
it("admits a later run from the company's ready launch artifact with a lookup and no transfer", async () => {
|
|
const version = randomUUID(), first = await artifactFixture(version), other = await artifactFixture(version);
|
|
const second = { ...other, storage: first.storage };
|
|
const stored = await retainNativeProviderArtifacts({ ...first, source: first.providerArtifacts });
|
|
const put = vi.spyOn(first.storage, "putFile"), get = vi.spyOn(first.storage, "getObject");
|
|
let receipt: Awaited<ReturnType<typeof retainReadyNativeProviderArtifacts>> = null;
|
|
try {
|
|
const startedAt = performance.now();
|
|
receipt = await retainReadyNativeProviderArtifacts({ db: second.db, owner: second.owner, manifest: second.manifest,
|
|
launchRoot: second.launchRoot, accountUid: 1000 });
|
|
const elapsedMs = performance.now() - startedAt;
|
|
console.info(`[provider-artifact-gate] ready artifact admission took ${elapsedMs.toFixed(1)} ms`);
|
|
expect(elapsedMs).toBeLessThan(1_000);
|
|
expect(receipt).toMatchObject({ artifactId: stored.artifactId, closureDigest: stored.closureDigest,
|
|
providerPackDigest: second.manifest.digest, launchRoot: second.launchRoot, accountUid: 1000 });
|
|
expect(put).not.toHaveBeenCalled();
|
|
expect(get).not.toHaveBeenCalled();
|
|
} finally { put.mockRestore(); get.mockRestore(); }
|
|
const [run] = await second.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, second.runId));
|
|
expect(run.runnerProfileJson?.nativeProviderArtifacts).toMatchObject({ artifactId: stored.artifactId });
|
|
const pins = await second.db.select().from(nativeProviderArtifactReferences).where(and(
|
|
eq(nativeProviderArtifactReferences.artifactId, stored.artifactId), eq(nativeProviderArtifactReferences.runId, second.runId)));
|
|
expect(pins).toHaveLength(1);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement); await mkdir(replacement);
|
|
await restoreNativeProviderArtifacts({ ...second, companyId: second.companyId, receipt: receipt!, targetRoot: join(replacement, "provider-artifacts") });
|
|
expect(await readFile(join(replacement, "provider-artifacts/opencodeProxy"), "utf8")).toBe(second.original.opencodeProxy);
|
|
});
|
|
it("loads only the ready artifact whose launch files match, however many other versions the company keeps", async () => {
|
|
const version = randomUUID(), first = await artifactFixture(version), other = await artifactFixture(version);
|
|
const second = { ...other, storage: first.storage };
|
|
const stored = await retainNativeProviderArtifacts({ ...first, source: first.providerArtifacts });
|
|
const [kept] = await first.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, stored.artifactId));
|
|
const descriptor = kept.descriptor as { entries: { name: string; sha256: string }[] };
|
|
await first.db.insert(nativeProviderArtifacts).values(Array.from({ length: 200 }, (_, index) => ({
|
|
companyId: second.companyId, status: "ready", reference: { decoy: index },
|
|
closureDigest: `sha256:${createHash("sha256").update(`decoy ${version} ${index}`).digest("hex")}`,
|
|
descriptor: { ...descriptor, entries: descriptor.entries.map(entry => entry.name === "opencodeProxy"
|
|
? { ...entry, sha256: `sha256:${createHash("sha256").update(`older proxy ${index}`).digest("hex")}` } : entry) },
|
|
})));
|
|
const candidates = await readyNativeProviderArtifactCandidates(second.db, second.companyId, second.manifest);
|
|
expect(candidates.map(row => row.id)).toEqual([stored.artifactId]);
|
|
await expect(retainReadyNativeProviderArtifacts({ db: second.db, owner: second.owner, manifest: second.manifest,
|
|
launchRoot: second.launchRoot, accountUid: 1000 })).resolves.toMatchObject({ artifactId: stored.artifactId });
|
|
});
|
|
it("captures a launch artifact whose bytes differ from every ready artifact", async () => {
|
|
const first = await artifactFixture(randomUUID()), second = await artifactFixture(randomUUID());
|
|
await retainNativeProviderArtifacts({ ...first, source: first.providerArtifacts });
|
|
await expect(retainReadyNativeProviderArtifacts({ db: second.db, owner: second.owner, manifest: second.manifest,
|
|
launchRoot: second.launchRoot, accountUid: 1000 })).resolves.toBeNull();
|
|
const [run] = await second.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, second.runId));
|
|
expect(run.runnerProfileJson?.nativeProviderArtifacts).toBeUndefined();
|
|
});
|
|
it.each(["upload", "download"] as const)("bounds a stalled %s with its own timeout and real cause", async step => {
|
|
const f = await artifactFixture();
|
|
const bounds = { ...nativeProviderArtifactTransferBounds };
|
|
Object.assign(nativeProviderArtifactTransferBounds, { minimumStepMs: 50, minimumBytesPerSecond: Number.MAX_SAFE_INTEGER });
|
|
const stalled = (signal?: AbortSignal) => new Promise<never>((_resolve, reject) => {
|
|
signal?.addEventListener("abort", () => reject(signal.reason), { once: true });
|
|
});
|
|
const stall = step === "upload"
|
|
? vi.spyOn(f.storage, "putFile").mockImplementation(async input => stalled(input.signal))
|
|
: vi.spyOn(f.storage, "getObject").mockImplementation(async (_companyId, _key, options) => stalled(options?.signal));
|
|
try {
|
|
const failure = retainNativeProviderArtifacts({ ...f, source: f.providerArtifacts });
|
|
await expect(failure).rejects.toBeInstanceOf(NativeProviderArtifactTransferTimeoutError);
|
|
await expect(failure).rejects.toMatchObject({ step,
|
|
message: expect.stringMatching(new RegExp(`^native_provider_artifacts_transfer_timeout: the ${step} of \\d+ bytes did not finish within 1 s$`)),
|
|
});
|
|
} finally {
|
|
stall.mockRestore();
|
|
Object.assign(nativeProviderArtifactTransferBounds, bounds);
|
|
}
|
|
});
|
|
it("keeps independent overlapping uploads bounded and retains only their verified winner", async () => {
|
|
const version = randomUUID(), first = await artifactFixture(version), other = await artifactFixture(version);
|
|
const second = { ...other, storage: first.storage }, put = first.storage.putFile.bind(first.storage);
|
|
let release!: () => void, arrivals = 0;
|
|
const together = new Promise<void>(resolve => { release = resolve; });
|
|
const uploaded: string[] = [];
|
|
const calls = vi.spyOn(first.storage, "putFile").mockImplementation(async input => {
|
|
if (input.namespace === "native-provider-artifacts") {
|
|
expect(input.objectKey).toBeTypeOf("string"); uploaded.push(input.objectKey!); if (++arrivals === 2) release();
|
|
await together; expect("byteSize" in input ? input.byteSize : input.body.length).toBeLessThan(512 * 1024 * 1024 + 1024 * 1024);
|
|
}
|
|
return put(input);
|
|
});
|
|
const deleted = vi.spyOn(first.storage, "deleteObject");
|
|
const transfers = Promise.all([persistNativeHarnessBackup(first), persistNativeHarnessBackup(second)]);
|
|
void transfers.catch(() => undefined);
|
|
try {
|
|
try { await vi.waitFor(() => expect(arrivals).toBe(2), { timeout: 10_000 }); } finally { release(); }
|
|
await transfers;
|
|
} finally { calls.mockRestore(); }
|
|
expect(uploaded).toHaveLength(2);
|
|
const objects = await first.db.select().from(nativeProviderArtifacts).where(and(eq(nativeProviderArtifacts.companyId, first.companyId),
|
|
inArray(nativeProviderArtifacts.id, uploaded.map(key => key.split("/").at(-2)!))));
|
|
expect(objects).toHaveLength(1); expect(objects[0].status).toBe("ready");
|
|
expect(deleted.mock.calls.map(([, key]) => key)).toEqual(uploaded.filter(key => key !== objects[0].reference!.objectKey));
|
|
for (const f of [first, second]) {
|
|
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
expect(run.runnerProfileJson?.nativeProviderArtifacts).toMatchObject({ artifactId: objects[0].id });
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(await readFile(join(replacement, "current/provider-artifacts/opencodeProxy"), "utf8")).toBe(f.original.opencodeProxy);
|
|
}
|
|
});
|
|
it("keeps a resumable run pin when an interrupted backup is retired", async () => {
|
|
const f = await artifactFixture(), put = f.storage.putFile.bind(f.storage);
|
|
const interrupted = vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const result = await put(input);
|
|
if (input.namespace === "native-harness-backups") throw new Error("harness upload reply lost");
|
|
return result;
|
|
});
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("harness upload reply lost"); interrupted.mockRestore();
|
|
const [pending] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(pending.status).toBe("pending");
|
|
const receipt = pending.reference.providerArtifacts as NonNullable<Parameters<typeof restoreNativeProviderArtifacts>[0]["receipt"]>;
|
|
const identity = { runnerInstanceId: f.native.runnerInstanceId, environmentLeaseId: f.native.environmentLeaseId,
|
|
runId: f.runId, normalizedSessionId: f.identity.normalizedSessionId, turnId: f.native.turnId, itemId: f.native.itemId };
|
|
const controller = new DurablePrpControlPlane({ stateDirectory: join(f.home, "resumable-controller"), identity,
|
|
expectedRunnerVersion: "0.0.0", expectedRunnerDigest: "sha256:" + "1".repeat(64) });
|
|
const checkpoint = captureLegacyDurablePrpCheckpoint(JSON.stringify(controller.store.state), {
|
|
runId: f.runId, runnerInstanceId: identity.runnerInstanceId, normalizedSessionId: identity.normalizedSessionId },
|
|
{ version: "0.0.0", digest: "sha256:" + "1".repeat(64) });
|
|
await nativeRunnerCheckpointWriter(f.db, { ...f.owner, leaseOwner: f.owner.leaseOwner! })(checkpoint);
|
|
const old = new Date(Date.now() - 10 * 86_400_000); await settle(f, old);
|
|
await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, pending.id));
|
|
const leaseId = randomUUID(), providerLeaseId = randomUUID(), confirmedAt = new Date();
|
|
await f.db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, heartbeatRunId: f.runId,
|
|
issueId: f.issueId, provider: "daytona", providerLeaseId, status: "expired", cleanupStatus: "success", releasedAt: confirmedAt,
|
|
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
|
|
runId: f.runId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed", confirmedAt: confirmedAt.toISOString() } } });
|
|
const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId,
|
|
sessionParamsJson: { sessionId: f.identity.normalizedSessionId } }).returning();
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
const first = await retention.tickCheckpoints();
|
|
expect(first.expired).toBe(1); expect(first.expiredCheckpoints).toBe(0);
|
|
expect(await f.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.runId, f.runId)))
|
|
.toMatchObject([{ backupId: null, artifactId: receipt.artifactId }]);
|
|
expect((await loadNativeRunnerCheckpoint(f.db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId }))?.recoveryEvidence.normalizedSessionId)
|
|
.toBe(f.identity.normalizedSessionId);
|
|
await removeNativeHarnessBackup(f.providerArtifacts.sourceRoot);
|
|
const restored = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(restored);
|
|
await restoreNativeProviderArtifacts({ ...f, receipt, targetRoot: restored });
|
|
expect(await readFile(join(restored, "opencodeProxy"), "utf8")).toBe(f.original.opencodeProxy);
|
|
await f.db.delete(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id));
|
|
expect((await retention.tickCheckpoints()).expiredCheckpoints).toBe(1);
|
|
expect(await f.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, receipt.artifactId))).toEqual([]);
|
|
});
|
|
it("keeps the exact encrypted upload receipt when its artifact owner changes after upload", async () => {
|
|
const f = await artifactFixture(), put = f.storage.putFile.bind(f.storage);
|
|
vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const result = await put(input);
|
|
if (input.namespace === "native-provider-artifacts") await f.db.update(nativeRunFinalizations)
|
|
.set({ controllerGeneration: 2 }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
return result;
|
|
});
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
|
const [pin] = await f.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.runId, f.runId));
|
|
const [artifact] = await f.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, pin.artifactId));
|
|
expect(artifact.status).toBe("pending");
|
|
expect(artifact.reference).toMatchObject({ companyId: f.companyId, artifactId: artifact.id, objectKey: expect.any(String), material: expect.any(Object) });
|
|
expect(await readFile(join(f.providerArtifacts.sourceRoot, "opencodeProxy"), "utf8")).toBe(f.original.opencodeProxy);
|
|
});
|
|
it("retires an abandoned artifact upload through checkpoint expiration without losing deletion retries", async () => {
|
|
const f = await artifactFixture(), put = f.storage.putFile.bind(f.storage);
|
|
const lostReply = vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
await put(input); throw new Error("artifact upload reply lost");
|
|
});
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("artifact upload reply lost"); lostReply.mockRestore();
|
|
const [pin] = await f.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.runId, f.runId));
|
|
const [artifact] = await f.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, pin.artifactId));
|
|
expect(artifact.reference).toMatchObject({ objectKey: expect.any(String), material: expect.any(Object) });
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]);
|
|
await f.db.insert(nativeRunnerCheckpoints).values({ companyId: f.companyId, runId: f.runId,
|
|
issueId: f.issueId, part: "header", material: { retained: "canonical header index fixture" } });
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await retention.tickCheckpoints()).expiredCheckpoints).toBe(0);
|
|
const old = new Date(Date.now() - 10 * 86_400_000);
|
|
await settle(f, old);
|
|
const leaseId = randomUUID(), providerLeaseId = randomUUID(), confirmedAt = new Date();
|
|
await f.db.insert(environmentLeases).values({ id: leaseId, companyId: f.companyId, heartbeatRunId: f.runId,
|
|
issueId: f.issueId, provider: "daytona", providerLeaseId, status: "expired", cleanupStatus: "success", releasedAt: confirmedAt,
|
|
metadata: { remoteExecutionTermination: { schema: "paperclip.remote-termination.v1", companyId: f.companyId,
|
|
runId: f.runId, leaseId, provider: "daytona", providerLeaseId, state: "destroyed", confirmedAt: confirmedAt.toISOString() } } });
|
|
const drop = vi.spyOn(f.storage, "deleteObject").mockRejectedValueOnce(new Error("delete reply lost"));
|
|
const [earlier] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId }).returning();
|
|
expect((await retention.tickCheckpoints()).expiredCheckpoints).toBe(0);
|
|
expect(drop).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)))[0].material)
|
|
.toEqual({ retained: "canonical header index fixture" });
|
|
await f.db.delete(agentTaskSessions).where(eq(agentTaskSessions.id, earlier.id));
|
|
expect((await retention.tickCheckpoints()).failed).toBe(1);
|
|
const [retired] = await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId));
|
|
expect(retired.material).toMatchObject({ schema: "paperclip.native-runner-checkpoint-retirement.v1", companyId: f.companyId,
|
|
runIds: [f.runId], sessionId: f.identity.normalizedSessionId });
|
|
expect(await f.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.artifactId, artifact.id))).toHaveLength(1);
|
|
await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId });
|
|
await expect(loadNativeRunnerCheckpoint(f.db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId }))
|
|
.rejects.toThrow("native_runner_checkpoint_retired");
|
|
const missingStorage = nativeHarnessBackupExpirationService(f.db);
|
|
expect((await missingStorage.tickCheckpoints()).failed).toBe(1);
|
|
expect(drop).toHaveBeenCalledTimes(1);
|
|
const [priorOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const currentOwner = { ...f.owner, generation: priorOwner.controllerGeneration, leaseOwner: "fresh-retired-source-claim" };
|
|
await f.db.update(nativeRunFinalizations).set({ phase: "observed", leaseOwner: currentOwner.leaseOwner,
|
|
leaseExpiresAt: new Date(Date.now() + 60_000) }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const identity = { runnerInstanceId: randomUUID(), environmentLeaseId: randomUUID(), runId: f.runId,
|
|
normalizedSessionId: f.identity.normalizedSessionId, turnId: randomUUID(), itemId: randomUUID() };
|
|
const controller = new DurablePrpControlPlane({ stateDirectory: join(f.home, "new-source-controller"), identity,
|
|
expectedRunnerVersion: "0.0.0", expectedRunnerDigest: "sha256:" + "1".repeat(64) });
|
|
const checkpoint = captureLegacyDurablePrpCheckpoint(JSON.stringify(controller.store.state), {
|
|
runId: f.runId, runnerInstanceId: identity.runnerInstanceId, normalizedSessionId: identity.normalizedSessionId },
|
|
{ version: "0.0.0", digest: "sha256:" + "1".repeat(64) });
|
|
try { await expect(nativeRunnerCheckpointWriter(f.db, currentOwner)(checkpoint)).rejects.toThrow("native_runner_checkpoint_retired"); }
|
|
finally { await f.db.update(nativeRunFinalizations).set({ phase: priorOwner.phase, leaseOwner: null,
|
|
leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, f.runId)); }
|
|
const consumerRunId = randomUUID(), consumerLeaseId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: consumerRunId, companyId: f.companyId, agentId: environment.current.agentId,
|
|
issueId: f.issueId, nativeIssueId: f.issueId, nativeSessionId: f.identity.normalizedSessionId, status: "running" });
|
|
await f.db.insert(nativeRunFinalizations).values({ runId: consumerRunId, companyId: f.companyId, issueId: f.issueId,
|
|
phase: "observed", controllerBootId: f.owner.bootId, controllerGeneration: 1, leaseOwner: "late-source-consumer",
|
|
leaseExpiresAt: new Date(Date.now() + 60_000) });
|
|
await f.db.update(issues).set({ status: "in_progress", executionRunId: consumerRunId }).where(eq(issues.id, f.issueId));
|
|
await f.db.update(environmentLeases).set({ metadata: { remoteExecutionTermination: {
|
|
schema: "paperclip.remote-termination.v1", companyId: f.companyId, runId: f.runId, leaseId, provider: "daytona",
|
|
providerLeaseId, state: "destroyed", confirmedAt: confirmedAt.toISOString() },
|
|
runnerTaskSourceConsumer: { leaseId: consumerLeaseId, runId: consumerRunId, accountIdentity: "owned-fixture-account", identity } } })
|
|
.where(eq(environmentLeases.id, leaseId));
|
|
await f.db.insert(environmentLeases).values({ id: consumerLeaseId, companyId: f.companyId, heartbeatRunId: consumerRunId,
|
|
issueId: f.issueId, provider: "daytona", providerLeaseId: randomUUID(), status: "active", metadata: { runnerTaskSourceRecovery: {
|
|
schema: "paperclip.runner.source-recovery.v1", sourceLeaseId: leaseId, consumerLeaseId, consumerRunId,
|
|
accountIdentity: "owned-fixture-account", controllerBootId: f.owner.bootId, controllerGeneration: 1, identity } } });
|
|
const lateCheckpoint = { ...checkpoint, reset: false, changedRecords: new Map([["cursor", checkpoint.records.get("cursor")!]]), removedRecords: [] };
|
|
await expect(nativeRunnerCheckpointWriter(f.db, { ...f.owner, authorityRunId: consumerRunId, generation: 1,
|
|
leaseOwner: "late-source-consumer" })(lateCheckpoint)).rejects.toThrow("native_runner_checkpoint_retired");
|
|
const restarted = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await restarted.tickCheckpoints()).expiredCheckpoints).toBe(1);
|
|
expect(await f.db.select().from(nativeProviderArtifacts).where(eq(nativeProviderArtifacts.id, artifact.id))).toEqual([]);
|
|
expect(await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId))).toEqual([]);
|
|
expect((await restarted.tickCheckpoints()).expiredCheckpoints).toBe(0);
|
|
expect(drop).toHaveBeenCalledTimes(2);
|
|
});
|
|
it("retains every artifact and checkpoint when a retirement family exceeds its run bound", async () => {
|
|
const f = await artifactFixture(), put = f.storage.putFile.bind(f.storage);
|
|
const lost = vi.spyOn(f.storage, "putFile").mockImplementation(async input => { await put(input); throw new Error("upload reply lost"); });
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("upload reply lost"); lost.mockRestore();
|
|
const old = new Date(Date.now() - 10 * 86_400_000); await settle(f, old);
|
|
await f.db.insert(nativeRunnerCheckpoints).values({ companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
|
part: "header", material: { retained: "original checkpoint" } });
|
|
const relatives = Array.from({ length: 100 }, () => randomUUID());
|
|
await f.db.insert(heartbeatRuns).values(relatives.map(id => ({ id, companyId: f.companyId, agentId: environment.current.agentId,
|
|
issueId: f.issueId, nativeIssueId: f.issueId, nativeSessionId: f.identity.normalizedSessionId, status: "failed", finishedAt: old })));
|
|
await f.db.insert(nativeRunFinalizations).values(relatives.map(runId => ({ runId, companyId: f.companyId, issueId: f.issueId, phase: "terminal_failure" })));
|
|
const remove = vi.spyOn(f.storage, "deleteObject");
|
|
const result = await nativeHarnessBackupExpirationService(f.db, { storage: f.storage }).tickCheckpoints();
|
|
expect(result.expiredCheckpoints).toBe(0); expect(remove).not.toHaveBeenCalled();
|
|
expect((await f.db.select().from(nativeRunnerCheckpoints).where(eq(nativeRunnerCheckpoints.runId, f.runId)))[0].material)
|
|
.toEqual({ retained: "original checkpoint" });
|
|
expect(await f.db.select().from(nativeProviderArtifactReferences).where(eq(nativeProviderArtifactReferences.runId, f.runId))).toHaveLength(1);
|
|
});
|
|
it("keeps the last published backup when object storage fails", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const put = vi.spyOn(f.storage, "putFile").mockRejectedValue(new Error("storage unavailable"));
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("storage unavailable"); put.mockRestore();
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]);
|
|
expect(rows.filter(row => row.status === "pending")).toHaveLength(1);
|
|
expect(await readFile(join(f.root, "manifest.json"), "utf8")).toContain("native-harness-backup");
|
|
});
|
|
async function commitWarmRun(f: Awaited<ReturnType<typeof fixture>>) {
|
|
const [run] = await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId));
|
|
const [result] = await f.db.insert(nativeRunResults).values({ companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
|
completionContractId: run.completionContractId!, serverFingerprint: "completed-warm-turn", schemaStatus: "accepted",
|
|
resultJson: { summary: "Retained provider history" }, canonicalSha256: "completed-warm-turn" }).returning();
|
|
await f.db.update(heartbeatRuns).set({ status: "succeeded", finishedAt: new Date() }).where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.update(nativeRunFinalizations).set({ phase: "committed", resultId: result.id, leaseOwner: null,
|
|
leaseExpiresAt: null }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
return result;
|
|
}
|
|
async function releaseWhileSavingWorkspace(f: Awaited<ReturnType<typeof fixture>>) {
|
|
const result = await commitWarmRun(f);
|
|
await f.db.update(nativeRunFinalizations).set({ phase: "workspace_finalizing" })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await f.db.update(heartbeatRuns).set({ status: "running", finishedAt: null })
|
|
.where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.update(issues).set({ executionRunId: f.runId }).where(eq(issues.id, f.issueId));
|
|
return result;
|
|
}
|
|
async function inactiveFixture() {
|
|
const [f] = await offlineFixtures(1);
|
|
await commitWarmRun(f);
|
|
await f.db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: environment.current.successor.bootId });
|
|
return { ...f, owner, inactiveSource: { normalizedSessionId: f.identity.normalizedSessionId,
|
|
runnerInstanceId: f.runnerInstanceId, providerLeaseId: f.providerLeaseId } };
|
|
}
|
|
it("pre-copies released history and restores its bytes while an unrelated execution stays active", async () => {
|
|
const f = await inactiveFixture(), active = await environment.seedRun(randomUUID());
|
|
const input = { storage: f.storage, signal: AbortSignal.timeout(15_000) };
|
|
expect(await prestageNativeHarnessBackups(f.db, input)).toEqual({ inspected: 1, published: 1, current: 0, deferred: {} });
|
|
expect(await prestageNativeHarnessBackups(f.db, input)).toEqual({ inspected: 1, published: 0, current: 1, deferred: {} });
|
|
await expect(migrateNativeHarnessBackups(f.db, { ...input, writersStopped: true })).rejects.toThrow("requires_offline_writers");
|
|
expect((await active.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, active.runId)))[0].status).toBe("running");
|
|
await removeNativeHarnessBackup(f.home);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", "codex-home", "sessions", "history.json"), "utf8"))).toEqual(f.identity);
|
|
});
|
|
it.each(["queued", "running"] as const)("defers pre-copy when the same task has a %s execution", async status => {
|
|
const f = await inactiveFixture(), put = vi.spyOn(f.storage, "putFile");
|
|
await f.db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId, agentId: environment.current.agentId,
|
|
issueId: f.issueId, nativeIssueId: f.issueId, nativeSessionId: f.identity.normalizedSessionId, status, runtimeMode: "native" });
|
|
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
|
|
.toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_active: 1 } });
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_harness_storage_prestage_source_active");
|
|
expect(put).not.toHaveBeenCalled();
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]);
|
|
});
|
|
it("defers failed history without treating pre-copy as offline authority", async () => {
|
|
const [f] = await offlineFixtures(1);
|
|
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
|
|
.toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_not_successful: 1 } });
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]);
|
|
});
|
|
it("defers a source being replaced online while offline migration still refuses it", async () => {
|
|
const f = await inactiveFixture();
|
|
await writeFile(join(f.root, "runner", "runner-state.json"), JSON.stringify({ runId: f.runId, lifecycle: "running" }));
|
|
expect(await prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) }))
|
|
.toEqual({ inspected: 1, published: 0, current: 0, deferred: { source_unavailable: 1 } });
|
|
await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) }))
|
|
.rejects.toThrow("native_harness_migration_source_unsettled");
|
|
});
|
|
async function releasedFixtures() {
|
|
const fixtures = await offlineFixtures(2);
|
|
for (const f of fixtures) {
|
|
await commitWarmRun(f);
|
|
await f.db.update(issues).set({ executionRunId: null, checkoutRunId: null }).where(eq(issues.id, f.issueId));
|
|
}
|
|
return fixtures;
|
|
}
|
|
async function damageSource(f: Awaited<ReturnType<typeof offlineFixture>>, kind: string) {
|
|
if (kind === "unsafe") {
|
|
await rename(f.root, f.root + "-retained");
|
|
await symlink(f.root + "-retained", f.root);
|
|
} else await writeFile(join(f.root, kind === "manifest" ? "manifest.json" : "runner/runner-state.json"), "{");
|
|
}
|
|
it.each(["source", "output"])("distinguishes %s ENOENT during archive capture after source preparation", async kind => {
|
|
const [bad, good] = await releasedFixtures(), capture = backupArchive.captureNativeHarnessArchive;
|
|
const warning = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
const archive = vi.spyOn(backupArchive, "captureNativeHarnessArchive").mockImplementation(async (root, target, signal, limit) => {
|
|
if (root === bad.root) {
|
|
if (kind === "source") await rename(root, root + "-retained");
|
|
else target = join(target, "missing-parent", "archive");
|
|
}
|
|
return capture(root, target, signal, limit);
|
|
});
|
|
try {
|
|
const preparation = prestageNativeHarnessBackups(bad.db, { storage: bad.storage, signal: AbortSignal.timeout(15_000) });
|
|
if (kind === "source") {
|
|
expect(await preparation).toEqual({ inspected: 2, published: 1, current: 0, deferred: { source_changed: 1 } });
|
|
expect(JSON.parse(warning.mock.calls[0][0])).toMatchObject({ scopeDigest: bad.scopeDigest,
|
|
reason: "native_harness_migration_source_unavailable" });
|
|
expect((await good.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, good.scopeDigest)))[0].status).toBe("current");
|
|
} else {
|
|
await expect(preparation).rejects.toMatchObject({ code: "ENOENT" });
|
|
expect(warning).not.toHaveBeenCalled();
|
|
expect(await good.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, good.scopeDigest))).toHaveLength(0);
|
|
}
|
|
expect(await bad.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, bad.scopeDigest))).toHaveLength(0);
|
|
} finally { archive.mockRestore(); warning.mockRestore(); }
|
|
});
|
|
it.each(["manifest", "runner", "unsafe"])("defers a damaged %s source and still copies the next eligible backup", async kind => {
|
|
const [bad, good] = await releasedFixtures(), warning = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
try {
|
|
await damageSource(bad, kind);
|
|
expect(await prestageNativeHarnessBackups(bad.db, { storage: bad.storage, signal: AbortSignal.timeout(15_000) }))
|
|
.toEqual({ inspected: 2, published: 1, current: 0, deferred: { source_unavailable: 1 } });
|
|
expect(warning).toHaveBeenCalledWith(expect.stringContaining(bad.scopeDigest));
|
|
expect(JSON.parse(warning.mock.calls[0][0])).toMatchObject({ scopeDigest: bad.scopeDigest,
|
|
reason: kind === "unsafe" ? "native_harness_migration_unsafe_source" : "native_harness_migration_source_invalid_json" });
|
|
expect(await bad.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, bad.scopeDigest))).toHaveLength(0);
|
|
expect((await good.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, good.scopeDigest)))[0].status).toBe("current");
|
|
await expect(migrateNativeHarnessBackups(bad.db, { storage: bad.storage, writersStopped: true, signal: AbortSignal.timeout(15_000) }))
|
|
.rejects.toThrow(kind === "unsafe" ? "native_harness_migration_unsafe_source" : /JSON|json/);
|
|
} finally { warning.mockRestore(); }
|
|
});
|
|
it.each(["manifest", "runner", "unsafe"])("defers %s damage at the publication fence and copies the next backup", async kind => {
|
|
const [bad, good] = await releasedFixtures(), put = bad.storage.putFile.bind(bad.storage);
|
|
const warning = vi.spyOn(console, "warn").mockImplementation(() => {});
|
|
const upload = vi.spyOn(bad.storage, "putFile").mockImplementation(async input => {
|
|
const result = await put(input);
|
|
if (input.objectKey?.includes(bad.scopeDigest)) await damageSource(bad, kind);
|
|
return result;
|
|
});
|
|
try {
|
|
expect(await prestageNativeHarnessBackups(bad.db, { storage: bad.storage, signal: AbortSignal.timeout(15_000) }))
|
|
.toEqual({ inspected: 2, published: 1, current: 0, deferred: { source_changed: 1 } });
|
|
expect(JSON.parse(warning.mock.calls[0][0])).toMatchObject({ scopeDigest: bad.scopeDigest,
|
|
reason: kind === "unsafe" ? "native_harness_migration_unsafe_source" : "native_harness_migration_source_invalid_json" });
|
|
expect((await bad.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, bad.scopeDigest))).some(row => row.status === "current")).toBe(false);
|
|
expect((await good.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, good.scopeDigest)))[0].status).toBe("current");
|
|
} finally { upload.mockRestore(); warning.mockRestore(); }
|
|
});
|
|
it("keeps a database SyntaxError fatal instead of deferring it as source damage", async () => {
|
|
const fixtures = await releasedFixtures(), f = fixtures[0], failure = new SyntaxError("database result decoding failed");
|
|
const transaction = vi.spyOn(f.db, "transaction").mockRejectedValueOnce(failure);
|
|
try {
|
|
await expect(prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: AbortSignal.timeout(15_000) })).rejects.toBe(failure);
|
|
for (const source of fixtures) expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, source.scopeDigest))).toHaveLength(0);
|
|
} finally { transaction.mockRestore(); }
|
|
});
|
|
it("keeps cancellation fatal when a source also changes during upload", async () => {
|
|
const [f] = await releasedFixtures(), cancellation = new AbortController(), failure = new Error("operator stopped");
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
const upload = vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const result = await put(input); await damageSource(f, "manifest"); cancellation.abort(failure); return result;
|
|
});
|
|
try {
|
|
await expect(prestageNativeHarnessBackups(f.db, { storage: f.storage, signal: cancellation.signal })).rejects.toBe(failure);
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).some(row => row.status === "current")).toBe(false);
|
|
} finally { upload.mockRestore(); }
|
|
});
|
|
it.each(["source_advanced", "queued_turn", "lease_revoked", "owner_changed", "run_failed"])("keeps the published source when %s happens during pre-copy upload", async change => {
|
|
const f = await inactiveFixture(); await persistNativeHarnessBackup(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const result = await put(input);
|
|
if (change === "source_advanced") await writeFile(join(f.root, "manifest.json"), (await readFile(join(f.root, "manifest.json"), "utf8")) + "\n");
|
|
if (change === "queued_turn") await f.db.insert(heartbeatRuns).values({ id: randomUUID(), companyId: f.companyId,
|
|
agentId: environment.current.agentId, issueId: f.issueId, status: "queued" });
|
|
if (change === "lease_revoked") await f.db.update(environmentLeases).set({ metadata: {} })
|
|
.where(eq(environmentLeases.providerLeaseId, f.providerLeaseId));
|
|
if (change === "owner_changed") await f.db.update(nativeRunFinalizations).set({ controllerGeneration: 2 })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (change === "run_failed") await f.db.update(heartbeatRuns).set({ status: "failed" }).where(eq(heartbeatRuns.id, f.runId));
|
|
return result;
|
|
});
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow(change === "owner_changed" ? "native_runner_checkpoint_owner_lost"
|
|
: change === "queued_turn" ? "native_harness_storage_prestage_source_active" : "native_harness_storage_prestage_source_changed");
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]);
|
|
expect(rows.filter(row => row.status === "pending")).toHaveLength(1);
|
|
expect(await readFile(join(f.root, "manifest.json"), "utf8")).toContain("paperclip.native-harness-backup.v1");
|
|
});
|
|
it("publishes with the captured provider owner while an accepted result is still saving its workspace", async () => {
|
|
const f = await fixture();
|
|
const result = await releaseWhileSavingWorkspace(f);
|
|
await expect(nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: f.owner.bootId })).rejects.toThrow("native_harness_storage_owner_missing");
|
|
await persistNativeHarnessBackup(f);
|
|
await removeNativeHarnessBackup(f.home);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "restored-pending-workspace");
|
|
ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", "codex", "sessions", "history.json"), "utf8"))).toEqual(f.identity);
|
|
const [coordinator] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(coordinator).toMatchObject({ phase: "workspace_finalizing", resultId: result.id, leaseOwner: null,
|
|
leaseExpiresAt: null, controllerBootId: f.owner.bootId, controllerGeneration: f.owner.generation });
|
|
});
|
|
it.each(["boot", "generation", "result", "successor"])("rejects a captured provider fence when its %s changes during upload", async change => {
|
|
const f = await fixture();
|
|
await persistNativeHarnessBackup(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
await releaseWhileSavingWorkspace(f);
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
vi.spyOn(f.storage, "putFile").mockImplementation(async input => {
|
|
const object = await put(input);
|
|
if (change === "successor") {
|
|
const nextRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: nextRunId, companyId: f.companyId,
|
|
agentId: environment.current.agentId, issueId: f.issueId, status: "running" });
|
|
await f.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, f.issueId));
|
|
} else {
|
|
await f.db.update(nativeRunFinalizations).set(change === "boot" ? { controllerBootId: randomUUID() }
|
|
: change === "generation" ? { controllerGeneration: 2 } : { resultId: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
}
|
|
return object;
|
|
});
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow(change === "successor"
|
|
? "native_harness_storage_owner_superseded" : "native_runner_checkpoint_owner_lost");
|
|
const selected = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
|
|
eq(nativeHarnessBackups.status, "current")));
|
|
expect(selected.map(row => row.id)).toEqual([before.id]);
|
|
});
|
|
it.each(["idle", "preparing", "native"])("preserves a released committed warm session with a %s successor", async state => {
|
|
const f = await fixture();
|
|
const result = await commitWarmRun(f);
|
|
let nextRunId: string | null = null;
|
|
if (state !== "idle") {
|
|
nextRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: nextRunId, companyId: f.companyId, agentId: environment.current.agentId,
|
|
issueId: f.issueId, status: "running", runtimeMode: state === "native" ? "native" : "legacy", executionStage: "preparing",
|
|
...(state === "native" ? { nativeIssueId: f.issueId } : {}),
|
|
controllerBootId: legacyControllerBootId, controllerLeaseExpiresAt: new Date(Date.now() + 120_000) });
|
|
if (state === "native") await f.db.insert(nativeRunFinalizations).values({ runId: nextRunId,
|
|
companyId: f.companyId, issueId: f.issueId, phase: "observed", controllerBootId: environment.current.successor.bootId,
|
|
controllerGeneration: 1, leaseOwner: "preparing-warm-successor", leaseExpiresAt: new Date(Date.now() + 120_000) });
|
|
}
|
|
await f.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, f.issueId));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: environment.current.successor.bootId });
|
|
await persistNativeHarnessBackup({ ...f, owner });
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "restored-warm-session"); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", "codex", "sessions", "history.json"), "utf8"))).toEqual(f.identity);
|
|
const [coordinator] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
expect(coordinator).toMatchObject({ phase: "committed", resultId: result.id, leaseOwner: null, leaseExpiresAt: null });
|
|
const [task] = await f.db.select().from(issues).where(eq(issues.id, f.issueId));
|
|
expect(task.executionRunId).toBe(nextRunId);
|
|
});
|
|
it.each(["unaccepted", "uncommitted", "wrong_boot"])("rejects a released warm owner with %s completion evidence", async state => {
|
|
const f = await fixture(); await commitWarmRun(f);
|
|
if (state === "unaccepted") await f.db.update(nativeRunFinalizations).set({ resultId: null }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
if (state === "uncommitted") await f.db.update(nativeRunFinalizations).set({ phase: "observed" }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await expect(nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId, runId: f.runId,
|
|
bootId: state === "wrong_boot" ? randomUUID() : environment.current.successor.bootId })).rejects.toThrow("native_harness_storage_owner_missing");
|
|
});
|
|
it("rejects an active predecessor after the task moves to a successor", async () => {
|
|
const f = await fixture();
|
|
const nextRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: nextRunId, companyId: f.companyId, agentId: environment.current.agentId,
|
|
nativeIssueId: f.issueId, issueId: f.issueId, status: "running", runtimeMode: "native" });
|
|
await f.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, f.issueId));
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_harness_storage_owner_superseded");
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toEqual([]);
|
|
});
|
|
it.each(["foreign_preparation", "expired_preparation", "dispatching_successor", "foreign_native"])("keeps the selected backup when a released owner has a %s", async state => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f); await commitWarmRun(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const nextRunId = randomUUID();
|
|
await f.db.insert(heartbeatRuns).values({ id: nextRunId, companyId: f.companyId, agentId: environment.current.agentId,
|
|
issueId: f.issueId, status: "running", runtimeMode: state === "foreign_native" ? "native" : "legacy",
|
|
...(state === "foreign_native" ? { nativeIssueId: f.issueId } : {}),
|
|
executionStage: state === "dispatching_successor" ? "dispatching" : "preparing",
|
|
controllerBootId: state === "foreign_preparation" ? randomUUID() : legacyControllerBootId,
|
|
controllerLeaseExpiresAt: state === "expired_preparation" ? new Date(0) : new Date(Date.now() + 120_000) });
|
|
if (state === "foreign_native") await f.db.insert(nativeRunFinalizations).values({ runId: nextRunId,
|
|
companyId: f.companyId, issueId: f.issueId, phase: "observed", controllerBootId: randomUUID(),
|
|
controllerGeneration: 1, leaseOwner: "foreign-warm-successor", leaseExpiresAt: new Date(Date.now() + 120_000) });
|
|
await f.db.update(issues).set({ executionRunId: nextRunId }).where(eq(issues.id, f.issueId));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: environment.current.successor.bootId });
|
|
await expect(persistNativeHarnessBackup({ ...f, owner })).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
|
|
eq(nativeHarnessBackups.status, "current")))).map(row => row.id)).toEqual([before.id]);
|
|
});
|
|
it.each(["generation", "result", "phase"])("rechecks a released owner's %s after object upload", async state => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f); await commitWarmRun(f);
|
|
await f.db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: environment.current.successor.bootId });
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
f.storage.putFile = async input => {
|
|
const result = await put(input);
|
|
await f.db.update(nativeRunFinalizations).set(state === "generation" ? { controllerGeneration: 2 }
|
|
: state === "result" ? { resultId: null } : { phase: "observed" }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
return result;
|
|
};
|
|
await expect(persistNativeHarnessBackup({ ...f, owner })).rejects.toThrow(state === "generation"
|
|
? "native_runner_checkpoint_owner_lost" : "native_harness_storage_owner_missing");
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
|
|
eq(nativeHarnessBackups.status, "current")))).map(row => row.id)).toEqual([before.id]);
|
|
});
|
|
it("cannot replace a newer conversation backup with a released predecessor's late upload", async () => {
|
|
const f = await fixture(); await commitWarmRun(f);
|
|
await f.db.update(issues).set({ executionRunId: null }).where(eq(issues.id, f.issueId));
|
|
const owner = await nativeHarnessBackupOwner(f.db, { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, bootId: environment.current.successor.bootId });
|
|
const next = await fixture();
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
f.storage.putFile = async input => {
|
|
const result = await put(input);
|
|
await persistNativeHarnessBackup({ ...next, scopeDigest: f.scopeDigest });
|
|
return result;
|
|
};
|
|
await expect(persistNativeHarnessBackup({ ...f, owner })).rejects.toThrow("native_harness_storage_owner_superseded");
|
|
const [current] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest),
|
|
eq(nativeHarnessBackups.status, "current")));
|
|
expect(current.runId).toBe(next.runId);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "latest-conversation"); ownedHomes.push(replacement);
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).toBe(true);
|
|
expect(JSON.parse(await readFile(join(replacement, "current", "codex", "sessions", "history.json"), "utf8"))).toEqual(next.identity);
|
|
});
|
|
it("fences an old controller whose upload finishes after ownership changes", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
f.storage.putFile = async input => {
|
|
const result = await put(input);
|
|
await f.db.update(nativeRunFinalizations).set({ controllerBootId: randomUUID(), controllerGeneration: 2 })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
return result;
|
|
};
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_runner_checkpoint_owner_lost");
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(rows.filter(row => row.status === "current").map(row => row.id)).toEqual([before.id]);
|
|
});
|
|
it("refuses publication after a pending upload was retired without demoting the current backup", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const [current] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const put = f.storage.putFile.bind(f.storage);
|
|
f.storage.putFile = async input => {
|
|
const result = await put(input);
|
|
await f.db.update(nativeHarnessBackups).set({ status: "retired" }).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "pending")));
|
|
return result;
|
|
};
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("native_harness_storage_publication_retired");
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, current.id)))[0].status).toBe("current");
|
|
const [retired] = await f.db.select().from(nativeHarnessBackups).where(and(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest), eq(nativeHarnessBackups.status, "retired")));
|
|
expect(await deleteRetiredNativeHarnessBackup(f.db, f.storage, retired.id)).toBe(true);
|
|
});
|
|
it("rejects corrupt objects and cross-company lookups without publishing a local cache", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const replacement = join(environment.current.runtimeRoot, randomUUID(), "failover-backups");
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, companyId: randomUUID(), backupRoot: replacement })).toBe(false);
|
|
const [row] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
await f.storage.deleteObject(f.companyId, row.reference.objectKey as string);
|
|
await f.storage.putFile({ companyId: f.companyId, namespace: "native-harness-backups", objectKey: row.reference.objectKey as string,
|
|
originalFilename: null, contentType: "application/octet-stream", body: Buffer.from("corrupted") });
|
|
await expect(restoreDurableNativeHarnessBackup({ ...f, backupRoot: replacement })).rejects.toThrow("hash_mismatch");
|
|
await expect(stat(join(replacement, "current"))).rejects.toMatchObject({ code: "ENOENT" });
|
|
});
|
|
async function settle(f: { db: Awaited<ReturnType<typeof fixture>>["db"]; runId: string; issueId: string }, closedAt = new Date()) {
|
|
await f.db.update(heartbeatRuns).set({ status: "failed", finishedAt: closedAt }).where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.update(nativeRunFinalizations).set({ phase: "terminal_failure", leaseOwner: null, leaseExpiresAt: null })
|
|
.where(eq(nativeRunFinalizations.runId, f.runId));
|
|
await f.db.update(issues).set({ status: "done", executionRunId: null, completedAt: closedAt }).where(eq(issues.id, f.issueId));
|
|
}
|
|
it("migrates a lease-verified local backup once and keeps its selected session intact", async () => {
|
|
const f = await fixture("codex-home");
|
|
const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base);
|
|
await rename(f.home, join(base, f.scopeDigest));
|
|
const root = join(base, f.scopeDigest, "failover-backups", "current");
|
|
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base);
|
|
const runnerInstanceId = randomUUID(), providerLeaseId = randomUUID();
|
|
const execution = { provider: { kind: "codex" }, binding: { companyId: f.companyId, issueId: f.issueId,
|
|
runId: f.runId, agentId: environment.current.agentId, executionWorkspaceId: randomUUID() },
|
|
workspace: { cwd: "/workspace", repoUrl: null, repoRef: null, branchName: null },
|
|
session: { normalizedSessionId: f.identity.normalizedSessionId, driverKind: "codex_app_server",
|
|
protocolVersion: 1, lifecyclePolicy: { mode: "per_turn", idleTimeoutMs: null } } } as never;
|
|
await chmod(join(root, "codex-home", "sessions"), 0o700);
|
|
await symlink("history.json", join(root, "codex-home", "sessions", "history-link"));
|
|
await chmod(join(root, "codex-home", "sessions"), 0o500);
|
|
const manifest = buildNativeHarnessBackupManifest({ backupRoot: root, execution, runnerInstanceId,
|
|
providerSessionIdentity: { providerSessionId: f.identity.providerSessionId }, sourceProviderLeaseId: providerLeaseId });
|
|
await writeFile(join(root, "manifest.json"), JSON.stringify(manifest));
|
|
const stamp = createNativeHarnessBackupStamp({ manifestPath: join(root, "manifest.json"), sessionScopeId: f.sessionScopeId,
|
|
authorizedProviderLeaseId: providerLeaseId, runnerInstanceId, normalizedSessionId: f.identity.normalizedSessionId, completedAt: manifest.completedAt });
|
|
expect(verifyNativeHarnessBackupStamp(stamp, providerLeaseId)).toBe(true);
|
|
expect(await verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.timeout(10_000))).toBe(true);
|
|
await expect(verifyNativeHarnessBackupStampAsync(stamp, providerLeaseId, AbortSignal.abort(new Error("cancelled")))).rejects.toThrow("cancelled");
|
|
await f.db.update(heartbeatRuns).set({ nativeSessionId: f.identity.normalizedSessionId, runnerInstanceId }).where(eq(heartbeatRuns.id, f.runId));
|
|
await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId, heartbeatRunId: f.runId,
|
|
providerLeaseId, status: "expired", metadata: { nativeHarnessBackup: stamp } });
|
|
const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId,
|
|
sessionParamsJson: { sessionId: f.identity.normalizedSessionId } }).returning();
|
|
expect(await hasLocalNativeHarnessObligations(f.db)).toBe(true);
|
|
await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) }))
|
|
.rejects.toThrow("native_harness_migration_requires_offline_writers");
|
|
const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.companyId, f.companyId));
|
|
for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! });
|
|
await f.db.update(nativeRunFinalizations).set({ recoveryState: "blocked" }).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const [preservedOwner] = await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId));
|
|
const [queued] = await f.db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
status: "queued", invocationSource: "on_demand", issueId: f.issueId }).returning();
|
|
await f.db.update(issues).set({ executionRunId: queued.id }).where(eq(issues.id, f.issueId));
|
|
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1);
|
|
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(1);
|
|
expect(await hasLocalNativeHarnessObligations(f.db)).toBe(false);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1);
|
|
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected);
|
|
expect((await f.db.select().from(nativeRunFinalizations).where(eq(nativeRunFinalizations.runId, f.runId)))[0]).toEqual(preservedOwner);
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, queued.id)))[0]).toEqual(queued);
|
|
expect((await f.db.select().from(issues).where(eq(issues.id, f.issueId)))[0].executionRunId).toBe(queued.id);
|
|
expect(JSON.parse(await readFile(join(root, "manifest.json"), "utf8"))).toEqual(manifest);
|
|
const restored = join(base, "replacement");
|
|
expect(await restoreDurableNativeHarnessBackup({ ...f, backupRoot: restored })).toBe(true);
|
|
expect(await readlink(join(restored, "current", "codex-home", "sessions", "history-link"))).toBe("history.json");
|
|
const changedManifest = JSON.stringify({ ...manifest, completedAt: new Date(Date.now() + 1_000).toISOString() });
|
|
await writeFile(join(root, "manifest.json"), changedManifest);
|
|
await expect(migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) }))
|
|
.rejects.toThrow("native_harness_migration_durable_source_conflict");
|
|
expect(await readFile(join(root, "manifest.json"), "utf8")).toBe(changedManifest);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(1);
|
|
});
|
|
it("preserves orphaned local backup files without blocking other enrollment", async () => {
|
|
const f = await fixture();
|
|
const base = join(environment.current.runtimeRoot, randomUUID()); ownedHomes.push(base); await mkdir(base);
|
|
await rename(f.home, join(base, f.scopeDigest));
|
|
vi.stubEnv("PAPERCLIP_RUNNER_STATE_DIR", base);
|
|
const root = join(base, f.scopeDigest, "failover-backups", "current");
|
|
const orphan = JSON.stringify({ runId: randomUUID(), lifecycle: "suspended" });
|
|
await writeFile(join(root, "runner", "runner-state.json"), orphan);
|
|
const ownedRuns = await f.db.select({ id: heartbeatRuns.id, issueId: heartbeatRuns.nativeIssueId }).from(heartbeatRuns)
|
|
.where(eq(heartbeatRuns.companyId, f.companyId));
|
|
for (const run of ownedRuns) await settle({ ...f, runId: run.id, issueId: run.issueId! });
|
|
expect(await migrateNativeHarnessBackups(f.db, { storage: f.storage, writersStopped: true, signal: AbortSignal.timeout(10_000) })).toBe(0);
|
|
expect(await readFile(join(root, "runner", "runner-state.json"), "utf8")).toBe(orphan);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0);
|
|
});
|
|
it("retains live work and retries an uncertain object deletion through the existing startup retention pass", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const old = new Date(Date.now() - 10 * 86_400_000);
|
|
await f.db.update(nativeHarnessBackups).set({ createdAt: old }).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await retention.tickCheckpoints()).expired).toBe(0);
|
|
await settle(f, old);
|
|
const drop = vi.spyOn(f.storage, "deleteObject").mockRejectedValueOnce(new Error("delete reply lost"));
|
|
expect((await retention.tickCheckpoints()).failed).toBe(1);
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest)))[0].status).toBe("retired");
|
|
expect((await retention.tickCheckpoints()).expired).toBe(1);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest))).toHaveLength(0);
|
|
expect(drop).toHaveBeenCalledTimes(2);
|
|
});
|
|
it("expires obsolete objects without clearing an open conversation or its exact lease backup", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const [obsolete] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
await writeFile(join(f.root, "manifest.json"), JSON.stringify({ revision: 2 }));
|
|
await persistNativeHarnessBackup(f);
|
|
const old = new Date(Date.now() - 10 * 86_400_000);
|
|
await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, obsolete.id));
|
|
const [selected] = await f.db.insert(agentTaskSessions).values({ companyId: f.companyId, agentId: environment.current.agentId,
|
|
adapterType: "paperclip_runner", taskKey: f.issueId, lastRunId: f.runId }).returning();
|
|
const [lease] = await f.db.insert(environmentLeases).values({ companyId: f.companyId, issueId: f.issueId,
|
|
heartbeatRunId: f.runId, status: "pending_cleanup", metadata: { nativeHarnessBackup: {
|
|
sessionScopeSha256: f.scopeDigest, manifestSha256: obsolete.manifestSha256 } } }).returning();
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await retention.tickCheckpoints()).expired).toBe(0);
|
|
await f.db.update(environmentLeases).set({ status: "expired" }).where(eq(environmentLeases.id, lease.id));
|
|
expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, obsolete.id))).toHaveLength(1);
|
|
expect((await retention.tickCheckpoints()).expired).toBe(1);
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(rows).toHaveLength(1); expect(rows[0].status).toBe("current");
|
|
expect((await f.db.select().from(agentTaskSessions).where(eq(agentTaskSessions.id, selected.id)))[0]).toEqual(selected);
|
|
expect((await f.db.select().from(heartbeatRuns).where(eq(heartbeatRuns.id, f.runId)))[0].status).toBe("running");
|
|
});
|
|
it("retains an in-flight pending upload and removes it once its owner settles", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const put = vi.spyOn(f.storage, "putFile").mockRejectedValueOnce(new Error("upload unavailable"));
|
|
await expect(persistNativeHarnessBackup(f)).rejects.toThrow("upload unavailable"); put.mockRestore();
|
|
const rows = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const pending = rows.find(row => row.status === "pending")!;
|
|
const old = new Date(Date.now() - 10 * 86_400_000);
|
|
await f.db.update(nativeHarnessBackups).set({ createdAt: old, updatedAt: old }).where(eq(nativeHarnessBackups.id, pending.id));
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await retention.tickCheckpoints()).expired).toBe(0);
|
|
await settle(f);
|
|
expect((await retention.tickCheckpoints()).expired).toBe(1);
|
|
const remaining = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
expect(remaining).toHaveLength(1); expect(remaining[0].status).toBe("current");
|
|
});
|
|
it("keeps deleted-source objects unchanged during a retention dry run", async () => {
|
|
const f = await fixture(); await persistNativeHarnessBackup(f);
|
|
const old = new Date(Date.now() - 10 * 86_400_000);
|
|
await f.db.update(nativeHarnessBackups).set({ createdAt: old, runId: randomUUID() })
|
|
.where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const [before] = await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.scopeDigest, f.scopeDigest));
|
|
const retention = nativeHarnessBackupExpirationService(f.db, { storage: f.storage });
|
|
expect((await retention.tickCheckpoints({ dryRun: true })).expired).toBe(0);
|
|
expect((await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id)))[0]).toEqual(before);
|
|
expect(await verifyDurableNativeHarnessBackup(f)).toBe(true);
|
|
expect((await retention.tickCheckpoints()).expired).toBe(1);
|
|
expect(await f.db.select().from(nativeHarnessBackups).where(eq(nativeHarnessBackups.id, before.id))).toHaveLength(0);
|
|
});
|
|
});
|