fix(workspaces): preserve completed shallow sandbox exports (#18221)

## Thinking Path

> - Paperclip coordinates agent work and preserves its results.
> - Remote execution copies completed workspace changes back to the
host.
> - An agent can fetch a published branch with shallow Git history.
> - A full Git bundle omits the shallow boundary metadata and can fail
host connectivity checks.
> - This pull request carries the boundaries through the full-bundle
fallback and imports through normal Git fetch.
> - Recovery must also recognize completed native work on a reusable
lease without treating it as ordinary executor reuse.
> - Completed work can be saved without another provider turn, while
existing host history remains connected.

## Linked Issues or Issue Description

**What happened?**
Sandbox finalization can fail with a missing parent commit after an
agent checks out a separately fetched shallow branch. Re-exporting a
full bundle repeats the failure because Git bundles do not encode
shallow boundaries.

**Expected behavior**
Import the completed branch and saved files. Preserve existing host
ancestry and concurrent host edits. Keep normal Git connectivity and
workspace merge checks.

**Steps to reproduce**
1. Prepare a sandbox from one host commit.
2. Fetch a different published branch with depth one and check it out.
3. Save a result file and restore the sandbox workspace.

**Paperclip version or commit**
Reproduced against master before this repair. The regression fails with
the missing-parent error before the production change.

Related public context: #15268 covers rewritten history integration.
This repair handles shallow bundle transport before integration.
Searches found no equivalent open repair.

## What Changed

- Export shallow boundaries beside the full bundle when the delta import
lacks history.
- Unpack into a temporary shared bare repository. Keep parent
connections already available on the host and import with Git's shallow
fetch protocol.
- Reject malformed boundary values and retain existing connectivity,
branch, and concurrent-work merge guards. Keep boundary probes within 15
seconds without consuming the separate 60-second Git import budget.
- Let source-bound preservation recover reusable leases only for the
finalizer-selected, accepted successful native result and matching
completion contract. Retain provider-stop, physical ownership and
host-writer checks; ordinary reusable sources remain blocked.
- Classify invalid Git metadata separately from omittable unsafe
archives. Keep native finalization failed and preserve service-owned
file recovery; carry the sender's dedicated rejection across the command
boundary without transient retries.
- Share the failure allowlist across bounded run/activity projections
and cancelled-preservation selection. Migration `0416` rebuilds the
matching partial index; task chat retains the non-retryable recovery
notice.
- Add real Git, filesystem sandbox and database recovery regressions.
Update execution semantics and affected verification maps.

## Verification

- Red: real filesystem sandbox restore failed with the same
missing-parent/revision-walk error before the export repair.
- Green: 240 focused Git, workspace restore, sandbox transport and
native-finalization tests pass. Coverage includes both transfer paths,
malformed/oversized metadata, a genuinely shallow host with its original
boundary intact, and concurrent host work with both merge parents
verified.
- Red/green: a 14-second simulated probe left the final real Git fetch
only 45,893 ms before the budget split. It now retains over 55 seconds
after the probes; a 16-second probe exhausts its own budget before host
import.
- Red/green: an oversized read progress report was retried three times
before its distinct metadata classification. It now aborts the first
read before payload consumption, leaves the host tip unchanged and does
not retry.
- Red/green: the real metadata error was swallowed by best-effort export
and native finalization incorrectly returned success. Both regressions
now pass; all 240 focused Git/restore/native-finalization tests pass. A
PostgreSQL/filesystem fault-injection test also verifies retained files,
failed run, service-owned recovery barrier, no preservation receipt and
no new wake.
- Red/green: duplicate metadata produced 47 Git process starts before
deduplication; the bounded-work regression now passes. Oversized byte
and boundary inputs fail before Git probes.
- Recovery red: both reusable lease policies exhausted recovery despite
a selected accepted successful result. The assembled
PostgreSQL/filesystem path now passes for retained and both reusable
policies, including idempotent re-admission after exhausted automatic
recovery, saved files, no agent wake, and once-only existing-result
finalization.
- Source verification controls reject ordinary reusable leases,
rejected/failed/unselected results, another physical allocation owner,
and an active host writer without changing recorded state.
- The 78-test PostgreSQL/filesystem recovery/source suite passed before
the metadata classification correction. Its new persisted
metadata-failure regression also passes. Full workspace typecheck and
build pass at final head `69f10daab2`.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Full local `pnpm test:run` attempts did not finish. They are not
counted as passes. Early focused runs exceeded 5-second and 30-second
fixture deadlines; the final serial suite passed all 176 tests with a
30-second per-test budget in 56.8 seconds.
- Red/green: both bounded PostgreSQL projections dropped the new code in
oversized results. All 19 source/projection cases now pass, plus an
actual index-predicate test that excludes a metadata repair hold and
still selects ordinary cancelled work.
- Red/green: an oversized sender `.git/shallow` lost its error type
before host validation. The real transport test now verifies one typed
rejection, no shallow-payload read, and an unchanged host tip. All 465
command/sandbox/task-chat tests pass, and token/migration gates pass.
- Full typecheck/build pass after these final contract corrections at
`69f10daab2`; all 20 PostgreSQL source/projection/index contracts pass.
All current-head CI checks passed; Greptile is 5/5 with no unresolved
review threads. Merged #18230 supplies the CI-cache and Linux
shell-fixture baseline repairs; their duplicate changes have been
removed from this PR. The final Runner lane passed after one justified
rerun of an unchanged authentication fixture timeout (all 11 fixture
tests also pass locally). Previous CI failures and incomplete local runs
remain recorded; live deployment and retained-run recovery remain
separate verification.

## Risks

- The fallback creates a temporary shared bare clone. Metadata is
limited to 64 KiB and 256 unique boundaries, duplicate/parent probes are
cached, and Git import has a 60-second command budget plus a separate
15-second boundary-probe budget. It runs only after a missing-history
import failure.
- Incorrect boundaries could hide real host ancestry. The importer
retains available parent connections, and regressions cover that case.
- The full-bundle fallback can transfer a bundle twice. This preserves
the existing fast delta path; removing that extra transfer is outside
this correctness repair.
- Preservation eligibility expands only for completed native exports on
reusable leases. It creates no provider turn and leaves ordinary
reusable leases ineligible. Database/filesystem regressions cover result
selection and competing physical/host ownership.
- The migration rebuilds one partial index; no tables, columns or user
records change. There are no agent execution admission, model, trust,
recovery permission, workflow or lockfile changes.

## Model Used

OpenAI Codex, GPT-6, with reasoning, code execution, and browser tools.
The exact serving variant and context-window size are not exposed in
this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-29 20:02:53 -05:00
1 parent 7d194e901f
commit ed8fae8dbb
35 files changed
+971 -49

No files matched your search

+5
View File
@@ -49,6 +49,11 @@ and leaves all existing cases passive. Source identities are not grants.
### Task query performance
The cancelled workspace-preservation partial index and bounded run/activity
projections share the restore-failure allowlist. Changing those failure codes
also changes the index predicate; use the normal migration-generation workflow.
A required metadata repair is excluded from ordinary cancelled preservation work.
Native session resume selects its newest progress barrier within the existing
32-run task/session window in one SQL snapshot. Only the chosen checkpoint source
returns its saved native input, checkpoint, tool fingerprint, wake envelope, and
+23
View File
@@ -710,6 +710,22 @@ The directory merge still preserves host-only
file changes under its existing rules. A changed host branch requires recovery.
An intentional reset to an ancestor exports a full Git bundle so restore keeps
the actual sandbox tip; an empty delta is reserved for an unchanged tip.
A full Git bundle from a shallow repository still omits history before its
shallow boundaries. When a delta import cannot satisfy its history, sandbox
restore exports those boundaries with the full bundle. The host unpacks it in
a temporary bare repository and uses Git's shallow fetch protocol to import it.
This preserves existing host ancestry and connectivity checks. File restore and
the normal concurrent-work merge must still finish before run finalization.
The boundary metadata has a 64 KiB transfer/read budget and at most 256 unique
commits. Duplicate entries do not repeat host Git probes. Boundary inspection
has a separate 15-second budget; cloning, unpacking and import share 60 seconds
without charging probe time. Oversized or malformed metadata fails before import
with a distinct non-retryable metadata classification. It requires file recovery
and never uses the unsafe-archive omission path to complete an accepted result.
The sandbox producer uses a dedicated exit status which the command client retains
and the export owner converts back to the typed metadata error. Bounded API
projections and cancelled-preservation selection use the shared failure allowlist;
task chat retains its recovery notice without offering another provider turn.
Unrelated sandbox history keeps the existing history-preserving graft only when
the recorded host has not advanced; it must not replace concurrent host work.
Warm sandbox reuse must match the current host Git tip and branch as well as the
@@ -1576,6 +1592,13 @@ and obtains a fresh stop receipt. An interrupted legacy run does not need an
accepted native result. A successful `workspace_recovery` operation and matching
recovery record prove persistence; adapter output or a user repair note cannot.
The failed run and unknown external effects remain historical facts.
A reusable lease can also owe native copyback: the export-repair action must
select its exact immutable descriptor and the finalizer's accepted successful
result under the same company, task, run and completion contract. Lease reuse
policy alone does not discharge that persistence obligation. Ordinary reusable
sources without this proof remain ineligible. The preservation owner still
requires an independently stopped, exclusively bound allocation and no host writer;
it saves files and resumes the existing finalizer without another provider turn.
Startup and the retained-workspace sweep reconstruct these obligations through
the retained workspace index. Committed Retry, recovery admission, and recovery
+1 -1
View File
@@ -321,7 +321,7 @@ API: `present` — POST /api/issues/{id}/recovery-actions/recover-workspace with
Agent access: `restricted` — board task-read and issue:mutate; not runtime:manage. Agents denied through REST/native fallback.
Policy: Exact terminal retained source/lease/reference; independent stop and physical ownership checks; no newer writer. Bounded queued repair preserves newer files and has retry state. Matching key/recovered state dedupes; permanent blocked source refuses. Recovery starts no provider turn.
Policy: Exact terminal retained source/lease/reference; independent stop and physical ownership checks; no newer writer. Bounded queued repair preserves newer files and has retry state. Matching key/recovered state dedupes; permanent blocked source refuses. Recovery starts no provider turn. For native export repair, a reusable lease is eligible only with its exact descriptor and the finalizer-selected accepted successful result under the same company/task/run/completion contract. This does not change actor access or permit another allocation owner or host writer. Invalid Git metadata is a non-retryable persistence failure requiring service repair; it cannot use unsafe-archive omission to complete the result. Run-detail/activity projections retain the allowlisted code, including bounded large-result retrieval, and cancelled-preservation discovery honors the same hold.
Evidence: `source-only` — [recovery routes](../../server/src/routes/issues.ts), [retained recovery](../../server/src/services/workspace-recovery.ts); reviewed at `3bbc2104b65caf9f7763938a01260a5673e12f7a`. No live provider proof.
+7
View File
@@ -305,6 +305,13 @@ guard. This changes no actor permission or session-reset authority. Follow the
[session recipe](../feature-map/runs-adapters.md#session) for recovery followed by
ordinary turns; deterministic runner checks do not establish live provider access.
Sandbox export carries shallow Git boundaries through the full-bundle fallback
and uses normal Git fetch connectivity checks. Existing host ancestry and merge
guards remain effective. This grants no task retry or operator recovery authority;
the saved-result export API still requires board runtime management permission.
Follow the [session recipe](../feature-map/runs-adapters.md#session) for separately
fetched shallow history; filesystem controls do not establish live recovery.
Plugin suspension receipts remain required through recovery and admission, then
are removed when the admitted session becomes live. A later warm run therefore
does not inherit an obsolete stopped-run identity. Repository-free fresh plugin
+12 -1
View File
@@ -399,7 +399,18 @@ For a source with a retained-file protection marker, exercise recovery through
the assembled environment runtime. Its exact preservation owner may reopen the
stopped allocation to read saved files; ordinary resume stays blocked. Reject a
changed claim, missing stop intent, wrong lease or wrong environment before a
provider call. Verify saved and newer local files, one preservation receipt, and
provider call. Repeat accepted-native-result copyback for both reusable lease policies. Require
the exact accepted result and completion contract; reject a rejected or unselected
result, another allocation owner, and an active host writer. Verify no provider
turn or wake is created, then finalize the original result once. Reject malformed
or oversized Git metadata through native finalization: the run must fail and
retain its file-recovery barrier, rather than successfully omitting the export.
Exercise both an oversized sender file and an oversized transfer report. Verify
one rejected export, no payload read after rejection, and unchanged host files.
Reload a large-result run and its activity: the failure code and task recovery
notice remain visible, and metadata repair offers no new provider turn. These database
and filesystem controls do not qualify a deployed provider.
Verify saved and newer local files, one preservation receipt, and
no additional agent turn. The PostgreSQL regression covers this boundary; the
opt-in provider probe also uses this runtime path with the protection marker.
+7
View File
@@ -102,6 +102,13 @@ The [remote continuity regression](../../doc/testing-native-remote-continuity.md
uses a real Linux runner and deterministic provider. Live provider acceptance is
a separate check.
In a fresh sandbox workspace, fetch a different published branch with depth one,
then save a local result and finish. Verify export imports the shallow history,
retains host-only edits, and completes finalization without another provider turn.
Repeat when the host already has that branch history; its ancestry must remain
available. The filesystem sandbox regression covers transfer and merge behavior,
not live provider acceptance.
For a plugin computer, complete a task, let its warm session suspend, then send
two follow-ups within the next warm window. Verify the first resumes the saved
session and the second keeps the same provider history without an identity
@@ -1027,6 +1027,8 @@ describe("command managed runtime", () => {
await expect(client.run("tar -cf workspace-download.tar .", { timeoutMs: 30_000 })).rejects.toThrow(
/stdout: tar: workspace-download\.tar: Cannot open: Permission denied/,
);
await expect(client.run("tar -cf workspace-download.tar .", { timeoutMs: 30_000 }))
.rejects.toMatchObject({ exitCode: 2, timedOut: false });
});
it("test_channel_round_trips_all_byte_values", () => {
@@ -189,7 +189,9 @@ function formatFailedCommandOutput(result: RunProcessResult): string {
function requireSuccessfulResult(result: RunProcessResult, action: string): void {
if (result.exitCode === 0 && !result.timedOut) return;
const detail = formatFailedCommandOutput(result);
throw new Error(`${action} failed with exit code ${result.exitCode ?? "null"}${detail}`);
throw Object.assign(new Error(`${action} failed with exit code ${result.exitCode ?? "null"}${detail}`), {
exitCode: result.exitCode, timedOut: result.timedOut,
});
}
function bufferToArrayBuffer(buffer: Buffer): ArrayBuffer {
@@ -0,0 +1,66 @@
import { randomUUID } from "node:crypto";
import { mkdtemp, readFile, rm, writeFile } from "node:fs/promises";
import os from "node:os";
import path from "node:path";
import { expect, it, vi } from "vitest";
import { fetchGitBundleIntoLocalRef, runLocalGit } from "./git-workspace-sync.js";
const timing = vi.hoisted(() => ({ enabled: false, elapsedProbeMs: 0, probeDurationMs: 14_000, fetchTimeout: 0 }));
vi.mock("node:child_process", async importOriginal => {
const actual = await importOriginal<typeof import("node:child_process")>();
return { ...actual, execFile: (...args: Parameters<typeof actual.execFile>) => {
const gitArgs = args[1] as string[];
if (timing.enabled && gitArgs.includes("fetch")) timing.fetchTimeout = Number((args[2] as { timeout: number }).timeout);
// Keep real Git and its callbacks. Only its observed probe duration is
// controlled, so a fast fixture can exercise a nearly spent probe budget.
if (timing.enabled && gitArgs.some(arg => arg.endsWith("^@"))) {
const callback = args[3] as (...values: unknown[]) => void;
args[3] = ((...values: unknown[]) => { timing.elapsedProbeMs += timing.probeDurationMs; callback(...values); }) as typeof args[3];
}
return actual.execFile(...args);
} };
});
it.each([14_000, 16_000])("separates transfer and probe budgets (probe duration %s)", async probeDurationMs => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-probe-budget-"));
const git = async (cwd: string, args: string[]) => (await runLocalGit(cwd, args)).stdout.trim();
const realNow = Date.now.bind(Date);
try {
const source = path.join(root, "source"), host = path.join(root, "host"), remote = path.join(root, "remote");
await git(root, ["init", source]);
await git(source, ["config", "user.name", "Test"]);
await git(source, ["config", "user.email", "test@example.com"]);
await writeFile(path.join(source, "saved.txt"), "base\n");
await git(source, ["add", "."]); await git(source, ["commit", "-m", "base"]);
await git(root, ["clone", source, host]);
await writeFile(path.join(source, "saved.txt"), "completed\n");
await git(source, ["commit", "-am", "completed"]);
const head = await git(source, ["rev-parse", "HEAD"]);
await git(root, ["clone", "--depth=1", `file://${source}`, remote]);
const bundlePath = path.join(root, "work.bundle"), shallowPath = path.join(remote, ".git", "shallow");
const exportRef = `refs/paperclip/export/${randomUUID()}`, importedRef = `refs/paperclip/import/${randomUUID()}`;
await git(remote, ["update-ref", exportRef, head]);
await git(remote, ["bundle", "create", bundlePath, exportRef]);
vi.spyOn(Date, "now").mockImplementation(() => realNow() + timing.elapsedProbeMs);
timing.enabled = true; timing.probeDurationMs = probeDurationMs;
if (probeDurationMs > 15_000) {
await expect(fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, shallowPath, exportRef, importedRef, baseSha: head }))
.rejects.toThrow("boundary inspection timed out");
expect(timing.fetchTimeout).toBe(0);
timing.enabled = false;
await expect(git(host, ["rev-parse", "--verify", importedRef])).rejects.toThrow();
expect(await readFile(path.join(host, "saved.txt"), "utf8")).toBe("base\n");
return;
}
expect(await fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, shallowPath, exportRef, importedRef, baseSha: head })).toBe(head);
expect(timing.elapsedProbeMs).toBe(14_000);
expect(timing.fetchTimeout).toBeGreaterThan(55_000);
timing.enabled = false;
expect(await git(host, ["show", `${importedRef}:saved.txt`])).toBe("completed");
expect(await git(host, ["fsck", "--connectivity-only"])).not.toContain("missing");
expect(await readFile(path.join(host, "saved.txt"), "utf8")).toBe("base\n");
} finally {
timing.enabled = false; timing.elapsedProbeMs = 0; timing.fetchTimeout = 0;
vi.restoreAllMocks(); await rm(root, { recursive: true, force: true });
}
});
@@ -578,6 +578,84 @@ describe("git workspace sync", () => {
}
});
it.each(["missing", "complete", "shallow"])("imports shallow full bundles without truncating existing host ancestry (host history: %s)", async (hostHistory) => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-shallow-bundle-"));
cleanupDirs.push(root);
const source = await createRepo(root);
await writeFile(path.join(source, "tracked.txt"), "host baseline\n");
await git(source, ["commit", "-am", "host baseline"]);
await git(source, ["branch", "baseline"]);
await writeFile(path.join(source, "tracked.txt"), "published work\n");
await git(source, ["commit", "-am", "published"]);
const published = await git(source, ["rev-parse", "HEAD"]);
const host = path.join(root, "host");
await git(root, hostHistory === "shallow"
? ["clone", "--depth=1", "--branch", "baseline", `file://${source}`, host]
: ["clone", source, host]);
const originalShallow = hostHistory === "shallow" ? await readFile(path.join(host, ".git", "shallow"), "utf8") : null;
if (hostHistory === "missing") {
// A host containing just the original base, never the separately fetched PR.
await git(host, ["reset", "--hard", "HEAD^"]);
await git(host, ["reflog", "expire", "--expire=now", "--all"]);
await git(host, ["update-ref", "-d", "refs/remotes/origin/main"]);
await git(host, ["symbolic-ref", "--delete", "refs/remotes/origin/HEAD"]);
await git(host, ["gc", "--prune=now"]);
await expect(git(host, ["cat-file", "-e", published])).rejects.toThrow();
}
const before = await git(host, ["rev-list", "HEAD"]);
const sandbox = path.join(root, "sandbox");
await git(root, ["clone", "--depth=1", `file://${source}`, sandbox]);
const bundlePath = path.join(root, "full.bundle");
const shallowPath = path.join(root, "shallow");
const exportRef = createRemoteGitExportRef("test");
const importedRef = createImportedGitRef("test");
await execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox, baseSha: published, exportRef, bundlePath, shallowPath, forceFullBundle: true,
})]);
const tracePath = path.join(root, "git-trace.json");
const previousTrace = process.env.GIT_TRACE2_EVENT;
process.env.GIT_TRACE2_EVENT = tracePath;
const gitStarts = async () => (await readFile(tracePath, "utf8")).trim().split("\n")
.filter((line) => JSON.parse(line).event === "start").length;
try {
expect(await fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, exportRef, importedRef,
baseSha: published, shallowPath })).toBe(published);
const uniqueStarts = await gitStarts();
await rm(tracePath);
await writeFile(shallowPath, `${published}\n`.repeat(12));
expect(await fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, exportRef, importedRef,
baseSha: published, shallowPath })).toBe(published);
// Duplicate sender metadata must not multiply host subprocess work.
expect(await gitStarts()).toBeLessThanOrEqual(uniqueStarts + 4);
await rm(tracePath);
await writeFile(shallowPath, `${published}\n`.repeat(2000));
await expect(fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, exportRef, importedRef,
baseSha: published, shallowPath })).rejects.toThrow("byte limit");
await expect(stat(tracePath)).rejects.toMatchObject({ code: "ENOENT" });
await writeFile(shallowPath, Array.from({ length: 257 }, (_, i) =>
(i + 1).toString(16).padStart(40, "0")).join("\n"));
await expect(fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, exportRef, importedRef,
baseSha: published, shallowPath })).rejects.toThrow("boundary limit");
await expect(stat(tracePath)).rejects.toMatchObject({ code: "ENOENT" });
} finally {
if (previousTrace === undefined) delete process.env.GIT_TRACE2_EVENT;
else process.env.GIT_TRACE2_EVENT = previousTrace;
}
expect(await git(host, ["rev-list", "HEAD"])).toBe(before);
if (originalShallow) expect(await readFile(path.join(host, ".git", "shallow"), "utf8")).toBe(originalShallow);
expect(await git(host, ["fsck", "--connectivity-only"])).not.toContain("missing");
await writeFile(shallowPath, `${published}\n../../config\n`);
await expect(fetchGitBundleIntoLocalRef({ localDir: host, bundlePath, exportRef,
importedRef: createImportedGitRef("invalid"), baseSha: published, shallowPath }))
.rejects.toThrow("Invalid Git shallow boundaries");
expect(await git(host, ["rev-list", "HEAD"])).toBe(before);
await writeFile(path.join(sandbox, ".git", "shallow"), `${published}\n`.repeat(2000));
await expect(execFile("sh", ["-c", buildRemoteGitDeltaBundleScript({
remoteDir: sandbox, baseSha: published, exportRef, bundlePath, shallowPath, forceFullBundle: true,
})])).rejects.toThrow("byte limit");
});
it("falls back to a full self-contained bundle when the sandbox lacks baseSha", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-git-full-"));
cleanupDirs.push(rootDir);
@@ -695,22 +695,120 @@ export async function deleteLocalGitRef(input: {
}).catch(() => undefined);
}
export const GIT_SHALLOW_METADATA_EXIT_CODE = 47;
export const GIT_SHALLOW_METADATA_ERROR_CODE = "WORKSPACE_RESTORE_INVALID_GIT_METADATA";
/** Invalid restore metadata is an integrity rejection, not a dropped transfer. */
export class GitShallowMetadataError extends Error {
readonly code = GIT_SHALLOW_METADATA_ERROR_CODE;
}
export const GIT_SHALLOW_MAX_BYTES = 64 * 1024;
export const GIT_SHALLOW_MAX_BOUNDARIES = 256;
export async function fetchGitBundleIntoLocalRef(input: {
localDir: string;
bundlePath: string;
exportRef: string;
importedRef: string;
baseSha: string;
/** Shallow boundaries accompanying a full bundle, which cannot encode them. */
shallowPath?: string;
}): Promise<string> {
const bundleSize = (await fs.stat(input.bundlePath).catch(() => null))?.size ?? 0;
if (bundleSize === 0) {
return input.baseSha;
}
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
const shallowSize = input.shallowPath ? (await fs.stat(input.shallowPath)).size : 0;
if (shallowSize > GIT_SHALLOW_MAX_BYTES) {
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${shallowSize} > ${GIT_SHALLOW_MAX_BYTES})`);
}
const shallow = input.shallowPath ? await fs.readFile(input.shallowPath, "utf8") : "";
const shallowBytes = Buffer.byteLength(shallow);
if (shallowBytes > GIT_SHALLOW_MAX_BYTES)
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${shallowBytes} > ${GIT_SHALLOW_MAX_BYTES})`);
if (shallow.trim()) {
const boundaries = [...new Set(shallow.trim().split("\n"))];
if (boundaries.length > GIT_SHALLOW_MAX_BOUNDARIES) throw new GitShallowMetadataError(`Git shallow metadata exceeds its boundary limit (${boundaries.length} > ${GIT_SHALLOW_MAX_BOUNDARIES})`);
let deadline = Date.now() + 60_000;
const importGit = (cwd: string, args: string[], options: NonNullable<Parameters<typeof runLocalGit>[2]> = {}) => {
const remaining = deadline - Date.now();
if (remaining <= 0) throw new Error("Git shallow history import timed out");
return runLocalGit(cwd, args, { ...options, timeout: Math.min(options.timeout ?? 15_000, remaining) });
};
const objectFormat = (await importGit(input.localDir, ["rev-parse", "--show-object-format"])).stdout.trim();
const oid = objectFormat === "sha256" ? /^[a-f0-9]{64}$/ : /^[a-f0-9]{40}$/;
if (!boundaries.every((boundary) => oid.test(boundary))) throw new GitShallowMetadataError(`Invalid Git shallow boundaries for host object format ${objectFormat}`);
const staging = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-bundle-"));
try {
// Git bundles have no shallow protocol. Unpack into a private repository
// carrying the sender's boundaries, then let ordinary Git fetch negotiate
// which boundaries the host actually needs. Never overwrite host shallow
// metadata or disable connectivity checks.
await importGit(input.localDir, ["clone", "--bare", "--shared", input.localDir, staging], { timeout: 60_000 });
const advertised = await importGit(staging, ["bundle", "unbundle", input.bundlePath], {
timeout: 60_000, maxBuffer: 1024 * 1024,
});
const head = advertised.stdout.split("\n").find((line) => line.endsWith(` ${input.exportRef}`))?.split(" ")[0];
if (!head || !oid.test(head)) throw new Error("Git bundle does not advertise the requested export ref");
const retainedBoundaries = (await fs.readFile(path.join(staging, "shallow"), "utf8").catch((error: NodeJS.ErrnoException) => {
if (error.code !== "ENOENT") throw error;
return "";
})).trim().split("\n").filter(Boolean);
// Boundary inspection has its own small budget. It must not consume the
// transfer budget needed to save a valid large bundle after the probes.
const probeStarted = Date.now(), probeDeadline = probeStarted + 15_000;
const probeGit = (args: string[]) => {
const remaining = probeDeadline - Date.now();
if (remaining <= 0) throw new Error("Git shallow boundary inspection timed out");
return runLocalGit(staging, args, { timeout: remaining });
};
const knownCommits = new Map<string, boolean>();
const hasCommit = async (commit: string): Promise<boolean> => {
if (knownCommits.has(commit)) return knownCommits.get(commit)!;
try {
await probeGit(["rev-parse", "--verify", "--quiet", `${commit}^{commit}`]);
knownCommits.set(commit, true);
return true;
} catch (error) {
if ((error as { code?: unknown }).code === 1) {
knownCommits.set(commit, false);
return false;
}
throw error;
}
};
for (const boundary of boundaries) {
// A shallow sender can stop before history the host already holds.
// Keep that connection when possible; adding its boundary unconditionally
// would hide a real merge-base and reject concurrent host changes.
if (!await hasCommit(boundary)) continue; // An unrelated tip is not in this bundle.
const parents = await probeGit(["rev-parse", `${boundary}^@`]);
let missingParent = false;
for (const parent of parents.stdout.trim().split("\n").filter(Boolean)) {
if (!await hasCommit(parent)) {
missingParent = true;
break;
}
}
if (missingParent) retainedBoundaries.push(boundary);
}
deadline += Date.now() - probeStarted;
if (retainedBoundaries.length) await fs.writeFile(path.join(staging, "shallow"), `${[...new Set(retainedBoundaries)].join("\n")}\n`);
await importGit(staging, ["update-ref", input.exportRef, head]);
await importGit(input.localDir, ["fetch", "--force", "--update-shallow", staging, `${input.exportRef}:${input.importedRef}`], {
timeout: 60_000, maxBuffer: 1024 * 1024,
});
} finally {
await fs.rm(staging, { recursive: true, force: true });
}
} else {
await runLocalGit(input.localDir, ["fetch", "--force", input.bundlePath, `${input.exportRef}:${input.importedRef}`], {
timeout: 60_000,
maxBuffer: 1024 * 1024,
});
}
const importedHead = await runLocalGit(input.localDir, ["rev-parse", input.importedRef], {
timeout: 10_000,
maxBuffer: 16 * 1024,
@@ -750,6 +848,8 @@ export function buildRemoteGitDeltaBundleScript(input: {
* failed because the host lacked the bundle's prerequisite.
*/
forceFullBundle?: boolean;
/** Export the sender's shallow boundaries alongside a full bundle. */
shallowPath?: string;
}): string {
const remoteDir = shellQuote(input.remoteDir);
const bundlePath = shellQuote(input.bundlePath);
@@ -758,6 +858,7 @@ export function buildRemoteGitDeltaBundleScript(input: {
const statusPath = input.statusPath ? shellQuote(input.statusPath) : null;
const cleanupParts = [
`rm -f ${bundlePath}`,
...(input.shallowPath ? [`rm -f ${shellQuote(input.shallowPath)}`] : []),
...(statusPath ? [`rm -f ${statusPath}`] : []),
`git -C ${remoteDir} update-ref -d ${exportRef} >/dev/null 2>&1 || true`,
];
@@ -767,6 +868,13 @@ export function buildRemoteGitDeltaBundleScript(input: {
input.cleanupBundle ? "trap cleanup EXIT" : "",
`mkdir -p ${shellQuote(path.posix.dirname(input.bundlePath))}`,
`rm -f ${bundlePath}`,
...(input.shallowPath ? [
`shallow_file=$(git -C ${remoteDir} rev-parse --path-format=absolute --git-path shallow)`,
`if [ -f "$shallow_file" ]; then`,
` dd if="$shallow_file" of=${shellQuote(input.shallowPath)} bs=${GIT_SHALLOW_MAX_BYTES + 1} count=1 2>/dev/null`,
`else : > ${shellQuote(input.shallowPath)}; fi`,
`[ "$(wc -c < ${shellQuote(input.shallowPath)})" -le ${GIT_SHALLOW_MAX_BYTES} ] || { printf '%s\\n' 'Git shallow metadata exceeds its byte limit' >&2; exit ${GIT_SHALLOW_METADATA_EXIT_CODE}; }`,
] : []),
// Choose the bundle boundary. A thin bundle `HEAD --not <baseSha>` records
// baseSha as a prerequisite the importer (host) must already hold. That
// assumption breaks in two real cases, and then `git fetch` on the host
@@ -932,6 +932,50 @@ describe("sandbox managed runtime", () => {
expect(await git(host, ["diff", "--cached", "--name-only"])).toBe("");
});
it("restores a separately fetched shallow branch whose boundary the host lacks", async () => {
const root = await mkdtemp(path.join(os.tmpdir(), "paperclip-shallow-restore-"));
cleanupDirs.push(root);
const host = path.join(root, "host");
const remote = path.join(root, "remote");
const origin = path.join(root, "origin");
await mkdir(host);
await git(host, ["init"]);
await git(host, ["config", "user.name", "Paperclip Test"]);
await git(host, ["config", "user.email", "test@paperclip.dev"]);
await writeFile(path.join(host, "tracked.txt"), "base\n");
await git(host, ["add", "."]);
await git(host, ["commit", "-m", "base"]);
const baseline = await git(host, ["rev-parse", "HEAD"]);
await git(root, ["clone", host, origin]);
await git(origin, ["config", "user.name", "Paperclip Test"]);
await git(origin, ["config", "user.email", "test@paperclip.dev"]);
for (const content of ["published", "review fix"]) {
await writeFile(path.join(origin, "tracked.txt"), content);
await git(origin, ["commit", "-am", content]);
}
const publishedHead = await git(origin, ["rev-parse", "HEAD"]);
const prepared = await prepareSandboxManagedRuntime({
...sandboxRuntimeInput(sandboxSpec("shallow", remote), "test-adapter", makeFilesystemClient(), host),
workspaceRestoreRetryDelaysMs: [],
});
// Like checking out a PR in a fresh task workspace: only its tip is fetched,
// and the sandbox still has its independent original shallow boundary.
await git(remote, ["fetch", "--depth=1", origin, publishedHead]);
await git(remote, ["checkout", "--detach", "FETCH_HEAD"]);
await writeFile(path.join(remote, "evidence.txt"), "saved result\n");
await writeFile(path.join(host, "local-only.txt"), "concurrent host work\n");
await prepared.restoreWorkspace();
expect(await readFile(path.join(host, "tracked.txt"), "utf8")).toBe("review fix");
expect(await readFile(path.join(host, "evidence.txt"), "utf8")).toBe("saved result\n");
expect(await readFile(path.join(host, "local-only.txt"), "utf8")).toBe("concurrent host work\n");
expect(await git(host, ["cat-file", "-t", publishedHead])).toBe("commit");
// Imported shallow history must not truncate the host's existing history.
expect(await git(host, ["rev-parse", "HEAD^"])).toBe(baseline);
expect(await git(host, ["fsck", "--connectivity-only"])).not.toContain("missing");
});
it("restores a sandbox reset to an ancestor with the matching clean working tree", async () => {
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-sandbox-reset-"));
cleanupDirs.push(rootDir);
@@ -4244,7 +4288,7 @@ describe("sandbox git-bundle export transport", () => {
expect(capture.readFilePaths.some((remotePath) => remotePath.endsWith("git-delta.bundle"))).toBe(true);
});
it("retries the full bundle through the native branch when the delta misses its prerequisite", async () => {
it.each([[false, false, false], [true, false, false], [false, true, false], [false, false, true]])("restores native bundles with concurrent work %s, oversized transfer %s, oversized source %s", async (concurrentHostCommit, oversizedTransfer, oversizedSource) => {
const capture: TransportCapture = { syncOutOperations: [], readFilePaths: [] };
const rootDir = await mkdtemp(path.join(os.tmpdir(), "paperclip-bundle-retry-"));
cleanupDirs.push(rootDir);
@@ -4268,19 +4312,61 @@ describe("sandbox git-bundle export transport", () => {
const stagedBase = await git(localWorkspaceDir, ["rev-parse", "HEAD"]);
const client = makeTransportClient(true, capture);
let shallowProgressCalls = 0, completedShallowRead = false;
if (oversizedTransfer) {
const read = client.readFile.bind(client);
client.readFile = async (remotePath, options) => {
if (remotePath.endsWith("git-shallow.txt")) {
shallowProgressCalls++;
await options?.onProgress?.(0, 64 * 1024 + 1);
completedShallowRead = true;
}
return read(remotePath, options);
};
}
let sourceCapChecks = 0;
if (oversizedSource) {
const run = client.run.bind(client);
client.run = async (command, options) => {
if (command.includes("git-shallow.txt") && command.includes("wc -c")) sourceCapChecks++;
return run(command, options);
};
}
const prepared = await prepareSandboxManagedRuntime(sandboxRuntimeInput(gitSpec(remoteWorkspaceDir), "test-adapter", client, localWorkspaceDir));
await commitInSandbox(remoteWorkspaceDir);
const sandboxHead = await git(remoteWorkspaceDir, ["rev-parse", "HEAD"]);
if (oversizedSource) await writeFile(path.join(remoteWorkspaceDir, ".git", "shallow"), `${sandboxHead}\n`.repeat(2000));
// The host drops below the staged base commit and prunes it. The delta
// bundle names that boundary commit as a prerequisite the host no longer
// holds, so the import fails and forces the full-bundle retry.
await git(localWorkspaceDir, ["reset", "--hard", firstCommit]);
if (concurrentHostCommit) {
await writeFile(path.join(localWorkspaceDir, "host-only.txt"), "concurrent host commit\n");
await git(localWorkspaceDir, ["add", "host-only.txt"]);
await git(localWorkspaceDir, ["commit", "-m", "host work after reset"]);
}
const hostHead = await git(localWorkspaceDir, ["rev-parse", "HEAD"]);
await git(localWorkspaceDir, ["reflog", "expire", "--expire=now", "--all"]);
await git(localWorkspaceDir, ["gc", "--prune=now"]);
await expect(git(localWorkspaceDir, ["cat-file", "-e", `${stagedBase}^{commit}`])).rejects.toThrow();
if (oversizedSource) {
await expect(prepared.restoreWorkspace()).rejects.toMatchObject({ code: "WORKSPACE_RESTORE_INVALID_GIT_METADATA" });
expect(sourceCapChecks).toBe(1);
expect(capture.readFilePaths.some(p => p.endsWith("git-shallow.txt"))).toBe(false);
expect(await git(localWorkspaceDir, ["rev-parse", "HEAD"])).toBe(hostHead);
return;
}
if (oversizedTransfer) {
await expect(prepared.restoreWorkspace()).rejects.toThrow("byte limit");
expect(shallowProgressCalls).toBe(1);
expect(completedShallowRead).toBe(false);
expect(await git(localWorkspaceDir, ["rev-parse", "HEAD"])).toBe(hostHead);
return;
}
await prepared.restoreWorkspace();
// Two bundle exports rode native file mappings: the delta attempt and the
@@ -4293,6 +4379,13 @@ describe("sandbox git-bundle export transport", () => {
// The full bundle was self-contained: the host repository now holds the
// sandbox head commit.
await expect(git(localWorkspaceDir, ["cat-file", "-e", `${sandboxHead}^{commit}`])).resolves.toBe("");
if (concurrentHostCommit) {
expect(await readFile(path.join(localWorkspaceDir, "host-only.txt"), "utf8")).toBe("concurrent host commit\n");
expect(await readFile(path.join(localWorkspaceDir, "remote-only.txt"), "utf8")).toBe("from sandbox\n");
expect((await git(localWorkspaceDir, ["rev-list", "--parents", "-1", "HEAD"])).split(" ").slice(1))
.toEqual([hostHead, sandboxHead]);
expect(await git(localWorkspaceDir, ["fsck", "--connectivity-only"])).not.toContain("missing");
}
});
it("reports the real transferred bytes for the native git-history export and workspace restore", async () => {
@@ -14,6 +14,9 @@ import path from "node:path";
import { promisify } from "node:util";
import {
buildRemoteGitDeltaBundleScript,
GIT_SHALLOW_METADATA_EXIT_CODE,
GIT_SHALLOW_MAX_BYTES,
GitShallowMetadataError,
isMissingGitPrerequisiteError,
createImportedGitRef,
createRemoteGitExportRef,
@@ -2052,6 +2055,8 @@ export async function prepareSandboxManagedRuntime(input: {
const remoteWorkspaceStatusPath = path.posix.join(runtimeRootDir, "workspace-status.txt");
const exportRef = createRemoteGitExportRef("sandbox");
const localBundlePath = path.join(tempDir, "git-delta.bundle");
const remoteShallowPath = path.posix.join(runtimeRootDir, "git-shallow.txt");
const localShallowPath = path.join(tempDir, "git-shallow.txt");
// Export the sandbox history and import it into the host workspace.
// The delta bundle assumes the host holds the bundle's boundary
@@ -2060,17 +2065,28 @@ export async function prepareSandboxManagedRuntime(input: {
// a full, self-contained bundle from the still-live sandbox rather
// than discard the completed run.
const exportAndImport = async (forceFullBundle: boolean): Promise<string> => withWorkspaceRestoreStep("git_export", async () => {
await input.client.run(
`sh -c ${shellQuote(buildRemoteGitDeltaBundleScript({
remoteDir: workspaceRemoteDir,
baseSha: gitSnapshot.headCommit,
exportRef,
bundlePath: remoteGitBundle,
statusPath: forceFullBundle ? undefined : remoteWorkspaceStatusPath,
forceFullBundle,
}))}`,
{ timeoutMs: input.spec.timeoutMs },
);
try {
await input.client.run(
`sh -c ${shellQuote(buildRemoteGitDeltaBundleScript({
remoteDir: workspaceRemoteDir,
baseSha: gitSnapshot.headCommit,
exportRef,
bundlePath: remoteGitBundle,
statusPath: forceFullBundle ? undefined : remoteWorkspaceStatusPath,
forceFullBundle,
shallowPath: forceFullBundle ? remoteShallowPath : undefined,
}))}`,
{ timeoutMs: input.spec.timeoutMs },
);
} catch (error) {
// Restore the owned producer refusal across clients that
// expose process status rather than the typed host error.
const status = error as { exitCode?: unknown; code?: unknown; timedOut?: unknown } | null;
if (status?.timedOut !== true && (status?.exitCode === GIT_SHALLOW_METADATA_EXIT_CODE
|| status?.code === GIT_SHALLOW_METADATA_EXIT_CODE))
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (> ${GIT_SHALLOW_MAX_BYTES})`);
throw error;
}
const gitExport = makeTransferProgress(
restoreSink,
"Exporting git history",
@@ -2110,6 +2126,18 @@ export async function prepareSandboxManagedRuntime(input: {
await fs.writeFile(localBundlePath, bundleBuffer);
}
await input.client.remove(remoteGitBundle).catch(() => undefined);
if (forceFullBundle) {
const shallow = toBuffer(await input.client.readFile(remoteShallowPath, {
onProgress: (transferred, total) => {
if (transferred > GIT_SHALLOW_MAX_BYTES || (total !== null && total > GIT_SHALLOW_MAX_BYTES)) {
throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${Math.max(transferred, total ?? 0)} > ${GIT_SHALLOW_MAX_BYTES})`);
}
},
}));
if (shallow.byteLength > GIT_SHALLOW_MAX_BYTES) throw new GitShallowMetadataError(`Git shallow metadata exceeds its byte limit (${shallow.byteLength} > ${GIT_SHALLOW_MAX_BYTES})`);
await fs.writeFile(localShallowPath, shallow);
await input.client.remove(remoteShallowPath).catch(() => undefined);
}
if (!forceFullBundle) {
remoteWorkspaceStatus = await input.client.readFile(remoteWorkspaceStatusPath)
.then((bytes) => toBuffer(bytes).toString("utf8").trim())
@@ -2123,6 +2151,7 @@ export async function prepareSandboxManagedRuntime(input: {
exportRef,
importedRef: importedRef!,
baseSha: gitSnapshot.headCommit,
shallowPath: forceFullBundle ? localShallowPath : undefined,
}));
});
@@ -6,6 +6,7 @@ import os from "node:os";
import path from "node:path";
import { afterEach, describe, expect, it, vi } from "vitest";
import { GitShallowMetadataError } from "./git-workspace-sync.js";
import { resolvePaperclipInstanceRootForAdapter } from "./server-utils.js";
import {
captureDirectorySnapshot,
@@ -197,6 +198,12 @@ describe("workspace restore merge", () => {
expect(describeWorkspaceRestoreFailure(classifyWorkspaceRestoreFailure(new Error(message)))).not.toContain("private");
});
it("keeps invalid metadata distinct from an omittable unsafe archive", () => {
const error = new GitShallowMetadataError("Invalid Git shallow boundaries /private/source");
expect(classifyWorkspaceRestoreFailure(error)).toBe("restore_invalid_metadata");
expect(describeWorkspaceRestoreFailure(classifyWorkspaceRestoreFailure(error))).not.toContain("private");
});
it("preserves the generic policy for other outbound command failures", () => {
expect(classifyWorkspaceRestoreFailure(new Error("Daytona outbound symlink-escape guard command failed (exit 1): transport failed"))).toBe("restore_failed");
});
@@ -308,6 +308,7 @@ export type WorkspaceRestoreFailureCode =
| "restore_permission_denied"
| "restore_lock_timeout"
| "restore_unsafe_archive"
| "restore_invalid_metadata"
| "restore_failed";
/**
@@ -330,6 +331,7 @@ export type WorkspaceRestoreOutcome =
*/
export function classifyWorkspaceRestoreFailure(error: unknown): WorkspaceRestoreFailureCode {
const code = error && typeof error === "object" ? (error as NodeJS.ErrnoException).code : undefined;
if (code === "WORKSPACE_RESTORE_INVALID_GIT_METADATA") return "restore_invalid_metadata";
if (code === "EACCES" || code === "EPERM") return "restore_permission_denied";
if (code === WORKSPACE_RESTORE_LOCK_TIMEOUT_CODE) return "restore_lock_timeout";
const message = error instanceof Error ? error.message : "";
@@ -361,6 +363,8 @@ export function describeWorkspaceRestoreFailure(code: WorkspaceRestoreFailureCod
return "the restore timed out waiting for the workspace merge lock";
case "restore_unsafe_archive":
return "the archive contains an unsafe link or path; workspace repair is required";
case "restore_invalid_metadata":
return "the restore metadata is invalid; workspace repair is required";
case "restore_failed":
return "the restore failed";
}
@@ -0,0 +1,6 @@
DROP INDEX "heartbeat_runs_cancelled_workspace_preservation_idx";--> statement-breakpoint
CREATE INDEX "heartbeat_runs_cancelled_workspace_preservation_idx" ON "heartbeat_runs" USING btree ("id","company_id") WHERE "heartbeat_runs"."runtime_mode" = 'native' AND "heartbeat_runs"."status" = 'cancelled'
AND "heartbeat_runs"."runner_profile_json"->'nativeWorkspaceSync'->>'state' = 'prepared'
AND "heartbeat_runs"."result_json"->'nativeCancellation'->>'dispatchState' = 'acknowledged'
AND ("heartbeat_runs"."result_json"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
AND "heartbeat_runs"."result_json"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE;
@@ -1,6 +1,6 @@
{
"id": "ff56ee93-8cfc-465a-b26a-1c12d10773ff",
"prevId": "c02426e4-de1a-4ce3-a541-99bb970c272a",
"id": "bbaeecd8-ff3f-4e70-a069-291bf3dc97bb",
"prevId": "dce41524-29a3-4552-ba5d-f0abefbf1ea1",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -1351,6 +1351,39 @@
"method": "btree",
"with": {}
},
"activity_log_company_entity_created_idx": {
"name": "activity_log_company_entity_created_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "entity_type",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "entity_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "created_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"activity_log_company_user_issue_actor_idx": {
"name": "activity_log_company_user_issue_actor_idx",
"columns": [
@@ -19046,7 +19079,46 @@
"name": "file_inventory",
"type": "jsonb",
"primaryKey": false,
"notNull": true
"notNull": true,
"default": "'[]'::jsonb"
},
"storage": {
"name": "storage",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"upload": {
"name": "upload",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"restore": {
"name": "restore",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"migration_at": {
"name": "migration_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false,
"default": "now()"
},
"maintenance_at": {
"name": "maintenance_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false,
"default": "now()"
},
"restore_retry_at": {
"name": "restore_retry_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
@@ -19077,6 +19149,38 @@
"concurrently": false,
"method": "btree",
"with": {}
},
"company_skill_snapshots_migration_idx": {
"name": "company_skill_snapshots_migration_idx",
"columns": [
{
"expression": "migration_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"company_skill_repository_snapshots\".\"migration_at\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
},
"company_skill_snapshots_maintenance_idx": {
"name": "company_skill_snapshots_maintenance_idx",
"columns": [
{
"expression": "maintenance_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"company_skill_repository_snapshots\".\"maintenance_at\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
@@ -19966,6 +20070,25 @@
"primaryKey": false,
"notNull": false
},
"package_snapshot_id": {
"name": "package_snapshot_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"package_file_map": {
"name": "package_file_map",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"migration_at": {
"name": "migration_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false,
"default": "now()"
},
"revision_number": {
"name": "revision_number",
"type": "integer",
@@ -20024,6 +20147,52 @@
}
},
"indexes": {
"company_skill_versions_migration_idx": {
"name": "company_skill_versions_migration_idx",
"columns": [
{
"expression": "migration_at",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"company_skill_versions\".\"migration_at\" is not null",
"concurrently": false,
"method": "btree",
"with": {}
},
"company_skill_versions_package_snapshot_idx": {
"name": "company_skill_versions_package_snapshot_idx",
"columns": [
{
"expression": "package_snapshot_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"company_skill_versions_repository_snapshot_idx": {
"name": "company_skill_versions_repository_snapshot_idx",
"columns": [
{
"expression": "repository_snapshot_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"company_skill_versions_skill_revision_idx": {
"name": "company_skill_versions_skill_revision_idx",
"columns": [
@@ -20135,6 +20304,19 @@
"onDelete": "restrict",
"onUpdate": "no action"
},
"company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk": {
"name": "company_skill_versions_package_snapshot_id_company_skill_repository_snapshots_id_fk",
"tableFrom": "company_skill_versions",
"tableTo": "company_skill_repository_snapshots",
"columnsFrom": [
"package_snapshot_id"
],
"columnsTo": [
"id"
],
"onDelete": "restrict",
"onUpdate": "no action"
},
"company_skill_versions_author_agent_id_agents_id_fk": {
"name": "company_skill_versions_author_agent_id_agents_id_fk",
"tableFrom": "company_skill_versions",
@@ -20580,6 +20762,78 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.skill_package_deletions": {
"name": "skill_package_deletions",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"provider": {
"name": "provider",
"type": "text",
"primaryKey": false,
"notNull": true
},
"archive_key": {
"name": "archive_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"manifest_key": {
"name": "manifest_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"delete_after": {
"name": "delete_after",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true
}
},
"indexes": {
"skill_package_deletions_due_idx": {
"name": "skill_package_deletions_due_idx",
"columns": [
{
"expression": "delete_after",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.company_transfer_runs": {
"name": "company_transfer_runs",
"schema": "",
@@ -33771,6 +34025,34 @@
}
},
"indexes": {
"heartbeat_runs_native_unpresented_response_idx": {
"name": "heartbeat_runs_native_unpresented_response_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "native_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" in ('succeeded', 'failed', 'timed_out') AND \"heartbeat_runs\".\"finished_at\" is not null AND jsonb_typeof(\"heartbeat_runs\".\"result_json\" #> '{presentationDecision,commentId}') is distinct from 'string'",
"concurrently": false,
"method": "btree",
"with": {}
},
"heartbeat_runs_cancelled_workspace_preservation_idx": {
"name": "heartbeat_runs_cancelled_workspace_preservation_idx",
"columns": [
@@ -33788,7 +34070,7 @@
}
],
"isUnique": false,
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE",
"where": "\"heartbeat_runs\".\"runtime_mode\" = 'native' AND \"heartbeat_runs\".\"status\" = 'cancelled'\n AND \"heartbeat_runs\".\"runner_profile_json\"->'nativeWorkspaceSync'->>'state' = 'prepared'\n AND \"heartbeat_runs\".\"result_json\"->'nativeCancellation'->>'dispatchState' = 'acknowledged'\n AND (\"heartbeat_runs\".\"result_json\"->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'\n AND \"heartbeat_runs\".\"result_json\"->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_invalid_metadata', 'restore_failed')) IS NOT TRUE",
"concurrently": false,
"method": "btree",
"with": {}
@@ -36475,6 +36757,40 @@
"method": "btree",
"with": {}
},
"issue_comments_review_response_receipt_idx": {
"name": "issue_comments_review_response_receipt_idx",
"columns": [
{
"expression": "created_by_run_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "author_agent_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"issue_comments\".\"author_type\" = 'agent' AND \"issue_comments\".\"metadata\"->>'authorizationReason' = 'allow_chat_run_presentation'",
"concurrently": false,
"method": "btree",
"with": {}
},
"issue_comments_company_idx": {
"name": "issue_comments_company_idx",
"columns": [
+7
View File
@@ -2899,6 +2899,13 @@
"when": 1793289638018,
"tag": "0415_zippy_zemo",
"breakpoints": true
},
{
"idx": 416,
"version": "7",
"when": 1793320102041,
"tag": "0416_salty_domino",
"breakpoints": true
}
]
}
+10 -1
View File
@@ -1,3 +1,4 @@
import { WORKSPACE_RESTORE_FAILURE_CODES } from "@paperclipai/shared";
import { sql } from "drizzle-orm";
import {
type AnyPgColumn,
@@ -19,6 +20,14 @@ import { agents } from "./agents.js";
import { agentWakeupRequests } from "./agent_wakeup_requests.js";
import { issues } from "./issues.js";
// Literal allowlist shared by bounded projections and the matching partial index.
// Constants are quoted here so PostgreSQL can prove the index predicate even
// when callers use prepared statements.
export const workspaceRestoreFailureCodesSql = sql.join(
WORKSPACE_RESTORE_FAILURE_CODES.map(code => sql.raw(`'${code.replaceAll("'", "''")}'`)),
sql`, `,
);
export const heartbeatRuns = pgTable(
"heartbeat_runs",
{
@@ -119,7 +128,7 @@ export const heartbeatRuns = pgTable(
AND ${table.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared'
AND ${table.resultJson}->'nativeCancellation'->>'dispatchState' = 'acknowledged'
AND (${table.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
AND ${table.resultJson}->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE`),
AND ${table.resultJson}->>'workspaceRestoreFailure' IN (${workspaceRestoreFailureCodesSql})) IS NOT TRUE`),
retainedRecoveryAdoptionIdx: index("heartbeat_runs_retained_recovery_adoption_idx")
.on(table.id).where(sql`${table.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
and not (coalesce(${table.resultJson}, '{}'::jsonb) ? 'systemRecoveryAdoption')
+1 -1
View File
@@ -137,7 +137,7 @@ export { routineDocuments } from "./routine_documents.js";
export { documentAnnotationThreads } from "./document_annotation_threads.js";
export { documentAnnotationComments } from "./document_annotation_comments.js";
export { documentAnnotationAnchorSnapshots } from "./document_annotation_anchor_snapshots.js";
export { heartbeatRuns } from "./heartbeat_runs.js";
export { heartbeatRuns, workspaceRestoreFailureCodesSql } from "./heartbeat_runs.js";
export { heartbeatRunEvents } from "./heartbeat_run_events.js";
export { providerTraceRecords } from "./provider_trace_records.js";
export { completionContracts } from "./completion_contracts.js";
+1
View File
@@ -3,6 +3,7 @@ export const WORKSPACE_RESTORE_FAILURE_CODES = [
"restore_permission_denied",
"restore_lock_timeout",
"restore_unsafe_archive",
"restore_invalid_metadata",
"restore_failed",
] as const;
+2 -1
View File
@@ -13,6 +13,7 @@ import {
environments,
heartbeatRunEvents,
heartbeatRuns,
workspaceRestoreFailureCodesSql,
issueComments,
issueDocuments,
issues,
@@ -109,7 +110,7 @@ export function activityService(db: Db) {
)
) end,
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
in ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')
in (${workspaceRestoreFailureCodesSql})
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
'workspaceRestorePath', case when length(${heartbeatRuns.resultJson} ->> 'workspaceRestorePath') <= 180
then ${heartbeatRuns.resultJson} -> 'workspaceRestorePath' end,
+2 -1
View File
@@ -26,6 +26,7 @@ import {
agentTaskSessions,
heartbeatRunEvents,
heartbeatRuns,
workspaceRestoreFailureCodesSql,
issues,
} from "@paperclipai/db";
import { notFound } from "../../errors.js";
@@ -238,7 +239,7 @@ const heartbeatRunSafeResultJsonColumn = sql<Record<string, unknown> | null>`
'workspaceRestoreRecovery', case when ${heartbeatRuns.resultJson} #>> '{workspaceRestoreRecovery,schema}' = 'paperclip.workspace-restore-recovery.v1'
then jsonb_build_object('schema', 'paperclip.workspace-restore-recovery.v1') end,
'workspaceRestoreFailure', case when ${heartbeatRuns.resultJson} ->> 'workspaceRestoreFailure'
in ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')
in (${workspaceRestoreFailureCodesSql})
then ${heartbeatRuns.resultJson} -> 'workspaceRestoreFailure' end,
'cancellation', case when jsonb_typeof(${heartbeatRuns.resultJson} -> 'cancellation') = 'object'
then jsonb_strip_nulls(jsonb_build_object(
@@ -3,7 +3,7 @@ import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator
import { registerScopedDeadlineWork } from "../scoped-deadline-work.js";
import { notifyExecutionRunWork } from "../execution-control-notifications.js";
import { and, asc, eq, gt, inArray, isNull, or, sql } from "drizzle-orm";
import { environmentLeases, heartbeatRuns, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { environmentLeases, heartbeatRuns, workspaceRestoreFailureCodesSql, issues, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { recordRequiredWorkspaceRestoreFailure } from "../legacy-execution-recovery.js";
import { hasRequiredWorkspaceRecovery, nativeRunHasNoWorkspaceAccess } from "../workspace-restore-recovery-state.js";
import { logActivity } from "../activity-log.js";
@@ -165,7 +165,7 @@ function pendingCancelledWorkspace() {
AND ${heartbeatRuns.runnerProfileJson}->'nativeWorkspaceSync'->>'state' = 'prepared'
AND ${heartbeatRuns.resultJson}->'nativeCancellation'->>'dispatchState' = 'acknowledged'
AND (${heartbeatRuns.resultJson}->'workspaceRestoreRecovery'->>'schema' = 'paperclip.workspace-restore-recovery.v1'
AND ${heartbeatRuns.resultJson}->>'workspaceRestoreFailure' IN ('restore_permission_denied', 'restore_lock_timeout', 'restore_unsafe_archive', 'restore_failed')) IS NOT TRUE`;
AND ${heartbeatRuns.resultJson}->>'workspaceRestoreFailure' IN (${workspaceRestoreFailureCodesSql})) IS NOT TRUE`;
}
export function registerCancelledWorkspacePreservationWork(db: Db,
@@ -1,4 +1,5 @@
import { describe, expect, it, vi, beforeEach } from "vitest";
import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync";
import type { Db } from "@paperclipai/db";
const log = vi.hoisted(() => vi.fn());
vi.mock("../heartbeat-run-events.js", () => ({ appendHeartbeatRunEvent: log }));
@@ -33,6 +34,15 @@ describe("best effort native workspace export", () => {
expect(JSON.stringify(log.mock.calls[0][1])).not.toContain("private");
});
it("keeps rejected Git metadata behind the workspace persistence barrier", async () => {
const { db, select } = fixture();
const error = new GitShallowMetadataError("Invalid Git shallow boundaries for host object format sha1");
await expect(restoreNativeWorkspaceBestEffort({ db, runId: "run", restore: async () => { throw error; } }))
.rejects.toBe(error);
expect(select).not.toHaveBeenCalled();
expect(log).not.toHaveBeenCalled();
});
it("does not hide transport failures or ownership loss", async () => {
const { db, select } = fixture();
const restore = async () => { throw new Error("transport unavailable"); };
@@ -14,6 +14,7 @@ export function classifyNativeWorkspaceFailure(error: unknown): {
const message = error instanceof Error ? error.message.trim() : "";
const code: NativeWorkspaceFailureCode =
message === "workspace_sync_out_unrecoverable" || message.includes("daytona_sandbox_not_found")
|| classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
? "workspace_sync_out_unrecoverable"
: message === "workspace_sync_out_unsafe_archive" || classifyWorkspaceRestoreFailure(error) === "restore_unsafe_archive"
? "workspace_sync_out_unsafe_archive"
@@ -1,4 +1,5 @@
import { beforeEach, describe, expect, it, vi } from "vitest";
import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync";
import type { Db } from "@paperclipai/db";
const sync = vi.hoisted(() => ({ resume: vi.fn() }));
@@ -56,6 +57,13 @@ describe("native workspace finalization without a manual export repair", () => {
expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: `${code}\n` });
});
it("fails copyback for invalid Git metadata instead of successfully omitting saved files", async () => {
sync.resume.mockRejectedValueOnce(new GitShallowMetadataError("Git shallow metadata exceeds its byte limit"));
const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(), runId: "run", environmentRuntime: {} as never });
expect(result).toMatchObject({ status: "failed", exitCode: 1, stderr: "workspace_sync_out_unrecoverable\n" });
expect(sync.resume).toHaveBeenCalledOnce();
});
it("does not treat a missing workspace reference as an omitted unsafe archive", async () => {
sync.resume.mockResolvedValueOnce(false);
const result = await resumeNativeWorkspaceFinalization({ db: fixtureDb(), runId: "run", environmentRuntime: {} as never });
@@ -1,3 +1,4 @@
import { classifyNativeWorkspaceFailure } from "./native-workspace-failure.js";
import { recordNativeFinalizationFailure } from "./native-run-finalizer.js";
import { inspectPluginWorkspaceForFinalization } from "./native-plugin-workspace-finalization.js";
import type { WorkspaceWorkers } from "../plugin-workspaces.js";
@@ -237,12 +238,8 @@ export async function attemptNativeWorkspaceFinalization(input: {
};
} catch (error) {
await ownership.assertHeld();
const code =
error instanceof Error &&
(error.message === "workspace_sync_out_unrecoverable" ||
error.message.includes("daytona_sandbox_not_found"))
? "workspace_sync_out_unrecoverable"
: "workspace_sync_out_failed";
const code = classifyNativeWorkspaceFailure(error).code === "workspace_sync_out_unrecoverable"
? "workspace_sync_out_unrecoverable" : "workspace_sync_out_failed";
return workspaceSyncFailure(code);
}
}
@@ -80,20 +80,23 @@ export async function prepareSandboxStopAndRetain(db: Db, lease: Lease) {
}
/** Historical release rows can lack a stop receipt. Re-inspect the exact
* recorded allocation using stop-only dispatch; never infer that it stopped. */
* recorded allocation using stop-only dispatch; never infer that it stopped.
* The preservation owner verifies source eligibility and exclusive allocation
* ownership before calling. Lease reuse policy does not authorize destruction. */
export async function prepareHistoricalSandboxStopVerification(db: Db, lease: typeof environmentLeases.$inferSelect) {
if (lease.status !== "released" || !lease.heartbeatRunId || !lease.providerLeaseId ||
!["retain_on_failure", "ephemeral"].includes(lease.leasePolicy) || lease.metadata?.pendingCleanupInFlight === true)
lease.metadata?.pendingCleanupInFlight === true)
return null;
return prepareStopAndRetain(db, lease, "released");
}
/** A repair-only resume invalidates the old stop receipt atomically. This is
* separate from ordinary release; a released timestamp alone cannot grant it. */
* separate from ordinary release; a released timestamp alone cannot grant it.
* Source eligibility belongs to the preservation owner, including accepted
* native exports whose original allocation policy is reusable. */
export async function prepareStoppedSandboxForRecovery(db: Db, lease: typeof environmentLeases.$inferSelect) {
if (!hasRemoteTerminationReceipt(lease) || lease.status !== "released"
|| (lease.metadata?.remoteExecutionTermination as { state?: string })?.state !== "stopped"
|| !["retain_on_failure", "ephemeral"].includes(lease.leasePolicy)) return null;
|| (lease.metadata?.remoteExecutionTermination as { state?: string })?.state !== "stopped") return null;
return prepareStopAndRetain(db, lease, "released");
}
@@ -1,10 +1,13 @@
import { randomUUID } from "node:crypto";
import { WORKSPACE_RESTORE_FAILURE_CODES } from "@paperclipai/shared";
import { createHeartbeatRunState } from "./heartbeat/run-state.js";
import { activityService } from "./activity.js";
import { randomBytes, randomUUID } from "node:crypto";
import { afterAll, beforeAll, describe, expect, it, vi } from "vitest";
import express from "express";
import request from "supertest";
import { eq, sql } from "drizzle-orm";
import { activityLog, agents, agentWakeupRequests, applyPendingMigrations, companies, createDb, environmentLeases, environments,
heartbeatRuns, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
heartbeatRuns, issueRecoveryActions, issues, workspaceOperations, completionContracts, nativeRunResults, nativeRunFinalizations } from "@paperclipai/db";
import { startEmbeddedPostgresTestDatabase } from "../__tests__/helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { inspectWorkspaceRecoverySource } from "./workspace-recovery-source.js";
@@ -76,6 +79,38 @@ describe("workspace source verification diagnostics", () => {
};
}
it.each(WORKSPACE_RESTORE_FAILURE_CODES)("preserves %s through both run API projections", async code => {
const f = await seed();
await db.update(heartbeatRuns).set({ resultJson: { workspaceRestoreFailure: code, opaque: randomBytes(100_000).toString("hex"),
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] } } })
.where(eq(heartbeatRuns.id, f.runId));
const [runs, activity] = await Promise.all([
createHeartbeatRunState(db).getRun(f.runId),
activityService(db).runsForIssue(f.companyId, f.issueId),
]);
expect(runs?.resultJson?.opaque).toBeUndefined();
const activityResult = activity[0].resultJson;
expect({ run: runs?.resultJson?.workspaceRestoreFailure,
activity: activityResult && "workspaceRestoreFailure" in activityResult ? activityResult.workspaceRestoreFailure : undefined })
.toEqual({ run: code, activity: code });
});
it("keeps a cancelled metadata-repair hold out of the preservation index while retaining ordinary work", async () => {
const f = await seed();
const [index] = await db.execute(sql`select pg_get_expr(indpred, indrelid) as predicate
from pg_index where indexrelid = 'heartbeat_runs_cancelled_workspace_preservation_idx'::regclass`);
await db.update(heartbeatRuns).set({ status: "cancelled", resultJson: {
nativeCancellation: { dispatchState: "acknowledged" }, workspaceRestoreFailure: "restore_invalid_metadata",
workspaceRestoreRecovery: { schema: "paperclip.workspace-restore-recovery.v1", leaseIds: [f.lease.id] },
} }).where(eq(heartbeatRuns.id, f.runId));
const candidates = () => db.execute(sql`select id from heartbeat_runs where id = ${f.runId}
and ${sql.raw(String(index.predicate))}`);
expect(await candidates()).toHaveLength(0);
await db.update(heartbeatRuns).set({ resultJson: { nativeCancellation: { dispatchState: "acknowledged" } } })
.where(eq(heartbeatRuns.id, f.runId));
expect(await candidates()).toHaveLength(1);
});
it.each(["lease_policy_not_retained", "source_lease_not_bound"])("identifies the rejected contract: %s", async reason => {
const f = await seed();
if (reason === "lease_policy_not_retained") {
@@ -91,6 +126,34 @@ describe("workspace source verification diagnostics", () => {
});
});
it.each(["rejected", "failed_result", "unselected", "shared_allocation", "active_writer"])("keeps reusable export guards: %s", async kind => {
const f = await seed();
const contractId = randomUUID(), resultId = randomUUID();
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard",
completionAuthority: "server_arbiter", incompleteCriteriaPolicy: "preserve_non_terminal",
contractJson: {}, canonicalSha256: "contract", createdByActorType: "system", createdByActorId: "test" });
await db.update(heartbeatRuns).set({ completionContractId: contractId }).where(eq(heartbeatRuns.id, f.runId));
await db.insert(nativeRunResults).values({ id: resultId, companyId: f.companyId, issueId: f.issueId, runId: f.runId,
completionContractId: contractId, schemaStatus: kind === "rejected" ? "rejected" : "accepted",
serverFingerprint: "result", canonicalSha256: "result", resultJson: {
terminal: { runTerminalState: kind === "failed_result" ? "failed" : "succeeded" } } });
await db.insert(nativeRunFinalizations).values({ companyId: f.companyId, issueId: f.issueId, runId: f.runId,
phase: "terminal_failure", resultId: kind === "unselected" ? null : resultId });
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
if (kind === "shared_allocation") await db.insert(environmentLeases).values({ companyId: f.companyId,
environmentId: f.environment.id, provider: "daytona", providerLeaseId: f.lease.providerLeaseId,
leasePolicy: "reuse_by_environment", status: "released" });
if (kind === "active_writer") await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: f.agentId,
issueId: f.issueId, status: "running" });
const before = await snapshot(f);
await expect(inspectWorkspaceRecoverySource(db, f)).rejects.toMatchObject({
code: kind === "shared_allocation" ? "allocation_ownership_ambiguous"
: kind === "active_writer" ? "workspace_writer_active" : "source_binding_invalid",
});
expect(await snapshot(f)).toEqual(before);
});
it("inspects a resolved retained source through HTTP without revalidating or changing recorded recovery", async () => {
const f = await seed();
await db.update(environmentLeases).set({ leasePolicy: "reuse_by_environment" }).where(eq(environmentLeases.id, f.lease.id));
@@ -1,5 +1,5 @@
import { and, eq, inArray, ne, or, sql } from "drizzle-orm";
import { environmentLeases, environments, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, type Db } from "@paperclipai/db";
import { environmentLeases, environments, heartbeatRuns, issueRecoveryActions, nativeRunFinalizations, nativeRunResults, type Db } from "@paperclipai/db";
import { adapterExecutionControls } from "./adapter-execution-control.js";
import { readNativeWorkspaceSyncReference } from "./native-runtime/native-workspace-sync.js";
import { SystemRecoveryFailure } from "./system-recovery-state.js";
@@ -98,6 +98,19 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
if (coordinator?.leaseOwner || adapterExecutionControls.has(run.id) ||
(run.controllerLeaseExpiresAt && run.controllerLeaseExpiresAt > new Date()))
throw new SystemRecoveryFailure("agent_stop_unconfirmed", "recheck_evidence");
// Reusable allocations can still owe copyback for a completed native result.
// This authorizes only preservation of the exact descriptor, never reuse for
// agent execution. Stop verification and physical ownership checks remain below.
const acceptedExportResult = reference && workspaceExportCauses.includes(action.cause)
&& run.runtimeMode === "native" && coordinator?.issueId === input.issueId
&& coordinator.resultId && run.completionContractId
? (await db.select({ id: nativeRunResults.id }).from(nativeRunResults).where(and(
eq(nativeRunResults.companyId, input.companyId), eq(nativeRunResults.issueId, input.issueId),
eq(nativeRunResults.runId, run.id), eq(nativeRunResults.id, coordinator.resultId),
eq(nativeRunResults.completionContractId, run.completionContractId),
eq(nativeRunResults.schemaStatus, "accepted"),
sql`${nativeRunResults.resultJson}#>>'{terminal,runTerminalState}' = 'succeeded'`,
)).limit(1))[0] : null;
const sources = [];
for (const leaseId of leaseIds as string[]) {
const [lease] = await db.select().from(environmentLeases).where(and(eq(environmentLeases.companyId, input.companyId),
@@ -109,7 +122,10 @@ export async function inspectWorkspaceRecoverySource(db: Db, input: WorkspaceRec
if (!lease.environmentId) throw invalid("lease_environment_missing");
if (!lease.providerLeaseId) throw invalid("lease_allocation_missing");
if (!lease.provider || lease.provider === "local") throw invalid("lease_provider_invalid");
if (!["retain_on_failure", "ephemeral"].includes(lease.leasePolicy)) throw invalid("lease_policy_not_retained", lease.leasePolicy);
const reusableExport = descriptor && acceptedExportResult
&& ["reuse_by_environment", "reuse_by_execution_workspace"].includes(lease.leasePolicy);
if (!["retain_on_failure", "ephemeral"].includes(lease.leasePolicy) && !reusableExport)
throw invalid("lease_policy_not_retained", lease.leasePolicy);
if (typeof remoteCwd !== "string" || !remoteCwd.startsWith("/")) throw invalid("lease_cwd_invalid");
if (descriptor && lease.providerLeaseId !== descriptor.providerLeaseId) throw invalid("allocation_reference_mismatch");
if (descriptor && remoteCwd !== descriptor.remoteCwd) throw invalid("cwd_reference_mismatch");
+38 -2
View File
@@ -9,6 +9,8 @@ import { executionWorkspaceService } from "./execution-workspaces.js";
import { workspaceOperationService } from "./workspace-operations.js";
import { instanceSettingsService } from "./instance-settings.js";
import { projectTaskInterruption } from "./task-interruption-presentation.js";
import * as nativeWorkspaceSync from "./native-runtime/native-workspace-sync.js";
import { GitShallowMetadataError } from "@paperclipai/adapter-utils/git-workspace-sync";
import { readSystemRecovery } from "./system-recovery-state.js";
import { adoptSystemRecovery } from "./system-recovery.js";
import { getStorageService } from "../storage/index.js";
@@ -496,8 +498,11 @@ describe("automatic retained workspace recovery", () => {
expect((await Promise.all(observed)).some(rows => rows.some(row => row.status === "succeeded"))).toBe(true);
});
it("re-enters only the existing finalizer for an accepted result, once per intent, preserving failure history", async () => {
const f = await seed();
it.each(["retain_on_failure", "reuse_by_environment", "reuse_by_execution_workspace"] as const)("re-enters only the accepted result finalizer for %s, preserving files and failure history", async leasePolicy => {
const f = await seed("native", true, "keep_running");
await db.update(environmentLeases).set({ leasePolicy }).where(eq(environmentLeases.id, f.lease.id));
await db.update(issueRecoveryActions).set({ cause: "native_workspace_sync_out_retry_exhausted" }).where(eq(issueRecoveryActions.id, f.actionId));
await db.update(heartbeatRuns).set({ resultJson: {} }).where(eq(heartbeatRuns.id, f.runId));
const contractId = randomUUID(), resultId = randomUUID();
await db.insert(completionContracts).values({ id: contractId, companyId: f.companyId, issueId: f.issueId,
revision: 1, schemaVersion: "paperclip.completion-contract.v1", policyVersion: "test", risk: "standard",
@@ -510,7 +515,24 @@ describe("automatic retained workspace recovery", () => {
await db.update(nativeRunFinalizations).set({ resultId, attempt: 3, failureCode: "native_workspace_sync_out_retry_exhausted" })
.where(eq(nativeRunFinalizations.runId, f.runId));
expect(await resumePreservedResultFinalization(db, f.companyId, f.runId)).toBe(false);
const priorAction = await action(f.actionId);
await db.update(issueRecoveryActions).set({ ownerType: "system", evidence: { ...priorAction.evidence,
systemRecovery: failSystemRecovery(createSystemRecovery(f.runId),
new SystemRecoveryFailure("source_binding_invalid", "service_required")),
workspaceRecovery: { schema: "paperclip.workspace-recovery.v1", runId: f.runId, status: "blocked",
stage: "verifying_stop", attempt: 1, attemptLimit: 6, requestIds: [] },
} }).where(eq(issueRecoveryActions.id, f.actionId));
const request = { ...f.input, actorId: "board", idempotencyKey: randomUUID() };
await requestWorkspaceRecovery(request);
const admitted = await action(f.actionId);
await requestWorkspaceRecovery(request);
expect(await action(f.actionId)).toEqual(admitted);
expect(admitted.ownerType).toBe("system");
await recoverRetainedWorkspace(f.input);
expect(readSystemRecovery((await action(f.actionId)).evidence)?.state).toBe("complete");
expect(await readFile(path.join(f.host, "saved.txt"), "utf8")).toBe("finished work\n");
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
const attempts = await Promise.all([resumePreservedResultFinalization(db, f.companyId, f.runId),
resumePreservedResultFinalization(db, f.companyId, f.runId)]);
expect(attempts.filter(Boolean)).toHaveLength(1);
@@ -583,6 +605,20 @@ describe("automatic retained workspace recovery", () => {
sourceRunIds: [laterId], cwd: next })).toEqual([]);
}, 30_000);
it("retains files and the recovery barrier when native export rejects Git metadata", async () => {
const f = await seed();
const restore = vi.spyOn(nativeWorkspaceSync, "resumeNativeWorkspaceSync")
.mockRejectedValueOnce(new GitShallowMetadataError("Git shallow metadata exceeds its byte limit"));
try { await recoverRetainedWorkspace(f.input); } finally { restore.mockRestore(); }
expect(readSystemRecovery((await action(f.actionId)).evidence)).toMatchObject({ state: "exhausted",
failure: { code: "snapshot_verification_failed", retryClass: "service_required" }, preservationOperationIds: [] });
expect(await hasVerifiedWorkspaceRepair(db, await run(f.runId))).toBe(false);
expect(await readFile(path.join(f.remote, "saved.txt"), "utf8")).toBe("finished work\n");
expect(await readFile(path.join(f.host, "newer.txt"), "utf8")).toBe("newer local work\n");
expect(await db.select().from(agentWakeupRequests).where(eq(agentWakeupRequests.companyId, f.companyId))).toHaveLength(0);
expect((await run(f.runId)).status).toBe("failed");
}, 30_000);
it("reconciles an uploaded snapshot after a lost upload acknowledgement without exporting again", async () => {
const f = await seed("legacy", false);
const target = targets.get(f.lease.id)!;
@@ -1,3 +1,4 @@
import { classifyWorkspaceRestoreFailure } from "@paperclipai/adapter-utils/workspace-restore-merge";
import { notifyNativeResultCommit } from "./native-result-commit-notifications.js";
import { notifyNativeWorkspaceFinalization } from "./native-workspace-finalization-notifications.js";
import { notifyNativeStatusReassessment } from "./native-status-reassessment-notifications.js";
@@ -271,6 +272,8 @@ export async function recoverRetainedWorkspace(input: Source & { db: Db; environ
}
const classification = classifyNativeWorkspaceFailure(error);
const failure = error instanceof SystemRecoveryFailure ? error
: classifyWorkspaceRestoreFailure(error) === "restore_invalid_metadata"
? new SystemRecoveryFailure("snapshot_verification_failed", "service_required")
: classification.code === "workspace_sync_out_unsafe_archive"
? new SystemRecoveryFailure("archive_unsafe", "service_required")
: classification.code === "workspace_sync_out_unrecoverable"
+3 -2
View File
@@ -1401,7 +1401,8 @@ describe("TaskChatThread runtime transcript selection", () => {
});
it.each(["matching", "document_only", "absent", "other_revision"] as const)("reports a restore failure with %s saved-plan evidence", (evidence) => {
it.each((["matching", "document_only", "absent", "other_revision"] as const).flatMap(evidence =>
["restore_unsafe_archive", "restore_invalid_metadata"].map(code => [evidence, code] as const)))("reports %s saved-plan evidence for %s without offering another provider turn", (evidence, code) => {
if (evidence !== "absent") planState.data = planDocument();
renderThread({
issueId: "issue-1",
@@ -1414,7 +1415,7 @@ describe("TaskChatThread runtime transcript selection", () => {
runId: "restore-run", runtimeMode: "legacy", status: "failed", errorCode: "workspace_restore_failed",
agentId: "agent-1", agentName: "Runner", adapterType: "grok_local",
createdAt: "2026-08-25T18:00:00.000Z", startedAt: "2026-08-25T18:00:00.000Z", finishedAt: "2026-08-25T18:00:02.000Z",
resultJson: { workspaceRestoreFailure: "restore_unsafe_archive", workspaceRestorePath: ".claude/skills/paperclip", finalResponseRecorded: false,
resultJson: { workspaceRestoreFailure: code, workspaceRestorePath: ".claude/skills/paperclip", finalResponseRecorded: false,
...(evidence === "document_only" ? { savedPlanRevisionId: "revision-3" } : {}),
},
}],
+2 -1
View File
@@ -1831,7 +1831,8 @@ export function TaskChatThread(props: TaskChatThreadProps) {
runId: source.status === "cancelled" ? undefined : source.id,
...(branchUnavailable || aiRequest?.status === "pending" ? { retryable: false } : {}),
...(restoreFailed ? {
retryable: meta?.resultJson?.workspaceRestoreFailure !== "restore_unsafe_archive",
retryable: meta?.resultJson?.workspaceRestoreFailure !== "restore_unsafe_archive"
&& meta?.resultJson?.workspaceRestoreFailure !== "restore_invalid_metadata",
collapsible: true,
runHref: meta?.agentId ? `/agents/${encodeURIComponent(agentMap?.get(meta.agentId)?.urlKey ?? meta.agentId)}/runs/${encodeURIComponent(source.id)}` : undefined,
planHref: savedPlan ? "#document-plan" : undefined,