fix(native): expire settled native workspace sync folders (#17358)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs sync their workspace with a remote sandbox through a
per-run folder under `<instance>/native-workspace-sync/<run>/`
> - The folder holds a descriptor, a manifest, and workspace and git
seed archives, and a seed can be more than a gigabyte
> - Only a few finalization commit paths delete the folder, so many runs
leave it on disk forever
> - Instance disks fill up with folders that no code will read again
> - This pull request adds a bounded, deadline-driven retention worker
that removes a folder only when no reader can still need it
> - The benefit is that instance disks stop growing with stale sync
folders, and recovery and export paths keep their files

## Linked Issues or Issue Description

**What happened?**

`cleanupNativeWorkspaceSync(runId)` is the only code that deletes
`native-workspace-sync/<run>/`. It runs only when finalization commits
through specific paths. Runs that end in `terminal_failure`, retained
recovery, reconstruction, superseded runs, commits through other paths,
and server crashes leave the folder forever. On one instance, 22 folders
older than 7 days held 6.8 GB. Single seed archives were up to 1.4 GB.

**Expected behavior**

A run's sync folder is removed after a retention window, once the run
and its finalization are settled and no recovery or export path can
still read it. The shared `generations` seed cache stays.

**Steps to reproduce**

1. Run a native agent with a remote sandbox and let the run end in
`terminal_failure` (or crash the server during finalization).
2. Wait more than 7 days.
3. Look in `<instance>/native-workspace-sync/`. The run's folder and its
seed archives are still there.

**Paperclip version or commit**

`master` at 837add0caa

**Deployment mode**

Any deployment that runs native agents with remote sandboxes.

**Agent adapter(s) involved**

Native runtime with remote sandbox workspace sync (for example Codex on
Daytona).

## What Changed

- Added
`server/src/services/native-runtime/native-workspace-sync-retention.ts`.
It removes `native-workspace-sync/<name>/` when `<name>` is a UUID, the
entry is a real directory, its newest file is older than the window, and
one of these holds:
  - no `heartbeat_runs` row has that id (orphan);
- the run has no valid `nativeWorkspaceSync` reference and is settled:
terminal status, `finished_at` before the window, no live recorded
process (see below), no live controller lease, no
`nativeWorkspaceFinalizationOwner` receipt;
- the run is settled as above, its `native_run_finalizations` row (if
any) has no `lease_owner` and no `next_attempt_at` and is `committed`
(any recovery state) or `terminal_failure` with no or `blocked` recovery
state, no active/escalated export-cause or owner-unverified recovery
action names it, no lease carries `nativeWorkspaceExportResume` or
`pendingCleanupInFlight`, retained recovery cannot still use it
(referenced lease destroyed, or no scheduled/running/awaiting_evidence
system recovery and no queued/recovering workspace recovery), and
historical recovery adoption (`adoptHistoricalWorkspaceRecoveries`) has
no pending candidate for it.
- Each folder is decided in one transaction that holds
`pg_try_advisory_xact_lock` on the run's finalization ownership key. The
rule is evaluated while the lock is held. A held lock skips the folder.
The folder is renamed to `.expired-<uuid>` in that transaction and
deleted after commit. The next sweep removes leftover `.expired-*`
folders. `generations`, non-UUID names, symlinks and files are never
touched. Each tick tries at most 50 leftovers, then checks at most 200
folders and removes at most 50. Leftovers and folders each continue from
their own cursor, so undeletable leftovers do not block run folders. A
leftover that cannot be deleted schedules the 15-minute retry. Deadline
discovery reads run state 200 IDs at a time and stops when asked to. It
supports dry run and `shouldContinue`.
- A recorded `process_pid` / `process_group_id` blocks removal only
while it can still be the run's own process. This reuses the rule from
`conversation-continuation.ts`: PIDs are ignored when every lease of the
run is remote (they are sandbox PIDs), and a local PID or process group
blocks only while it exists on this host and its start time matches
`process_started_at`. An unreadable start time keeps the folder. A
recycled PID does not.
- Separate finding, not changed here: native runs do not clear
`process_pid` when they finish. It is cleared only by the status
arbiter's `release_run_resources` effect (cancellations and commits) and
by the recovery paths. A failed run gets a `schedule_retry` effect
instead, so its PID stays set. For remote targets the stored value is
the sandbox PID. Other readers already treat these PIDs as liveness
hints only, so this PR does not change that write path.
- Exported `nativeWorkspaceFinalizationLockKey` from
`native-workspace-finalization-ownership.ts` and used it in
`withNativeWorkspaceFinalizationOwnership` and the retention worker, so
both use the same key derivation.
- Exported `nativeWorkspaceSyncStateRoot()` from
`native-workspace-sync.ts`; `descriptorDirectory` now uses it.
- Registered the worker on the shared delivery-work coordinator
(`DELIVERY_QUEUES.nativeWorkspaceSyncRetention`) in
`server/src/index.ts`, next to the git restore point worker. It follows
`rules/sleep-aware-background-work.mdc`: no interval. It runs at startup
and at real deadlines: the earliest settled run's finish (or an orphan's
newest file) plus the window; a recheck one window later for folders it
kept; and a 15-minute retry after a failure or a held lock. With no
folders, `nextRunAt` is `null`. The run folder is now created through
one helper in `native-workspace-sync.ts` that notifies process-local
listeners; the worker wakes on that signal only while it has no
deadline, because any scheduled deadline is never later than a new
folder's own deadline. `index.ts` unsubscribes the listener on shutdown.
It pauses passive maintenance during idle drain.
- Added `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED` (default on
unless `false`), `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN`
(default off) and `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS`
(default `7`, minimum `1`) in `server/src/config.ts`. Documented them in
`doc/DEVELOPING.md` and added a "Native workspace sync folders" section
to `doc/recoverable-disk-cleanup.md`.

## Verification

- `pnpm exec vitest run
src/__tests__/native-workspace-sync-retention.test.ts
src/__tests__/native-workspace-sync-retention-config.test.ts` in
`server/`: 62 tests pass. They cover:
- deletes a settled committed run (also with a left-over
`resuming_session` recovery state), a blocked `terminal_failure` with no
recovery owner, an orphan, and an unreferenced settled run;
- keeps runs whose local process or process group is still alive, or
whose live process start time cannot be read; keeps a running run with a
PID; deletes terminal runs whose recorded local process exited, whose
PID was recycled, or whose PID came from a destroyed sandbox;
- keeps recent files, recently finished runs, unsettled runs, a live
controller lease, an owner receipt, `observed` and `retryable_failure`
phases, a live `lease_owner`, a scheduled `next_attempt_at`, and each
non-blocked recovery state on `terminal_failure`;
- keeps runs with an open export-cause or owner-unverified recovery
action, `nativeWorkspaceExportResume` or `pendingCleanupInFlight` lease
metadata, retained recovery in flight on a lease that was not destroyed,
or pending historical recovery adoption; deletes when the retained lease
was destroyed;
- never touches `generations`, non-UUID names, symlinks or files; dry
run deletes nothing; leftover `.expired-*` folders are removed; an
advisory lock held by another transaction makes it skip; ticks are
bounded and resume from a cursor;
- `nextDeadlineAt` returns `null` with no folders, now with leftovers,
the settled finish plus the window, the orphan mtime plus the window,
and one window later for kept folders; the registered worker goes quiet
with no folders, retries after a held lock or an undeletable leftover,
and the real delivery-work coordinator wakes again when the real
producer (`writeDescriptor`) creates a run folder; leftover removal is
bounded and undeletable leftovers do not block run folders.
- Mutation checks: I removed each guard one at a time (advisory lock,
owner receipt, process pid, finished_at window, lease owner, next
attempt, phase, recovery state, export recovery, owner-unverified,
export resume, retained recovery, backfill pending,
symlink/real-directory check, dry run, and both mtime checks together,
plus the folder-creation wake, the process liveness rule (restoring the
old `process_pid IS NULL` rule, the sandbox exemption, the process group
check, the recycled-PID check, the unreadable start time), the leftover
batch limit, the leftover cursor and the leftover retry delay). Each
mutation made at least one test fail. I restored the code after each
one. Removing only the first mtime check does not fail a test, because
the second check inside the transaction still keeps the folder.
- `pnpm exec vitest run` on related suites in `server/`:
`native-workspace-sync.test.ts`,
`native-workspace-sync-history.test.ts`,
`native-workspace-finalizer.test.ts`,
`native-workspace-finalizer-failure.test.ts`,
`native-harness-backup-expiration*.test.ts`,
`git-restore-point-ledger*.test.ts`,
`delivery-work-coordinator.test.ts`, `chat-delivery-work.test.ts`. All
pass.
- `pnpm run typecheck` in `server/` passes.
- Greptile review: earlier rounds; all findings fixed with tests; final
score 5/5 with no open threads.
- Not run: the full server test suite and an end-to-end run against a
real instance. I did not measure disk savings on a live instance.

## Risks

- Deleting a folder that a reader still needs would make that path fall
back or fail. Same-run resume and export run under the finalization
ownership lock or before the run settles, and the worker checks the rule
while it holds the same lock. Retained workspace recovery already
handles a missing folder (ENOENT) by snapshotting the sandbox. The rule
keeps the folder for every in-flight or reopenable recovery state listed
above.
- A committed finalization row with a stale recovery state and no lease
owner is treated as settled. Nothing reads the recovery state of
committed rows.
- The worker is on by default. Operators can set
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN=true` first to see
candidates in the log, or turn it off with
`PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED=false`.
- The first sweep on a large instance does more filesystem work. It is
bounded to 200 folders checked and 50 removed per tick.
- The folder-creation wake is process-local. A folder created by another
server process sharing the same instance directory is picked up at the
next deadline or restart.
- A stale `process_pid` no longer blocks removal. If a run's local
process is still alive with its recorded start time, the folder stays.
Live work is also still guarded by the run status, controller lease,
finalization lease and ownership lock checks.
- No schema migration.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

doc/DEVELOPING.md

@@ -2597,6 +2597,13 @@ Environment overrides:
   negative, empty, or non-numeric value falls back to the default.
   `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and
   `PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates.
+- `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS=<days>` sets how long a settled
+  run's native workspace sync folder (descriptor, manifest and seed archives under
+  `<instance>/native-workspace-sync/<run>`) is kept (see
+  `doc/recoverable-disk-cleanup.md`). The default is `7` and the minimum is `1`. A
+  negative, empty, or non-numeric value falls back to the default.
+  `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED=false` turns retention off, and
+  `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN=true` only reports candidates.
 - `PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true` lets git cache expiration remove closed
   worktrees with unpushed or uncommitted work after saving an encrypted restore point
   (see `doc/recoverable-disk-cleanup.md`). It is off by default.

doc/recoverable-disk-cleanup.md

@@ -93,6 +93,64 @@ The rest of the session directory, including its control-plane state, is kept.
 Each batch that finds candidates logs them with candidate and reclaimed allocated
 bytes, cleared task sessions, and skip reasons.
 
+## Native workspace sync folders
+
+Each native run that syncs a workspace with a remote sandbox writes a folder at
+`<instance>/native-workspace-sync/<run>/` with its descriptor, manifest, and the
+workspace and git seed archives. A seed can reach more than a gigabyte. Only the
+same run reads the folder again: restart recovery, finalization export, board export
+retry, and retained workspace recovery. Later runs adopt warm workspaces from lease
+metadata and the shared `generations` cache, which this worker never touches.
+
+A retention worker on the shared delivery-work scheduler removes a run's folder once
+nothing can read it. It runs at startup and at real deadlines: the earliest settled
+run's finish (or an orphan folder's newest file) plus the retention window, a
+recheck one window later for folders it had to keep, and a 15-minute retry after a
+failure, a held finalization lock or a leftover it could not delete. With no
+folders it schedules nothing; when a run then creates its folder in this server
+process, the worker wakes and schedules that folder's deadline. A folder
+named for a run id, that is a real directory and whose newest file is older than
+the window, is removed when one of these holds:
+
+- no run with that id exists;
+- the run has no valid workspace sync reference and is settled: it `succeeded`,
+  `failed`, was `cancelled`, `timed_out` or `interrupted`, finished before the
+  window, has no live process, no live controller lease and no finalization owner
+  receipt;
+- the run is settled as above, and also:
+  - its native run finalization, if any, has no lease owner or scheduled retry and is
+    `committed`, or `terminal_failure` with no recovery state or `blocked`;
+  - no active or escalated workspace export or owner-unverified recovery action
+    names the run;
+  - no lease of the run carries `nativeWorkspaceExportResume` or a cleanup in flight;
+  - its referenced lease was destroyed, or no retained workspace recovery of the run
+    is scheduled, running, awaiting evidence, queued or recovering;
+  - historical recovery adoption has no pending work for the run.
+
+Failed native runs can keep their recorded process ID after they finish. A recorded
+process blocks removal only while it can still be the run's own process: the run has a
+local lease or no lease, and the process ID or process group still exists on this
+host with the recorded start time (or a start time that cannot be read). A process ID
+from a remote sandbox, or one the operating system reused, does not block removal.
+
+Each folder is checked in one transaction that holds the run's finalization
+ownership lock, so it never races a finalization or export of the same run. A held
+lock skips the folder. Inside that transaction the folder is renamed to
+`.expired-<UUID>`, and it is deleted after commit. The next sweep finishes an
+interrupted deletion. Each batch tries at most 50 leftovers, then checks at most 200
+folders and removes at most 50. Leftovers and folders each continue from where the
+previous batch stopped, so a leftover that cannot be deleted does not block others. Deadline discovery reads
+run state 200 folders at a time and stops when the server drains or shuts down.
+
+| Environment variable | Default | Effect |
+| --- | --- | --- |
+| `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED` | `true` | Set `false` to stop removing folders |
+| `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN` | `false` | Set `true` to report candidates without removing folders |
+| `PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS` | `7` | Days a run must be finished, and its folder untouched, before removal (minimum `1`) |
+
+Each batch that finds candidates logs them with candidate and reclaimed allocated
+bytes and skip reasons.
+
 ## Git workspace restore points
 
 Git cache expiration (`PAPERCLIP_GIT_CACHE_EXPIRATION_ENABLED=true`) normally removes a

server/src/__tests__/native-workspace-sync-retention-config.test.ts

@@ -0,0 +1,40 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import { loadConfig } from "../config.ts";
+
+describe("native workspace sync retention config parsing", () => {
+  afterEach(() => {
+    vi.unstubAllEnvs();
+  });
+
+  it("is enabled with a seven day window by default", () => {
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED", undefined);
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN", undefined);
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", undefined);
+    expect(loadConfig()).toMatchObject({
+      nativeWorkspaceSyncRetentionEnabled: true,
+      nativeWorkspaceSyncRetentionDryRun: false,
+      nativeWorkspaceSyncRetentionDays: 7,
+    });
+  });
+
+  it("accepts explicit controls", () => {
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED", "false");
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN", "true");
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", "3");
+    expect(loadConfig()).toMatchObject({
+      nativeWorkspaceSyncRetentionEnabled: false,
+      nativeWorkspaceSyncRetentionDryRun: true,
+      nativeWorkspaceSyncRetentionDays: 3,
+    });
+  });
+
+  it.each(["", "-1", "many"])("falls back to seven days for %j", (value) => {
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", value);
+    expect(loadConfig()).toMatchObject({ nativeWorkspaceSyncRetentionDays: 7 });
+  });
+
+  it("never keeps folders for less than one day", () => {
+    vi.stubEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", "0");
+    expect(loadConfig()).toMatchObject({ nativeWorkspaceSyncRetentionDays: 1 });
+  });
+});

server/src/__tests__/native-workspace-sync-retention.test.ts

@@ -0,0 +1,665 @@
+import { randomUUID } from "node:crypto";
+import os from "node:os";
+import path from "node:path";
+import { promises as fs } from "node:fs";
+import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
+import { sql } from "drizzle-orm";
+import {
+  agents,
+  companies,
+  createDb,
+  environmentLeases,
+  heartbeatRuns,
+  issueRecoveryActions,
+  issues,
+  nativeRunFinalizations,
+} from "@paperclipai/db";
+import {
+  getEmbeddedPostgresTestSupport,
+  startEmbeddedPostgresTestDatabase,
+} from "./helpers/embedded-postgres.js";
+import { nativeWorkspaceFinalizationLockKey } from "../services/native-runtime/native-workspace-finalization-ownership.ts";
+import {
+  NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS,
+  nativeWorkspaceSyncRetentionService,
+  registerNativeWorkspaceSyncRetentionWork,
+} from "../services/native-runtime/native-workspace-sync-retention.ts";
+import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.ts";
+import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.ts";
+import {
+  nativeWorkspaceSyncInternals,
+  nativeWorkspaceSyncStateRoot,
+} from "../services/native-runtime/native-workspace-sync.ts";
+
+const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
+const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
+
+if (!embeddedPostgresSupport.supported) {
+  console.warn(
+    `Skipping embedded Postgres native workspace sync retention tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
+  );
+}
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+async function exists(target: string) {
+  return fs.lstat(target).then(() => true, () => false);
+}
+
+type SeedInput = {
+  runStatus?: string;
+  finishedDaysAgo?: number | null;
+  folderAgeDays?: number;
+  reference?: boolean;
+  owner?: boolean;
+  processPid?: number | null;
+  processGroupId?: number | null;
+  processStartedAt?: Date | null;
+  leaseProvider?: string | null;
+  controllerLeaseDaysFromNow?: number | null;
+  finalization?: {
+    phase: string;
+    recoveryState?: string | null;
+    leaseOwner?: string | null;
+    nextAttemptDaysFromNow?: number | null;
+  } | null;
+  leaseMetadata?: Record<string, unknown>;
+  leasePolicy?: string;
+  recoveryAction?: { cause?: string; status?: string; evidence?: Record<string, unknown> };
+  resultJson?: Record<string, unknown>;
+};
+
+describeEmbeddedPostgres("native workspace sync retention", () => {
+  let db!: ReturnType<typeof createDb>;
+  let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
+  let stateRoot = "";
+  const now = new Date();
+
+  beforeAll(async () => {
+    tempDb = await startEmbeddedPostgresTestDatabase("paperclip-native-workspace-sync-retention-");
+    db = createDb(tempDb.connectionString);
+  }, 20_000);
+
+  afterEach(async () => {
+    livePids = new Map();
+    await db.delete(issueRecoveryActions);
+    await db.delete(nativeRunFinalizations);
+    await db.delete(environmentLeases);
+    await db.delete(heartbeatRuns);
+    await db.delete(issues);
+    await db.delete(agents);
+    await db.delete(companies);
+    if (stateRoot) await fs.rm(stateRoot, { recursive: true, force: true });
+    stateRoot = "";
+  });
+
+  afterAll(async () => {
+    await tempDb?.cleanup();
+  });
+
+  async function root() {
+    if (!stateRoot) stateRoot = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-native-workspace-sync-"));
+    return stateRoot;
+  }
+
+  async function writeFolder(name: string, ageDays: number) {
+    const folder = path.join(await root(), name);
+    await fs.mkdir(folder, { recursive: true });
+    const files = ["descriptor.json", "manifest.sqlite", "workspace-seed.tar", "git-seed.tar"];
+    for (const file of files) await fs.writeFile(path.join(folder, file), "x".repeat(4096));
+    const at = new Date(now.getTime() - ageDays * DAY_MS);
+    for (const file of files) await fs.utimes(path.join(folder, file), at, at);
+    await fs.utimes(folder, at, at);
+    return folder;
+  }
+
+  async function seed(input: SeedInput = {}) {
+    const companyId = randomUUID();
+    await db.insert(companies).values({ id: companyId, name: "Sync", issuePrefix: `S${companyId.slice(0, 6)}` });
+    const [agent] = await db.insert(agents).values({ companyId, name: "Coder" }).returning();
+    const [issue] = await db.insert(issues).values({ companyId, title: "Work", status: "done" }).returning();
+    const runId = randomUUID();
+    const leaseId = randomUUID();
+    const runStatus = input.runStatus ?? "succeeded";
+    const finishedDaysAgo = input.finishedDaysAgo === undefined ? 10 : input.finishedDaysAgo;
+    const profile: Record<string, unknown> = {};
+    if (input.reference ?? true) {
+      profile.nativeWorkspaceSync = {
+        schema: "paperclip.native-workspace-sync/v3",
+        state: "finalized",
+        descriptorSha256: "a".repeat(64),
+        baselineSha256: "b".repeat(64),
+        finalHostSha256: null,
+        workspaceId: randomUUID(),
+        leaseId,
+        providerLeaseId: "provider-lease",
+        remoteCwd: "/workspace",
+      };
+    }
+    if (input.owner) profile.nativeWorkspaceFinalizationOwner = { token: randomUUID() };
+    await db.insert(heartbeatRuns).values({
+      id: runId,
+      companyId,
+      agentId: agent!.id,
+      status: runStatus,
+      issueId: issue!.id,
+      nativeIssueId: issue!.id,
+      finishedAt: finishedDaysAgo === null ? null : new Date(now.getTime() - finishedDaysAgo * DAY_MS),
+      processPid: input.processPid ?? null,
+      processGroupId: input.processGroupId ?? null,
+      processStartedAt: input.processStartedAt ?? null,
+      controllerLeaseExpiresAt: input.controllerLeaseDaysFromNow == null
+        ? null
+        : new Date(now.getTime() + input.controllerLeaseDaysFromNow * DAY_MS),
+      runnerProfileJson: profile,
+      resultJson: input.resultJson ?? null,
+    });
+    const finalization = input.finalization === undefined ? { phase: "committed" } : input.finalization;
+    if (finalization) {
+      await db.insert(nativeRunFinalizations).values({
+        runId,
+        companyId,
+        issueId: issue!.id,
+        phase: finalization.phase,
+        recoveryState: finalization.recoveryState ?? null,
+        leaseOwner: finalization.leaseOwner ?? null,
+        nextAttemptAt: finalization.nextAttemptDaysFromNow == null
+          ? null
+          : new Date(now.getTime() + finalization.nextAttemptDaysFromNow * DAY_MS),
+      });
+    }
+    if (input.leaseMetadata || input.leasePolicy || input.leaseProvider !== undefined) {
+      await db.insert(environmentLeases).values({
+        id: leaseId,
+        companyId,
+        heartbeatRunId: runId,
+        status: "released",
+        provider: input.leaseProvider ?? null,
+        leasePolicy: input.leasePolicy ?? "ephemeral",
+        metadata: input.leaseMetadata ?? {},
+      });
+    }
+    if (input.recoveryAction) {
+      await db.insert(issueRecoveryActions).values({
+        companyId,
+        sourceIssueId: issue!.id,
+        kind: "active_run_watchdog",
+        ownerType: "system",
+        cause: input.recoveryAction.cause ?? "native_continuation_requires_reconciliation",
+        fingerprint: randomUUID(),
+        status: input.recoveryAction.status ?? "active",
+        nextAction: "Recover",
+        evidence: { runId, systemRecovery: { state: "complete" }, ...input.recoveryAction.evidence },
+      });
+    }
+    const folder = await writeFolder(runId, input.folderAgeDays ?? 9);
+    return { companyId, runId, folder };
+  }
+
+  const processStart = new Date(now.getTime() - 20 * DAY_MS);
+  let livePids = new Map<number, string | null>();
+
+  const processProbe = {
+    mayBeAlive: (pid: number) => livePids.has(pid),
+    startedAt: async (pid: number) => livePids.get(pid) ?? null,
+  };
+
+  function service() {
+    return nativeWorkspaceSyncRetentionService(db, { stateRoot, now: () => now, processProbe });
+  }
+
+  async function expectKept(input: SeedInput, reason: string) {
+    const { folder } = await seed(input);
+    const result = await service().tick();
+    expect(result.expired).toBe(0);
+    expect(result.skipped[reason]).toBe(1);
+    expect(await exists(folder)).toBe(true);
+  }
+
+  it("deletes a settled committed run's folder after the retention window", async () => {
+    const { folder } = await seed();
+    const result = await service().tick();
+    expect(result).toMatchObject({ checked: 1, candidates: 1, expired: 1, failed: 0 });
+    expect(result.reclaimedBytes).toBeGreaterThan(0);
+    expect(await exists(folder)).toBe(false);
+    expect((await fs.readdir(stateRoot)).filter((name) => name.startsWith(".expired-"))).toEqual([]);
+  });
+
+  it("deletes committed runs whose recovery state was left behind without a lease", async () => {
+    const { folder } = await seed({ finalization: { phase: "committed", recoveryState: "resuming_session" } });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("deletes blocked terminal failures with no recovery owner", async () => {
+    const { folder } = await seed({
+      runStatus: "failed",
+      finalization: { phase: "terminal_failure", recoveryState: "blocked" },
+      recoveryAction: { status: "resolved", cause: "native_workspace_sync_out_unrecoverable" },
+    });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("deletes orphaned folders and unreferenced settled runs", async () => {
+    const orphan = await writeFolder(randomUUID(), 9);
+    const { folder } = await seed({ reference: false, finalization: { phase: "observed" } });
+    const result = await service().tick();
+    expect(result.expired).toBe(2);
+    expect(await exists(orphan)).toBe(false);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("keeps folders with recent files or recently finished runs", async () => {
+    const recentFiles = await seed({ folderAgeDays: 2 });
+    const recentRun = await seed({ finishedDaysAgo: 3 });
+    const result = await service().tick();
+    expect(result.expired).toBe(0);
+    expect(result.skipped).toMatchObject({ recent_files: 1, recently_finished: 1 });
+    expect(await exists(recentFiles.folder)).toBe(true);
+    expect(await exists(recentRun.folder)).toBe(true);
+  });
+
+  it("keeps runs that are not settled", async () => {
+    await expectKept({ runStatus: "running", finishedDaysAgo: null }, "run_unsettled");
+  });
+
+  it("keeps runs whose local process is still alive", async () => {
+    livePids.set(1234, processStart.toISOString());
+    await expectKept({ processPid: 1234, processStartedAt: processStart, leaseProvider: "local" }, "process_live");
+  });
+
+  it("keeps runs whose local process is alive with an unreadable start time", async () => {
+    livePids.set(1234, null);
+    await expectKept({ processPid: 1234, processStartedAt: processStart }, "process_live");
+  });
+
+  it("keeps runs whose local process group is still alive", async () => {
+    livePids.set(-4321, null);
+    await expectKept({ processPid: 4321, processGroupId: 4321 }, "process_live");
+  });
+
+  it("keeps a running run that still records a process", async () => {
+    livePids.set(1234, processStart.toISOString());
+    await expectKept({ runStatus: "running", finishedDaysAgo: null, processPid: 1234 }, "run_unsettled");
+  });
+
+  it("deletes a terminal run whose recorded local process has exited", async () => {
+    const { folder } = await seed({
+      runStatus: "failed",
+      finalization: { phase: "terminal_failure" },
+      processPid: 1234,
+      processGroupId: 1234,
+      processStartedAt: processStart,
+      leaseProvider: "local",
+    });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("deletes a terminal run whose recorded pid was recycled by another process", async () => {
+    livePids.set(1234, new Date(now.getTime() - DAY_MS).toISOString());
+    const { folder } = await seed({ processPid: 1234, processStartedAt: processStart });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("deletes a terminal run whose stale pid came from a destroyed sandbox", async () => {
+    livePids.set(1234, processStart.toISOString());
+    livePids.set(-1234, null);
+    const { folder } = await seed({
+      runStatus: "cancelled",
+      finalization: { phase: "terminal_failure" },
+      processPid: 1234,
+      processGroupId: 1234,
+      processStartedAt: processStart,
+      leaseProvider: "daytona",
+      leaseMetadata: { remoteExecutionTermination: { state: "destroyed" } },
+    });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("keeps runs with a live controller lease", async () => {
+    await expectKept({ controllerLeaseDaysFromNow: 1 }, "controller_live");
+  });
+
+  it("keeps runs with a finalization owner receipt, even without a reference", async () => {
+    await expectKept({ owner: true, reference: false }, "finalization_owner");
+  });
+
+  it("keeps unsettled finalization phases", async () => {
+    await expectKept({ finalization: { phase: "observed" } }, "finalization_unsettled");
+  });
+
+  it("keeps retryable failures without a result", async () => {
+    await expectKept({ finalization: { phase: "retryable_failure" } }, "finalization_unsettled");
+  });
+
+  it("keeps finalizations with a live lease owner", async () => {
+    await expectKept({ finalization: { phase: "committed", leaseOwner: "controller" } }, "finalization_leased");
+  });
+
+  it("keeps finalizations with a scheduled retry", async () => {
+    await expectKept(
+      { finalization: { phase: "terminal_failure", nextAttemptDaysFromNow: 1 } },
+      "finalization_retry_scheduled",
+    );
+  });
+
+  it.each(["awaiting_evidence", "awaiting_runner_reattach", "resuming_session", "bootstrap_incomplete"])(
+    "keeps terminal failures in recovery state %s",
+    async (recoveryState) => {
+      await expectKept({ finalization: { phase: "terminal_failure", recoveryState } }, "finalization_recovering");
+    },
+  );
+
+  it.each(["native_workspace_sync_out_unrecoverable", "native_workspace_sync_out_retry_exhausted"])(
+    "keeps runs with an open %s recovery action",
+    async (cause) => {
+      await expectKept({ recoveryAction: { cause, status: "escalated" } }, "export_recovery");
+    },
+  );
+
+  it("keeps runs with an open owner-unverified recovery action", async () => {
+    await expectKept(
+      { recoveryAction: { cause: "native_workspace_finalization_owner_unverified" } },
+      "owner_unverified",
+    );
+  });
+
+  it("keeps runs whose export resume is recorded on the lease", async () => {
+    await expectKept({ leaseMetadata: { nativeWorkspaceExportResume: { attempt: 1 } } }, "export_resume");
+  });
+
+  it("keeps runs whose lease cleanup is in flight", async () => {
+    await expectKept({ leaseMetadata: { pendingCleanupInFlight: true } }, "export_resume");
+  });
+
+  it.each([
+    { systemRecovery: { state: "scheduled" } },
+    { systemRecovery: { state: "running" } },
+    { systemRecovery: { state: "awaiting_evidence" } },
+    { workspaceRecovery: { status: "queued" } },
+    { workspaceRecovery: { status: "recovering" } },
+  ])("keeps retained recovery %j on a lease that was not destroyed", async (evidence) => {
+    await expectKept(
+      { leaseMetadata: { systemRecoveryAdoption: {} }, recoveryAction: { status: "resolved", evidence } },
+      "retained_recovery",
+    );
+  });
+
+  it("deletes when the retained lease was destroyed", async () => {
+    const { folder } = await seed({
+      leaseMetadata: { remoteExecutionTermination: { state: "destroyed" } },
+      recoveryAction: { status: "resolved", evidence: { systemRecovery: { state: "scheduled" } } },
+    });
+    expect((await service().tick()).expired).toBe(1);
+    expect(await exists(folder)).toBe(false);
+  });
+
+  it("keeps runs that historical recovery adoption has not reached", async () => {
+    await expectKept({ leasePolicy: "retain_on_failure" }, "recovery_backfill_pending");
+  });
+
+  it("keeps runs with an unadopted export recovery action", async () => {
+    const { runId, folder } = await seed({
+      recoveryAction: { status: "resolved", cause: "native_workspace_sync_out_unrecoverable" },
+    });
+    await db.execute(sql`update issue_recovery_actions set evidence = evidence - 'systemRecovery'
+      where evidence->>'runId' = ${runId}`);
+    const result = await service().tick();
+    expect(result.skipped.recovery_backfill_pending).toBe(1);
+    expect(await exists(folder)).toBe(true);
+  });
+
+  it("never touches generations, non-UUID names, symlinks, or files", async () => {
+    const base = await root();
+    const generations = await writeFolder("generations", 30);
+    const other = await writeFolder("not-a-run", 30);
+    const target = await writeFolder(".outside", 30);
+    const linkName = randomUUID();
+    await fs.symlink(target, path.join(base, linkName));
+    const fileName = randomUUID();
+    await fs.writeFile(path.join(base, fileName), "x");
+    const old = new Date(now.getTime() - 30 * DAY_MS);
+    await fs.utimes(path.join(base, fileName), old, old);
+    const result = await service().tick();
+    expect(result.expired).toBe(0);
+    expect(result.skipped.unsafe_path).toBe(2);
+    for (const entry of [generations, other, target, path.join(base, linkName), path.join(base, fileName)]) {
+      expect(await exists(entry)).toBe(true);
+    }
+  });
+
+  it("deletes nothing during a dry run", async () => {
+    const { folder } = await seed();
+    const leftover = await writeFolder(`.expired-${randomUUID()}`, 1);
+    const result = await nativeWorkspaceSyncRetentionService(db, { stateRoot, now: () => now, dryRun: true }).tick();
+    expect(result).toMatchObject({ candidates: 1, expired: 0, leftoversRemoved: 0 });
+    expect(await exists(folder)).toBe(true);
+    expect(await exists(leftover)).toBe(true);
+  });
+
+  it("bounds leftover removal by the batch limits", async () => {
+    const leftovers: string[] = [];
+    for (let index = 0; index < 3; index++) leftovers.push(await writeFolder(`.expired-${randomUUID()}`, 0));
+    const folder = await writeFolder(randomUUID(), 9);
+    expect(await service().tick({ shouldContinue: () => false })).toMatchObject({ leftoversRemoved: 0, truncated: true });
+    const retention = service();
+    const first = await retention.tick({ maxRemovals: 2 });
+    expect(first).toMatchObject({ leftoversRemoved: 2, expired: 1, truncated: true });
+    expect(await exists(folder)).toBe(false);
+    const rest = await retention.tick({ maxRemovals: 2 });
+    expect(rest).toMatchObject({ leftoversRemoved: 1, expired: 0, truncated: false });
+    for (const leftover of leftovers) expect(await exists(leftover)).toBe(false);
+  });
+
+  it("keeps expiring run folders and moves past leftovers that cannot be removed", async () => {
+    const stuck: string[] = [];
+    for (let index = 0; index < 2; index++) {
+      const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
+      await fs.chmod(leftover, 0o500);
+      stuck.push(leftover);
+    }
+    const folder = await writeFolder(randomUUID(), 9);
+    const retention = service();
+    try {
+      const first = await retention.tick({ maxRemovals: 1 });
+      expect(first).toMatchObject({ failed: 1, expired: 1, truncated: true });
+      expect(await exists(folder)).toBe(false);
+      const second = await retention.tick({ maxRemovals: 1 });
+      expect(second).toMatchObject({ failed: 1, truncated: false });
+    } finally {
+      for (const leftover of stuck) await fs.chmod(leftover, 0o700);
+    }
+  });
+
+  it("removes leftover expired folders", async () => {
+    const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
+    const retiring = await writeFolder(`.retiring-${randomUUID()}`, 30);
+    const result = await service().tick();
+    expect(result.leftoversRemoved).toBe(1);
+    expect(await exists(leftover)).toBe(false);
+    expect(await exists(retiring)).toBe(true);
+  });
+
+  it("skips a run while its finalization ownership lock is held", async () => {
+    const { companyId, runId, folder } = await seed();
+    let result: Awaited<ReturnType<ReturnType<typeof service>["tick"]>> | undefined;
+    await db.transaction(async (tx) => {
+      await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(companyId, runId)}, 0))`);
+      result = await service().tick();
+    });
+    expect(result?.skipped.finalization_locked).toBe(1);
+    expect(await exists(folder)).toBe(true);
+    expect((await service().tick()).expired).toBe(1);
+  });
+
+  it("bounds each tick and resumes from its cursor", async () => {
+    const first = await writeFolder(randomUUID(), 9);
+    const second = await writeFolder(randomUUID(), 9);
+    const retention = service();
+    const initial = await retention.tick({ maxRemovals: 1 });
+    expect(initial).toMatchObject({ expired: 1, truncated: true });
+    const rest = await retention.tick({ maxRemovals: 1 });
+    expect(rest).toMatchObject({ expired: 1, truncated: false });
+    expect(await exists(first)).toBe(false);
+    expect(await exists(second)).toBe(false);
+  });
+
+  it("stops when asked to", async () => {
+    const folder = await writeFolder(randomUUID(), 9);
+    const result = await service().tick({ shouldContinue: () => false });
+    expect(result).toMatchObject({ checked: 0, truncated: true });
+    expect(await exists(folder)).toBe(true);
+  });
+
+  describe("scheduled worker", () => {
+    function register() {
+      let queue = "";
+      let task!: { run: (signal: AbortSignal) => Promise<unknown>; nextRunAt?: () => number | null };
+      const coordinator = {
+        register: (registeredQueue: string, registered: typeof task) => {
+          queue = registeredQueue;
+          task = registered;
+          return { ready: Promise.resolve(), wake: () => {} };
+        },
+      };
+      const onResult = vi.fn();
+      registerNativeWorkspaceSyncRetentionWork(coordinator as never, service(), { onResult });
+      return { queue: () => queue, task: () => task, onResult };
+    }
+
+    it("goes quiet with no folders and schedules the next real deadline", async () => {
+      await root();
+      const worker = register();
+      expect(worker.queue()).toBe(DELIVERY_QUEUES.nativeWorkspaceSyncRetention);
+      await worker.task().run(new AbortController().signal);
+      expect(worker.task().nextRunAt?.()).toBeNull();
+      await seed({ finishedDaysAgo: 3, folderAgeDays: 4 });
+      await worker.task().run(new AbortController().signal);
+      expect(worker.task().nextRunAt?.()).toBe(now.getTime() + 4 * DAY_MS);
+      expect(worker.onResult).toHaveBeenCalledTimes(2);
+    });
+
+    it("waits the retry delay when a leftover cannot be removed", async () => {
+      const base = await root();
+      const leftover = await writeFolder(`.expired-${randomUUID()}`, 0);
+      await fs.writeFile(path.join(leftover, "nested"), "x");
+      await fs.chmod(leftover, 0o500);
+      try {
+        const worker = register();
+        await worker.task().run(new AbortController().signal);
+        expect(worker.onResult.mock.calls[0]![0]).toMatchObject({ failed: 1 });
+        expect(worker.task().nextRunAt?.()).toBeGreaterThanOrEqual(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
+      } finally {
+        await fs.chmod(leftover, 0o700);
+      }
+      expect(await exists(base)).toBe(true);
+    });
+
+    it("wakes the quiet coordinator when a run creates its folder", async () => {
+      const home = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-sync-home-"));
+      const originalHome = process.env.PAPERCLIP_HOME;
+      const originalInstance = process.env.PAPERCLIP_INSTANCE_ID;
+      process.env.PAPERCLIP_HOME = home;
+      process.env.PAPERCLIP_INSTANCE_ID = "retention";
+      const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: () => {} });
+      const onResult = vi.fn();
+      const worker = registerNativeWorkspaceSyncRetentionWork(
+        coordinator,
+        nativeWorkspaceSyncRetentionService(db, { stateRoot: nativeWorkspaceSyncStateRoot() }),
+        { onResult },
+      );
+      try {
+        await worker.ready;
+        expect(onResult).toHaveBeenCalledTimes(1);
+        expect(coordinator.nextWakeAt()).toBeNull();
+        await nativeWorkspaceSyncInternals.writeDescriptor({
+          schema: "paperclip.native-workspace-sync/v3",
+          state: "prepared",
+          baselineSha256: "b".repeat(64),
+          finalHostSha256: null,
+          binding: { runId: randomUUID(), workspaceId: randomUUID(), leaseId: randomUUID(), providerLeaseId: "lease", remoteCwd: "/workspace" },
+        } as never);
+        await vi.waitFor(() => expect(onResult).toHaveBeenCalledTimes(2));
+        await vi.waitFor(() => expect(coordinator.nextWakeAt()).not.toBeNull());
+        const wakeAt = coordinator.nextWakeAt()!;
+        expect(wakeAt).toBeGreaterThan(Date.now() + 6 * DAY_MS);
+      } finally {
+        worker.unsubscribe();
+        await coordinator.stop();
+        if (originalHome === undefined) delete process.env.PAPERCLIP_HOME;
+        else process.env.PAPERCLIP_HOME = originalHome;
+        if (originalInstance === undefined) delete process.env.PAPERCLIP_INSTANCE_ID;
+        else process.env.PAPERCLIP_INSTANCE_ID = originalInstance;
+        await fs.rm(home, { recursive: true, force: true });
+      }
+    });
+
+    it("retries soon when the finalization lock was held", async () => {
+      const { companyId, runId } = await seed();
+      const worker = register();
+      const before = Date.now();
+      await db.transaction(async (tx) => {
+        await tx.execute(sql`select pg_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(companyId, runId)}, 0))`);
+        await worker.task().run(new AbortController().signal);
+      });
+      const next = worker.task().nextRunAt?.() ?? 0;
+      expect(next).toBeGreaterThanOrEqual(before + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
+      expect(next).toBeLessThanOrEqual(Date.now() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
+    });
+  });
+
+  describe("nextDeadlineAt", () => {
+    it("is null without folders", async () => {
+      await root();
+      await writeFolder("generations", 0);
+      expect(await service().nextDeadlineAt()).toBeNull();
+    });
+
+    it("is null when the state root does not exist", async () => {
+      expect(await nativeWorkspaceSyncRetentionService(db, {
+        stateRoot: path.join(os.tmpdir(), `missing-${randomUUID()}`),
+        now: () => now,
+      }).nextDeadlineAt()).toBeNull();
+    });
+
+    it("retries leftover expired folders after the retry delay", async () => {
+      await writeFolder(`.expired-${randomUUID()}`, 0);
+      expect(await service().nextDeadlineAt()).toBe(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
+    });
+
+    it("stops scanning when asked to and retries later", async () => {
+      await writeFolder(randomUUID(), 1);
+      expect(await service().nextDeadlineAt({ shouldContinue: () => false }))
+        .toBe(now.getTime() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS);
+    });
+
+    it("carries the earliest deadline across batches", async () => {
+      for (let index = 0; index < 205; index++) await writeFolder(randomUUID(), 1);
+      await writeFolder("ffffffff-ffff-4fff-8fff-ffffffffffff", 6);
+      const deadline = await service().nextDeadlineAt();
+      expect(Math.abs(deadline! - (now.getTime() + DAY_MS))).toBeLessThan(1000);
+    });
+
+    it("is the earliest settled run finish plus the retention window", async () => {
+      await seed({ finishedDaysAgo: 3, folderAgeDays: 4 });
+      await seed({ finishedDaysAgo: 2, folderAgeDays: 4 });
+      expect(await service().nextDeadlineAt()).toBe(now.getTime() + 4 * DAY_MS);
+    });
+
+    it("uses the folder's newest file for orphans", async () => {
+      await writeFolder(randomUUID(), 5);
+      const deadline = await service().nextDeadlineAt();
+      expect(Math.abs(deadline! - (now.getTime() + 2 * DAY_MS))).toBeLessThan(1000);
+    });
+
+    it("rechecks unsettled or kept folders one retention window later", async () => {
+      await seed({ runStatus: "running", finishedDaysAgo: null, folderAgeDays: 30 });
+      await seed({ finalization: { phase: "observed" } });
+      expect(await service().nextDeadlineAt()).toBe(now.getTime() + 7 * DAY_MS);
+    });
+  });
+});

server/src/config.ts

@@ -91,6 +91,9 @@ export interface Config {
   nativeHarnessBackupExpirationEnabled: boolean;
   nativeHarnessBackupExpirationDryRun: boolean;
   nativeHarnessBackupExpirationDays: number;
+  nativeWorkspaceSyncRetentionEnabled: boolean;
+  nativeWorkspaceSyncRetentionDryRun: boolean;
+  nativeWorkspaceSyncRetentionDays: number;
   gitRestorePointRetentionDays: number;
   gitRestorePointsEnabled: boolean;
   gitRestorePointTerminalDelayDays: number;
@@ -341,6 +344,7 @@ export function loadConfig(): Config {
   const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0);
   const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60);
   const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7));
+  const nativeWorkspaceSyncRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DAYS", 7));
   const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90));
   const gitRestorePointTerminalDelayDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_TERMINAL_DELAY_DAYS", 3);
   const gitRestorePointIdleDays = nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_IDLE_DAYS", 7);
@@ -401,6 +405,9 @@ export function loadConfig(): Config {
     nativeHarnessBackupExpirationEnabled: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED !== "false",
     nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true",
     nativeHarnessBackupExpirationDays,
+    nativeWorkspaceSyncRetentionEnabled: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_ENABLED !== "false",
+    nativeWorkspaceSyncRetentionDryRun: process.env.PAPERCLIP_NATIVE_WORKSPACE_SYNC_RETENTION_DRY_RUN === "true",
+    nativeWorkspaceSyncRetentionDays,
     gitRestorePointRetentionDays,
     gitRestorePointsEnabled: process.env.PAPERCLIP_GIT_RESTORE_POINTS_ENABLED === "true",
     gitRestorePointTerminalDelayDays,

server/src/index.ts

@@ -13,6 +13,10 @@ import {
   nativeHarnessBackupExpirationService,
 } from "./services/native-runtime/native-harness-backup-expiration.js";
 import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js";
+import {
+  nativeWorkspaceSyncRetentionService,
+  registerNativeWorkspaceSyncRetentionWork,
+} from "./services/native-runtime/native-workspace-sync-retention.js";
 import { markGitRestorePointExpired } from "./services/git-restore-point-storage.js";
 import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";
 import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
@@ -1351,6 +1355,7 @@ async function startServerWithDatabaseTeardown(
     hasPending: () => caseHandoffDeliveries.hasPending(),
   });
   let unregisterTerminalWorkspaceProbe: (() => void) | undefined;
+  let unsubscribeNativeWorkspaceSyncRetention: (() => void) | undefined;
   const fastResponses = app.locals.fastResponses as ReturnType<typeof import("./services/fast-responses.js").fastResponseService>;
   const fastResponseWorker = deliveryWork.register(DELIVERY_QUEUES.fastResponse, {
     retryMs: 100, run: (signal) => fastResponses.sweepPending(signal), hasPending: () => fastResponses.hasPending(),
@@ -1514,6 +1519,19 @@ async function startServerWithDatabaseTeardown(
         logger.info(result, "git restore point retention");
       }
     });
+    if (config.nativeWorkspaceSyncRetentionEnabled) {
+      unsubscribeNativeWorkspaceSyncRetention = registerNativeWorkspaceSyncRetentionWork(deliveryWork, nativeWorkspaceSyncRetentionService(db as any, {
+        retentionDays: config.nativeWorkspaceSyncRetentionDays,
+        dryRun: config.nativeWorkspaceSyncRetentionDryRun,
+      }), {
+        shouldContinue: () => !heartbeatSchedulerStopped && !isIdleTaskDrainActive() && !isWarmStandby(),
+        onResult: (result) => {
+          if (result.candidates > 0 || result.leftoversRemoved > 0 || result.failed > 0) {
+            logger.info({ ...result, dryRun: config.nativeWorkspaceSyncRetentionDryRun }, "native workspace sync retention");
+          }
+        },
+      }).unsubscribe;
+    }
     let lastTerminalWorkspaceSkipLogAt = 0;
     const terminalWorkspaceWorker = registerTerminalWorkspaceWork(deliveryWork, {
       ...terminalWorkspaces,
@@ -2072,6 +2090,7 @@ async function startServerWithDatabaseTeardown(
     unsubscribeChatCompletions();
     await deliveryWork.stop();
     unregisterTerminalWorkspaceProbe?.();
+    unsubscribeNativeWorkspaceSyncRetention?.();
     if (heartbeatSchedulerInterval) {
       clearInterval(heartbeatSchedulerInterval);
       heartbeatSchedulerInterval = null;

server/src/services/delivery-work-notifications.ts

@@ -39,6 +39,7 @@ export const DELIVERY_QUEUES = {
   chatCompletion: "chat-completions",
   connection: "connection-continuations",
   gitRestorePoints: "git-restore-points",
+  nativeWorkspaceSyncRetention: "native-workspace-sync-retention",
   question: "question-responses",
   pluginLifecycle: "plugin-resource-lifecycle",
   toolAction: "tool-action-receipts",

server/src/services/native-runtime/native-workspace-finalization-ownership.ts

@@ -17,6 +17,10 @@ import {
 import { currentWorkspaceWorker, requestWorkspaceWorkerStopProof, type WorkspaceWorker, type WorkspaceWorkerStopProof } from "./native-workspace-platform-proof.js";
 
 const OWNER_KEY = "nativeWorkspaceFinalizationOwner";
+
+export function nativeWorkspaceFinalizationLockKey(companyId: string, runId: string): string {
+  return `native-workspace-finalization:${companyId}:${runId}`;
+}
 const activeOwnerTokens = new Set<string>();
 // Positive same-boot evidence, retained only when durable cleanup could not be
 // confirmed. Absence from the active set alone never proves another token joined.
@@ -75,7 +79,7 @@ export async function withNativeWorkspaceFinalizationOwnership<T>(
   const finishAcquisition = beginRunPhase("execution.workspace_ownership_wait");
   try {
     return await withDedicatedDbConnection(input.db, (dedicated) => dedicated.transaction(async (lock) => {
-      const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${`native-workspace-finalization:${input.companyId}:${input.runId}`}, 0)) as acquired,
+      const rows = await lock.execute(sql`select pg_try_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(input.companyId, input.runId)}, 0)) as acquired,
         (select setting::bigint from pg_settings where name = 'idle_in_transaction_session_timeout') as idle_timeout_ms`);
       if (rows[0]?.acquired !== true) return { acquired: false } as const;
       return withWorkspaceLockKeepalive(lock as unknown as Db, Number(rows[0].idle_timeout_ms), async () => {

server/src/services/native-runtime/native-workspace-sync-retention.ts

@@ -0,0 +1,460 @@
+import { randomUUID } from "node:crypto";
+import { promises as fs } from "node:fs";
+import path from "node:path";
+import { inArray, sql } from "drizzle-orm";
+import { heartbeatRuns, type Db } from "@paperclipai/db";
+import { logger } from "../../middleware/logger.js";
+import type { createDeliveryWorkCoordinator } from "../delivery-work-coordinator.js";
+import { DELIVERY_QUEUES } from "../delivery-work-notifications.js";
+import { readProcessStartedAt } from "../hot-restart.js";
+import { workspaceExportCauses } from "../workspace-recovery-source.js";
+import { nativeWorkspaceFinalizationLockKey } from "./native-workspace-finalization-ownership.js";
+import { NATIVE_WORKSPACE_OWNER_UNVERIFIED } from "./native-workspace-finalization-recovery-state.js";
+import {
+  nativeWorkspaceSyncStateRoot,
+  onNativeWorkspaceSyncRunDirectory,
+  readNativeWorkspaceSyncReference,
+} from "./native-workspace-sync.js";
+
+export const NATIVE_WORKSPACE_SYNC_RETENTION_DEFAULT_DAYS = 7;
+export const NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS = 15 * 60 * 1000;
+
+const DAY_MS = 24 * 60 * 60 * 1000;
+const UUID_PATTERN = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
+const EXPIRED_PREFIX = ".expired-";
+const DEADLINE_BATCH = 200;
+const SETTLED_RUN_STATUSES = ["succeeded", "failed", "cancelled", "timed_out", "interrupted"];
+const RETAINED_RECOVERY_STATES = ["scheduled", "running", "awaiting_evidence"];
+const RETAINED_WORKSPACE_RECOVERY_STATUSES = ["queued", "recovering"];
+const RESTORE_RECOVERY_SCHEMA = "paperclip.workspace-restore-recovery.v1";
+
+export type NativeWorkspaceSyncRetentionOptions = {
+  retentionDays?: number;
+  dryRun?: boolean;
+  maxEntries?: number;
+  maxRemovals?: number;
+  now?: () => Date;
+  stateRoot?: string;
+  shouldContinue?: () => boolean;
+  processProbe?: ProcessProbe;
+};
+
+export type ProcessProbe = {
+  mayBeAlive: (pid: number) => boolean;
+  startedAt: (pid: number) => Promise<string | null>;
+};
+
+export type NativeWorkspaceSyncRetentionResult = {
+  checked: number;
+  candidates: number;
+  candidateBytes: number;
+  expired: number;
+  reclaimedBytes: number;
+  leftoversRemoved: number;
+  failed: number;
+  truncated: boolean;
+  skipped: Record<string, number>;
+};
+
+type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
+
+type RunRow = {
+  company_id: string;
+  status: string;
+  finished_at: string | Date | null;
+  process_pid: number | null;
+  process_group_id: number | null;
+  process_started_at: string | Date | null;
+  controller_live: boolean;
+  owner_present: boolean;
+  reference: unknown;
+};
+
+function emptyResult(): NativeWorkspaceSyncRetentionResult {
+  return {
+    checked: 0,
+    candidates: 0,
+    candidateBytes: 0,
+    expired: 0,
+    reclaimedBytes: 0,
+    leftoversRemoved: 0,
+    failed: 0,
+    truncated: false,
+    skipped: {},
+  };
+}
+
+function isExpiredLeftover(name: string) {
+  return name.startsWith(EXPIRED_PREFIX) && UUID_PATTERN.test(name.slice(EXPIRED_PREFIX.length));
+}
+
+async function isRealDirectory(target: string): Promise<boolean> {
+  const stat = await fs.lstat(target).catch(() => null);
+  return Boolean(stat?.isDirectory() && !stat.isSymbolicLink());
+}
+
+async function treeStats(target: string): Promise<{ newestMtimeMs: number; bytes: number }> {
+  const stat = await fs.lstat(target);
+  let newestMtimeMs = stat.mtimeMs;
+  let bytes = stat.blocks * 512;
+  if (stat.isDirectory()) {
+    for (const name of await fs.readdir(target)) {
+      const child = await treeStats(path.join(target, name)).catch(() => null);
+      if (!child) continue;
+      newestMtimeMs = Math.max(newestMtimeMs, child.newestMtimeMs);
+      bytes += child.bytes;
+    }
+  }
+  return { newestMtimeMs, bytes };
+}
+
+async function newestMtimeMs(target: string): Promise<number> {
+  const stat = await fs.lstat(target);
+  let newest = stat.mtimeMs;
+  if (stat.isDirectory()) {
+    for (const name of await fs.readdir(target)) {
+      newest = Math.max(newest, await newestMtimeMs(path.join(target, name)).catch(() => 0));
+    }
+  }
+  return newest;
+}
+
+async function removeTree(target: string): Promise<void> {
+  await fs.rm(target, { recursive: true, force: true });
+}
+
+function finalizationBlocker(row: {
+  phase: string;
+  lease_owner: string | null;
+  next_attempt_at: unknown;
+  recovery_state: string | null;
+} | undefined): string | null {
+  if (!row) return null;
+  if (row.lease_owner !== null) return "finalization_leased";
+  if (row.next_attempt_at !== null) return "finalization_retry_scheduled";
+  if (row.phase === "committed") return null;
+  if (row.phase !== "terminal_failure") return "finalization_unsettled";
+  if (row.recovery_state !== null && row.recovery_state !== "blocked") return "finalization_recovering";
+  return null;
+}
+
+async function retentionBlocker(
+  tx: Tx,
+  input: { runId: string; run: RunRow; settledBefore: Date; processProbe: ProcessProbe },
+): Promise<string | null> {
+  const { runId, run } = input;
+  const finishedAt = run.finished_at ? new Date(run.finished_at) : null;
+  if (!SETTLED_RUN_STATUSES.includes(run.status)) return "run_unsettled";
+  if (!finishedAt || finishedAt >= input.settledBefore) return "recently_finished";
+  if (await runProcessMayBeLive(tx, input)) return "process_live";
+  if (run.controller_live) return "controller_live";
+  if (run.owner_present) return "finalization_owner";
+  const reference = readNativeWorkspaceSyncReference(run.reference);
+  if (run.reference == null || !reference) return null;
+
+  const [finalization] = (await tx.execute(sql`
+    select phase, lease_owner, next_attempt_at, recovery_state
+    from native_run_finalizations where run_id = ${runId}
+  `)) as unknown as Array<{ phase: string; lease_owner: string | null; next_attempt_at: unknown; recovery_state: string | null }>;
+  const finalizationReason = finalizationBlocker(finalization);
+  if (finalizationReason) return finalizationReason;
+
+  const [owners] = (await tx.execute(sql`
+    select
+      exists(
+        select 1 from issue_recovery_actions
+        where company_id = ${run.company_id} and evidence->>'runId' = ${runId}
+          and status in ('active', 'escalated')
+          and cause in (${sql.join(workspaceExportCauses.map((cause) => sql`${cause}`), sql`, `)})
+      ) as export_recovery,
+      exists(
+        select 1 from issue_recovery_actions
+        where company_id = ${run.company_id} and evidence->>'runId' = ${runId}
+          and status in ('active', 'escalated') and cause = ${NATIVE_WORKSPACE_OWNER_UNVERIFIED}
+      ) as owner_unverified,
+      exists(
+        select 1 from environment_leases
+        where company_id = ${run.company_id}
+          and (heartbeat_run_id = ${runId} or id::text = ${reference.leaseId})
+          and (coalesce(metadata, '{}'::jsonb) ? 'nativeWorkspaceExportResume'
+            or metadata->>'pendingCleanupInFlight' = 'true')
+      ) as export_resume,
+      exists(
+        select 1 from environment_leases
+        where company_id = ${run.company_id} and id::text = ${reference.leaseId}
+          and metadata->'remoteExecutionTermination'->>'state' = 'destroyed'
+      ) as lease_destroyed,
+      exists(
+        select 1 from issue_recovery_actions
+        where company_id = ${run.company_id} and evidence->>'runId' = ${runId}
+          and (evidence->'systemRecovery'->>'state' in (${sql.join(RETAINED_RECOVERY_STATES.map((state) => sql`${state}`), sql`, `)})
+            or evidence->'workspaceRecovery'->>'status' in (${sql.join(RETAINED_WORKSPACE_RECOVERY_STATUSES.map((status) => sql`${status}`), sql`, `)}))
+      ) as retained_recovery,
+      exists(
+        select 1 from issue_recovery_actions
+        where company_id = ${run.company_id} and evidence->>'runId' = ${runId}
+          and not (evidence ? 'systemRecovery')
+          and (evidence->'workspaceRestoreRecovery'->>'schema' = ${RESTORE_RECOVERY_SCHEMA}
+            or cause in (${sql.join(workspaceExportCauses.map((cause) => sql`${cause}`), sql`, `)}))
+      ) or exists(
+        select 1 from heartbeat_runs r
+        where r.id = ${runId}
+          and not (coalesce(r.result_json, '{}'::jsonb) ? 'systemRecoveryAdoption')
+          and (r.result_json->'workspaceRestoreRecovery'->>'schema' = ${RESTORE_RECOVERY_SCHEMA}
+            or exists(
+              select 1 from environment_leases l
+              where l.heartbeat_run_id = r.id and l.lease_policy = 'retain_on_failure'
+                and not (coalesce(l.metadata, '{}'::jsonb) ? 'systemRecoveryAdoption')
+            ))
+      ) as backfill_pending
+  `)) as unknown as Array<Record<string, boolean>>;
+  if (owners?.export_recovery) return "export_recovery";
+  if (owners?.owner_unverified) return "owner_unverified";
+  if (owners?.export_resume) return "export_resume";
+  if (!owners?.lease_destroyed && owners?.retained_recovery) return "retained_recovery";
+  if (owners?.backfill_pending) return "recovery_backfill_pending";
+  return null;
+}
+
+const defaultProcessProbe: ProcessProbe = {
+  mayBeAlive: (pid) => {
+    try {
+      process.kill(pid, 0);
+      return true;
+    } catch (error) {
+      return (error as NodeJS.ErrnoException).code !== "ESRCH";
+    }
+  },
+  startedAt: (pid) => readProcessStartedAt(pid).catch(() => null),
+};
+
+async function runProcessMayBeLive(
+  tx: Tx,
+  input: { runId: string; run: RunRow; processProbe: ProcessProbe },
+): Promise<boolean> {
+  const { run, processProbe } = input;
+  if (run.process_pid === null && run.process_group_id === null) return false;
+  const leases = (await tx.execute(sql`
+    select provider from environment_leases where company_id = ${run.company_id} and heartbeat_run_id = ${input.runId}
+  `)) as unknown as Array<{ provider: string }>;
+  if (leases.length > 0 && leases.every((lease) => lease.provider !== "local")) return false;
+  if (run.process_group_id !== null && processProbe.mayBeAlive(-run.process_group_id)) return true;
+  if (run.process_pid === null || !processProbe.mayBeAlive(run.process_pid)) return false;
+  if (!run.process_started_at) return true;
+  const observed = await processProbe.startedAt(run.process_pid);
+  return !observed || new Date(observed).getTime() === new Date(run.process_started_at).getTime();
+}
+
+async function readRun(tx: Tx, runId: string): Promise<RunRow | undefined> {
+  const [run] = (await tx.execute(sql`
+    select company_id, status, finished_at, process_pid, process_group_id, process_started_at,
+      (controller_lease_expires_at is not null and controller_lease_expires_at >= now()) as controller_live,
+      (runner_profile_json->>'nativeWorkspaceFinalizationOwner' is not null) as owner_present,
+      runner_profile_json->'nativeWorkspaceSync' as reference
+    from heartbeat_runs where id = ${runId}
+  `)) as unknown as RunRow[];
+  return run;
+}
+
+export function nativeWorkspaceSyncRetentionService(
+  db: Db,
+  defaults: NativeWorkspaceSyncRetentionOptions = {},
+) {
+  let cursor: string | null = null;
+  let leftoverCursor: string | null = null;
+
+  function settings(options: NativeWorkspaceSyncRetentionOptions) {
+    const retentionDays = Math.max(
+      1,
+      options.retentionDays ?? defaults.retentionDays ?? NATIVE_WORKSPACE_SYNC_RETENTION_DEFAULT_DAYS,
+    );
+    return {
+      retentionMs: retentionDays * DAY_MS,
+      dryRun: options.dryRun ?? defaults.dryRun ?? false,
+      maxEntries: Math.max(1, options.maxEntries ?? defaults.maxEntries ?? 200),
+      maxRemovals: Math.max(1, options.maxRemovals ?? defaults.maxRemovals ?? 50),
+      now: (options.now ?? defaults.now ?? (() => new Date()))(),
+      root: path.resolve(options.stateRoot ?? defaults.stateRoot ?? nativeWorkspaceSyncStateRoot()),
+      shouldContinue: options.shouldContinue ?? defaults.shouldContinue ?? (() => true),
+      processProbe: options.processProbe ?? defaults.processProbe ?? defaultProcessProbe,
+    };
+  }
+
+  async function listEntries(root: string) {
+    return (await fs.readdir(root).catch(() => [] as string[])).sort();
+  }
+
+  async function expireFolder(
+    root: string,
+    name: string,
+    input: { settledBefore: Date; dryRun: boolean; processProbe: ProcessProbe },
+    result: NativeWorkspaceSyncRetentionResult,
+  ) {
+    const skip = (reason: string) => {
+      result.skipped[reason] = (result.skipped[reason] ?? 0) + 1;
+    };
+    const target = path.join(root, name);
+    if (path.dirname(target) !== root || !(await isRealDirectory(target))) return skip("unsafe_path");
+    const stats = await treeStats(target);
+    if (stats.newestMtimeMs >= input.settledBefore.getTime()) return skip("recent_files");
+
+    const decision = await db.transaction(async (tx): Promise<{ reason: string } | { trash: string | null }> => {
+      const initial = await readRun(tx, name);
+      if (initial) {
+        const [lock] = (await tx.execute(sql`
+          select pg_try_advisory_xact_lock(hashtextextended(${nativeWorkspaceFinalizationLockKey(initial.company_id, name)}, 0)) as acquired
+        `)) as unknown as Array<{ acquired: boolean }>;
+        if (lock?.acquired !== true) return { reason: "finalization_locked" };
+      }
+      const run = initial ? await readRun(tx, name) : undefined;
+      if (initial && !run) return { reason: "run_changed" };
+      if (run) {
+        const reason = await retentionBlocker(tx, {
+          runId: name,
+          run,
+          settledBefore: input.settledBefore,
+          processProbe: input.processProbe,
+        });
+        if (reason) return { reason };
+      }
+      if (input.dryRun) return { trash: null };
+      if (!(await isRealDirectory(target))) return { reason: "unsafe_path" };
+      if ((await newestMtimeMs(target)) >= input.settledBefore.getTime()) return { reason: "recent_files" };
+      const trash = path.join(root, `${EXPIRED_PREFIX}${randomUUID()}`);
+      await fs.rename(target, trash);
+      return { trash };
+    });
+    if ("reason" in decision) return skip(decision.reason);
+    result.candidates++;
+    result.candidateBytes += stats.bytes;
+    if (!decision.trash) return;
+    await removeTree(decision.trash);
+    result.expired++;
+    result.reclaimedBytes += stats.bytes;
+  }
+
+  async function tick(options: NativeWorkspaceSyncRetentionOptions = {}) {
+    const config = settings(options);
+    const result = emptyResult();
+    const settledBefore = new Date(config.now.getTime() - config.retentionMs);
+    const names = await listEntries(config.root);
+    let leftoversTruncated = false;
+    if (!config.dryRun) {
+      const leftovers = names.filter(isExpiredLeftover);
+      let attempts = 0;
+      for (const name of leftoverCursor ? leftovers.filter((entry) => entry > leftoverCursor!) : leftovers) {
+        if (!config.shouldContinue()) {
+          result.truncated = true;
+          return result;
+        }
+        if (attempts >= config.maxRemovals) {
+          leftoversTruncated = true;
+          break;
+        }
+        attempts++;
+        leftoverCursor = name;
+        try {
+          await removeTree(path.join(config.root, name));
+          result.leftoversRemoved++;
+        } catch (err) {
+          result.failed++;
+          logger.warn({ err, entry: name }, "native workspace sync retention could not remove a leftover folder");
+        }
+      }
+    }
+    if (!leftoversTruncated) leftoverCursor = null;
+    const runIds = names.filter((name) => UUID_PATTERN.test(name));
+    const pending = cursor ? runIds.filter((name) => name > cursor!) : runIds;
+    for (const name of pending) {
+      if (!config.shouldContinue() || result.checked >= config.maxEntries || result.candidates >= config.maxRemovals) {
+        result.truncated = true;
+        break;
+      }
+      result.checked++;
+      cursor = name;
+      try {
+        await expireFolder(config.root, name, { settledBefore, dryRun: config.dryRun, processProbe: config.processProbe }, result);
+      } catch (err) {
+        result.failed++;
+        logger.warn({ err, runId: name }, "native workspace sync retention skipped a folder");
+      }
+    }
+    if (!result.truncated) cursor = null;
+    result.truncated ||= leftoversTruncated;
+    return result;
+  }
+
+  async function nextDeadlineAt(options: NativeWorkspaceSyncRetentionOptions = {}): Promise<number | null> {
+    const config = settings(options);
+    const at = config.now.getTime();
+    const retryAt = at + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS;
+    const names = await listEntries(config.root);
+    let next = !config.dryRun && names.some(isExpiredLeftover) ? retryAt : Number.POSITIVE_INFINITY;
+    const runIds = names.filter((name) => UUID_PATTERN.test(name));
+    for (let offset = 0; offset < runIds.length; offset += DEADLINE_BATCH) {
+      if (!config.shouldContinue()) return Math.min(next, retryAt);
+      const folders: Array<{ runId: string; newestMtimeMs: number }> = [];
+      for (const name of runIds.slice(offset, offset + DEADLINE_BATCH)) {
+        const target = path.join(config.root, name);
+        if (!(await isRealDirectory(target))) continue;
+        const newest = await newestMtimeMs(target).catch(() => null);
+        if (newest !== null) folders.push({ runId: name, newestMtimeMs: newest });
+      }
+      if (folders.length === 0) continue;
+      const runs = await db
+        .select({ id: heartbeatRuns.id, status: heartbeatRuns.status, finishedAt: heartbeatRuns.finishedAt })
+        .from(heartbeatRuns)
+        .where(inArray(heartbeatRuns.id, folders.map((folder) => folder.runId)));
+      const byId = new Map(runs.map((run) => [run.id, run]));
+      for (const folder of folders) {
+        const run = byId.get(folder.runId);
+        const settledAt = !run
+          ? folder.newestMtimeMs
+          : SETTLED_RUN_STATUSES.includes(run.status) && run.finishedAt
+            ? Math.max(folder.newestMtimeMs, new Date(run.finishedAt).getTime())
+            : null;
+        const due = settledAt === null ? at + config.retentionMs : settledAt + config.retentionMs;
+        next = Math.min(next, due > at ? due : at + config.retentionMs);
+      }
+    }
+    return Number.isFinite(next) ? next : null;
+  }
+
+  return { tick, nextDeadlineAt };
+}
+
+export function registerNativeWorkspaceSyncRetentionWork(
+  coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
+  service: ReturnType<typeof nativeWorkspaceSyncRetentionService>,
+  input: {
+    shouldContinue?: () => boolean;
+    onResult: (result: NativeWorkspaceSyncRetentionResult) => void;
+  },
+) {
+  let next: number | null = null;
+  const worker = coordinator.register(DELIVERY_QUEUES.nativeWorkspaceSyncRetention, {
+    retryMs: NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS,
+    pausePassiveMaintenance: true,
+    run: async (signal) => {
+      next = null;
+      const shouldContinue = () => !signal.aborted && (input.shouldContinue?.() ?? true);
+      const result = await service.tick({ shouldContinue });
+      input.onResult(result);
+      if (result.truncated) {
+        next = Date.now() + 1000;
+        return;
+      }
+      const deadline = await service.nextDeadlineAt({ shouldContinue });
+      const retry = result.failed > 0 || (result.skipped.finalization_locked ?? 0) > 0
+        ? Date.now() + NATIVE_WORKSPACE_SYNC_RETENTION_RETRY_MS
+        : null;
+      next = deadline === null ? retry : retry === null ? deadline : Math.min(deadline, retry);
+    },
+    hasPending: async () => false,
+    nextRunAt: () => next,
+  });
+  const unsubscribe = onNativeWorkspaceSyncRunDirectory(() => {
+    if (next === null) worker.wake();
+  });
+  return { ...worker, unsubscribe };
+}

server/src/services/native-runtime/native-workspace-sync.ts

@@ -150,12 +150,12 @@ function requireSafeSegment(value: string, label: string): string {
   return value;
 }
 
+export function nativeWorkspaceSyncStateRoot(): string {
+  return path.join(resolvePaperclipInstanceRoot(), STATE_ROOT_NAME);
+}
+
 function descriptorDirectory(runId: string): string {
-  return path.join(
-    resolvePaperclipInstanceRoot(),
-    STATE_ROOT_NAME,
-    requireSafeSegment(runId, "run_id"),
-  );
+  return path.join(nativeWorkspaceSyncStateRoot(), requireSafeSegment(runId, "run_id"));
 }
 
 function descriptorPath(runId: string, descriptorSha256: string): string {
@@ -249,11 +249,26 @@ async function ensurePrivateDirectory(dir: string): Promise<void> {
   }
 }
 
+const runDirectoryListeners = new Set<() => void>();
+
+export function onNativeWorkspaceSyncRunDirectory(listener: () => void): () => void {
+  runDirectoryListeners.add(listener);
+  return () => {
+    runDirectoryListeners.delete(listener);
+  };
+}
+
+async function ensureRunDirectory(runId: string): Promise<string> {
+  const dir = descriptorDirectory(runId);
+  await ensurePrivateDirectory(dir);
+  for (const listener of runDirectoryListeners) listener();
+  return dir;
+}
+
 async function writeDescriptor(
   descriptor: NativeWorkspaceSyncDescriptor,
 ): Promise<NativeWorkspaceSyncReference> {
-  const dir = descriptorDirectory(descriptor.binding.runId);
-  await ensurePrivateDirectory(dir);
+  const dir = await ensureRunDirectory(descriptor.binding.runId);
   const body = `${canonicalJson(descriptor)}\n`;
   const descriptorSha256 = createHash("sha256").update(body).digest("hex");
   const finalPath = descriptorPath(descriptor.binding.runId, descriptorSha256);
@@ -913,7 +928,7 @@ export async function prepareNativeWorkspaceSync(input: {
   let mode: WorkspaceInboundMode = "host_current";
   let reuseReason: StartupReuseReason = "created";
   const seedPaths = durableSeedPaths(input.runId);
-  await measureRunPhase("workspace_sync.descriptor_directory", () => ensurePrivateDirectory(descriptorDirectory(input.runId)));
+  await measureRunPhase("workspace_sync.descriptor_directory", () => ensureRunDirectory(input.runId));
 
   try {
   if (existingReference) {