fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row

## Linked Issues or Issue Description

No public issue exists. Bug report:

**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").

**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.

**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.

**Paperclip version or commit**
`master` at 6dd0349d52

**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).

## What Changed

- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.

## Verification

Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.

On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.

Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.

Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:

| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |

Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).

I did not verify the change on a hosted instance.

## Risks

- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-14 00:05:32 -06:00
1 parent b1f608296e
commit 3c3e6c2d75
3 files changed
+232 -64

No files matched your search

+11
View File
@@ -1723,6 +1723,17 @@ pinned history and an empty queue do not cause recurring scans. Deletion fences
new adoption, rechecks references, and removes every recorded upload candidate
and winning object within the company/workspace prefix.
When retention finds a pinned version, it records the first reference that holds
it in `metadata.retentionPin`: the current pointer, an importing parent, an
operation, an unfinished child, a run, or a recovery action. The candidate query
leaves that version out while the recorded reference still holds, so execution
notifications do not re-examine unchanged history. When the recorded reference
releases, the next pass checks every reference again. It records the next
reference that holds, or schedules or retires the version. The record is durable,
so a restarted server keeps the exclusion. A version without a record, such as
one saved before this change, is examined once. Wakes coalesce for up to one
second.
Database backups contain snapshot receipts, not S3 payloads or encryption keys.
Disaster recovery requires the database, object store, and original master key.
Unsaved live sandbox edits remain outside that saved-version guarantee. See
@@ -6,8 +6,8 @@ import { createServer } from "node:http";
import type { AddressInfo } from "node:net";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots,
executionWorkspaces, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots,
executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
import { createStorageService } from "../storage/service.js";
import { createS3StorageProvider } from "../storage/s3-provider.js";
@@ -16,6 +16,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { idleWorkSnapshot } from "./task-admission.js";
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
import { issueRecoveryActionService } from "./issue-recovery-actions.js";
import { notifyRemoteWorkspaceSnapshots, registerRemoteWorkspaceSnapshotRetentionWork, remoteWorkspaceSnapshotRetentionService } from "./remote-workspace-snapshot-retention.js";
@@ -108,6 +109,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "pc-retention-test-")); storage = createStorageService(createLocalDiskStorageProvider(root)); });
afterEach(async () => {
await db.delete(issueRecoveryActions); await db.delete(workspaceOperations); await db.delete(executionWorkspaceSnapshots);
await db.delete(heartbeatRuns); await db.delete(agents);
await db.delete(issues); await db.delete(executionWorkspaces); await db.delete(companies);
await fs.rm(root, { recursive: true, force: true });
vi.unstubAllEnvs();
@@ -340,6 +342,96 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
ownerType: "board", cause: "workspace_recovery", fingerprint: randomUUID(), nextAction: "Recover saved files" });
expect(await owner.tick()).toMatchObject({ pinned: 1, deleted: 0 }); expect(await owner.nextDeadlineAt()).toBeNull();
});
type Fixture = Awaited<ReturnType<typeof seed>>;
const failedMigration = (f: Fixture) => db.insert(workspaceOperations).values({ id: f.operationId, companyId: f.companyId,
executionWorkspaceId: f.workspaceId, phase: "remote_workspace_migration", status: "failed",
metadata: { lastFailure: { code: "source_expired_during_import", cohortId: randomUUID() } } });
const pins: Record<string, { hold: (f: Fixture) => Promise<() => Promise<unknown>> }> = {
workspace: { hold: async f => {
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { snapshotId: f.row.id } } }).where(eq(executionWorkspaces.id, f.workspaceId));
return () => db.update(executionWorkspaces).set({ metadata: {} }).where(eq(executionWorkspaces.id, f.workspaceId));
} },
import: { hold: async f => {
const [parent] = await db.insert(executionWorkspaceSnapshots).values({ companyId: f.companyId, executionWorkspaceId: randomUUID(),
sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "superseded", deleteAfter: new Date(now.getTime() + 86400_000),
metadata: { importedWorkspaceChildren: [{ workspaceId: f.workspaceId, snapshotId: f.row.id }] } }).returning();
return () => db.update(executionWorkspaceSnapshots).set({ status: "deleted", deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, parent!.id));
} },
operation: { hold: async f => {
await failedMigration(f);
return () => db.update(workspaceOperations).set({ metadata: { lastFailure: { code: "source_expired_during_import" },
retiredSave: { snapshotId: randomUUID() } } }).where(eq(workspaceOperations.id, f.operationId));
} },
child: { hold: async f => {
const [child] = await db.insert(executionWorkspaceSnapshots).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId,
sourceOperationId: randomUUID(), parentSnapshotId: f.row.id, generation: 2, epoch: 1, status: "pending",
deleteAfter: new Date(now.getTime() + 86400_000) }).returning();
return () => db.update(executionWorkspaceSnapshots).set({ status: "deleted", deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, child!.id));
} },
run: { hold: async f => {
const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Remote worker" }).returning();
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent!.id, status: "running",
runnerProfileJson: { remoteWorkspace: { workspaceId: f.workspaceId, snapshotId: f.row.id } } }).returning();
return () => db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
} },
recovery: { hold: async f => {
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
ownerType: "board", cause: "workspace_recovery", fingerprint: randomUUID(), nextAction: "Recover saved files" }).returning();
return () => db.update(issueRecoveryActions).set({ status: "resolved" }).where(eq(issueRecoveryActions.id, action!.id));
} },
};
it.each(Object.keys(pins))("leaves %s-pinned history out of later passes, across restarts, until that pin is released", async kind => {
const f = await seed(), release = await pins[kind]!.hold(f);
expect(await service().tick()).toMatchObject({ pinned: 1, deleted: 0 });
const owner = service();
expect(await owner.tick()).toMatchObject({ checked: 0, pinned: 0, deleted: 0 });
expect(await owner.tick()).toMatchObject({ checked: 0 });
expect((await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id)))[0])
.toMatchObject({ status: "superseded", deleteAfter: null });
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(true);
await release();
expect(await owner.tick()).toMatchObject({ checked: 1, pinned: 0, deleted: 1 });
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(false);
});
it("re-examines a pinned snapshot when its recorded pin is released but another still holds it", async () => {
const f = await seed();
const releaseRun = await pins.run!.hold(f);
const owner = service();
expect(await owner.tick()).toMatchObject({ pinned: 1 });
await failedMigration(f);
expect(await owner.tick()).toMatchObject({ checked: 0 });
await releaseRun();
expect(await owner.tick()).toMatchObject({ checked: 1, pinned: 1, deleted: 0 });
expect(await owner.tick()).toMatchObject({ checked: 0 });
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(true);
});
it("revisits a failed save's recorded pin once its abandoned upload is settled", async () => {
const f = await seed("pending");
await db.insert(workspaceOperations).values({ id: f.operationId, companyId: f.companyId, executionWorkspaceId: f.workspaceId,
phase: "remote_workspace_snapshot", status: "failed", metadata: { lastFailure: { code: "remote_workspace_save_abandoned" } } });
const owner = service();
expect(await owner.tick()).toMatchObject({ pinned: 1 });
expect(await owner.tick()).toMatchObject({ checked: 0 });
const [row] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id));
await db.update(executionWorkspaceSnapshots).set({ status: "delete_requested", objectKey: null,
metadata: { ...row!.metadata, abandonedUpload: { operationId: f.operationId } } }).where(eq(executionWorkspaceSnapshots.id, f.row.id));
expect(await owner.tick()).toMatchObject({ checked: 1, deleted: 1 });
});
it("does not re-examine unchanged pinned history on execution notifications", async () => {
for (let index = 0; index < 3; index++) await failedMigration(await seed());
const results: Array<{ checked: number; pinned: number }> = [];
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
const worker = registerRemoteWorkspaceSnapshotRetentionWork(coordinator, service(), result => { results.push(result); });
try {
await vi.waitFor(() => expect(results.reduce((sum, result) => sum + result.pinned, 0)).toBe(3));
const settled = results.length;
for (let index = 1; index <= 3; index++) {
await db.transaction(async tx => { await notifyExecutionRunWork(tx); });
await vi.waitFor(() => expect(results.length).toBeGreaterThanOrEqual(settled + index), { timeout: 4000 });
}
expect(results.slice(settled).reduce((sum, result) => sum + result.checked, 0)).toBe(0);
} finally { await worker.stop(); await coordinator.stop(); }
});
it("rejects a foreign upload key without deleting another company's object", async () => {
const f = await seed(); const foreignKey = `${randomUUID()}/workspace-snapshots/foreign`;
await db.update(executionWorkspaceSnapshots).set({ metadata: { uploads: [{ objectKey: foreignKey }] } }).where(eq(executionWorkspaceSnapshots.id, f.row.id));
@@ -355,7 +447,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
await new Promise(resolve => setTimeout(resolve, 30)); expect(run).toHaveBeenCalledOnce();
const f = await seed();
await db.transaction(async tx => { await notifyRemoteWorkspaceSnapshots(tx, `workspace:${f.workspaceId}`); });
await vi.waitFor(async () => expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false));
await vi.waitFor(async () => expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false), { timeout: 4000 });
expect(run).toHaveBeenCalledTimes(2);
} finally { await worker.stop(); await coordinator.stop(); }
});
@@ -407,7 +499,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
status: "resolved", outcome: "restored", resolutionNote: "Verified saved files" });
expect(resolved?.status).toBe("resolved");
try {
await vi.waitFor(() => expect(secondDeleteHeld).toBe(true));
await vi.waitFor(() => expect(secondDeleteHeld).toBe(true), { timeout: 4000 });
expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false);
expect((await storage.headObject(f.companyId, f.keys[1]!)).exists).toBe(true);
expect((await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id)))[0])
@@ -422,7 +514,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
expect(await owner.nextDeadlineAt()).toBeNull();
expect(idleWorkSnapshot().active).toBe(0);
expect(coordinator.nextWakeAt()).toBeNull();
});
}, { timeout: 4000 });
} finally { releaseSecondDelete(); await worker.stop(); await coordinator.stop(); }
});
});
@@ -1,4 +1,5 @@
import { and, asc, eq, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm";
import { and, asc, eq, exists, inArray, isNotNull, isNull, lte, ne, or, sql, type SQL } from "drizzle-orm";
import { alias } from "drizzle-orm/pg-core";
import { executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db";
import { readRemoteWorkspaceAuthority, type EnvironmentLease } from "@paperclipai/shared";
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
@@ -10,6 +11,7 @@ import { logger } from "../middleware/logger.js";
export const REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS = 7 * 86400_000;
export const REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS = 15 * 60_000;
export const REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS = 1_000;
const SNAPSHOT_ABORT_CONCURRENCY = 8;
type Snapshot = typeof executionWorkspaceSnapshots.$inferSelect;
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
@@ -58,56 +60,112 @@ function abandonedUploadOperationId(row: Snapshot) {
return typeof abandoned?.operationId === "string" && abandoned.operationId === row.sourceOperationId ? abandoned.operationId : null;
}
async function pinned(db: ReadDb, row: Snapshot) {
const [workspace] = await db.select({ id: executionWorkspaces.id, sourceIssueId: executionWorkspaces.sourceIssueId,
metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
eq(executionWorkspaces.companyId, row.companyId), eq(executionWorkspaces.id, row.executionWorkspaceId),
));
if ((workspace?.metadata?.remoteWorkspace as { snapshotId?: unknown } | undefined)?.snapshotId === row.id) return true;
const [importedBy] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, row.companyId), ne(executionWorkspaceSnapshots.status, "deleted"),
sql`${executionWorkspaceSnapshots.metadata} ? 'importedWorkspaceChildren'`,
sql`${executionWorkspaceSnapshots.metadata}->'importedWorkspaceChildren' @> ${JSON.stringify([
{ workspaceId: row.executionWorkspaceId, snapshotId: row.id },
])}::jsonb`,
)).limit(1);
if (importedBy) return true;
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
eq(workspaceOperations.companyId, row.companyId), eq(workspaceOperations.executionWorkspaceId, row.executionWorkspaceId),
or(inArray(workspaceOperations.status, ["running", "pending"]),
// A failed save keeps its snapshot pinned for its retry, as while it ran,
// until the final save retires it.
sql`(${workspaceOperations.status} = 'failed' and ${workspaceOperations.metadata} ? 'lastFailure'
and ${workspaceOperations.metadata}->'retiredSave' is null
and ${workspaceOperations.id}::text is distinct from ${abandonedUploadOperationId(row)})`,
sql`(${workspaceOperations.metadata}->'saveWork'->>'schema' = 'paperclip.remote-workspace-save-work.v1'
and (${workspaceOperations.metadata}#>'{saveWork,unknown}' is distinct from 'false'::jsonb
or case when jsonb_typeof(${workspaceOperations.metadata}#>'{saveWork,activeIds}') = 'array'
then jsonb_array_length(${workspaceOperations.metadata}#>'{saveWork,activeIds}') <> 0 else true end))`),
sql`(${workspaceOperations.id} = ${row.sourceOperationId}::uuid
or ${workspaceOperations.metadata}->>'snapshotId' = ${row.id}
or ${workspaceOperations.metadata}->>'parentSnapshotId' = ${row.id})`,
)).limit(1);
if (operation) return true;
const [child] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
eq(executionWorkspaceSnapshots.companyId, row.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, row.executionWorkspaceId),
eq(executionWorkspaceSnapshots.parentSnapshotId, row.id), inArray(executionWorkspaceSnapshots.status, ["pending", "uploaded"]),
)).limit(1);
if (child) return true;
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, row.companyId),
inArray(heartbeatRuns.status, ["queued", "running"]),
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'workspaceId' = ${row.executionWorkspaceId}`,
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'snapshotId' = ${row.id}`,
)).limit(1);
if (run) return true;
const [recovery] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
eq(issueRecoveryActions.companyId, row.companyId), inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`(${workspace?.sourceIssueId ? sql`${issueRecoveryActions.sourceIssueId} = ${workspace.sourceIssueId}::uuid` : sql`false`}
or ${issueRecoveryActions.evidence}->>'snapshotId' = ${row.id}
or ${issueRecoveryActions.evidence}->>'workspaceId' = ${row.executionWorkspaceId}
or ${issueRecoveryActions.evidence}->'remoteWorkspace'->>'snapshotId' = ${row.id})`,
)).limit(1);
return Boolean(recovery);
type Pin = { kind: "workspace" } | { kind: "import" | "child"; snapshotId: string } | { kind: "operation"; operationId: string }
| { kind: "run"; runId: string } | { kind: "recovery"; actionId: string };
type PinRow = { companyId: SQL; id: SQL; workspaceId: SQL; sourceOperationId: SQL; abandonedOperationId: SQL; sourceIssueId: SQL; importedChild: SQL };
const RETENTION_PIN = "retentionPin";
const UUID_PATTERN = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
const pinSource = alias(executionWorkspaceSnapshots, "retention_pin_source");
const snapshot = executionWorkspaceSnapshots;
function literalPinRow(row: Snapshot, sourceIssueId: string | null | undefined): PinRow {
return { companyId: sql`${row.companyId}::uuid`, id: sql`${row.id}::uuid`, workspaceId: sql`${row.executionWorkspaceId}::uuid`,
sourceOperationId: sql`${row.sourceOperationId}::uuid`, abandonedOperationId: sql`${abandonedUploadOperationId(row)}::text`,
sourceIssueId: sourceIssueId ? sql`${sourceIssueId}::uuid` : sql`null::uuid`,
importedChild: sql`${JSON.stringify([{ workspaceId: row.executionWorkspaceId, snapshotId: row.id }])}::jsonb` };
}
const correlatedPinRow: PinRow = {
companyId: sql`${snapshot.companyId}`, id: sql`${snapshot.id}`, workspaceId: sql`${snapshot.executionWorkspaceId}`,
sourceOperationId: sql`${snapshot.sourceOperationId}`,
abandonedOperationId: sql`(case when ${snapshot.status} = 'delete_requested'
and ${snapshot.metadata}->'abandonedUpload'->>'operationId' = ${snapshot.sourceOperationId}::text
then ${snapshot.sourceOperationId}::text end)`,
sourceIssueId: sql`(select w.source_issue_id from ${executionWorkspaces} w
where w.company_id = ${snapshot.companyId} and w.id = ${snapshot.executionWorkspaceId})`,
importedChild: sql`jsonb_build_array(jsonb_build_object('workspaceId', ${snapshot.executionWorkspaceId}::text, 'snapshotId', ${snapshot.id}::text))`,
};
const importPin = (row: PinRow) => and(sql`${pinSource.companyId} = ${row.companyId}`, ne(pinSource.status, "deleted"),
sql`${pinSource.metadata} ? 'importedWorkspaceChildren'`,
sql`${pinSource.metadata}->'importedWorkspaceChildren' @> ${row.importedChild}`);
const operationPin = (row: PinRow) => and(sql`${workspaceOperations.companyId} = ${row.companyId}`,
sql`${workspaceOperations.executionWorkspaceId} = ${row.workspaceId}`,
or(inArray(workspaceOperations.status, ["running", "pending"]),
sql`(${workspaceOperations.status} = 'failed' and ${workspaceOperations.metadata} ? 'lastFailure'
and ${workspaceOperations.metadata}->'retiredSave' is null
and ${workspaceOperations.id}::text is distinct from ${row.abandonedOperationId})`,
sql`(${workspaceOperations.metadata}->'saveWork'->>'schema' = 'paperclip.remote-workspace-save-work.v1'
and (${workspaceOperations.metadata}#>'{saveWork,unknown}' is distinct from 'false'::jsonb
or case when jsonb_typeof(${workspaceOperations.metadata}#>'{saveWork,activeIds}') = 'array'
then jsonb_array_length(${workspaceOperations.metadata}#>'{saveWork,activeIds}') <> 0 else true end))`),
sql`(${workspaceOperations.id} = ${row.sourceOperationId}
or ${workspaceOperations.metadata}->>'snapshotId' = ${row.id}::text
or ${workspaceOperations.metadata}->>'parentSnapshotId' = ${row.id}::text)`);
const childPin = (row: PinRow) => and(sql`${pinSource.companyId} = ${row.companyId}`,
sql`${pinSource.executionWorkspaceId} = ${row.workspaceId}`, sql`${pinSource.parentSnapshotId} = ${row.id}`,
inArray(pinSource.status, ["pending", "uploaded"]));
const runPin = (row: PinRow) => and(sql`${heartbeatRuns.companyId} = ${row.companyId}`, inArray(heartbeatRuns.status, ["queued", "running"]),
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'workspaceId' = ${row.workspaceId}::text`,
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'snapshotId' = ${row.id}::text`);
const recoveryPin = (row: PinRow) => and(sql`${issueRecoveryActions.companyId} = ${row.companyId}`,
inArray(issueRecoveryActions.status, ["active", "escalated"]),
sql`(${issueRecoveryActions.sourceIssueId} = ${row.sourceIssueId}
or ${issueRecoveryActions.evidence}->>'snapshotId' = ${row.id}::text
or ${issueRecoveryActions.evidence}->>'workspaceId' = ${row.workspaceId}::text
or ${issueRecoveryActions.evidence}->'remoteWorkspace'->>'snapshotId' = ${row.id}::text)`);
async function pinned(db: ReadDb, row: Snapshot): Promise<Pin | null> {
const [workspace] = await db.select({ sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata })
.from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, row.companyId), eq(executionWorkspaces.id, row.executionWorkspaceId)));
if ((workspace?.metadata?.remoteWorkspace as { snapshotId?: unknown } | undefined)?.snapshotId === row.id) return { kind: "workspace" };
const ref = literalPinRow(row, workspace?.sourceIssueId);
const [importedBy] = await db.select({ id: pinSource.id }).from(pinSource).where(importPin(ref)).limit(1);
if (importedBy) return { kind: "import", snapshotId: importedBy.id };
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(operationPin(ref)).limit(1);
if (operation) return { kind: "operation", operationId: operation.id };
const [child] = await db.select({ id: pinSource.id }).from(pinSource).where(childPin(ref)).limit(1);
if (child) return { kind: "child", snapshotId: child.id };
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(runPin(ref)).limit(1);
if (run) return { kind: "run", runId: run.id };
const [recovery] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(recoveryPin(ref)).limit(1);
return recovery ? { kind: "recovery", actionId: recovery.id } : null;
}
function recordedPinId(key: "snapshotId" | "operationId" | "runId" | "actionId") {
const value = sql`${snapshot.metadata}->${sql.raw(`'${RETENTION_PIN}'`)}->>${sql.raw(`'${key}'`)}`;
return sql`(case when ${value} ~* ${UUID_PATTERN} then (${value})::uuid end)`;
}
function recordedPinHolds(db: Db) {
const row = correlatedPinRow;
return sql`(case ${snapshot.metadata}->${sql.raw(`'${RETENTION_PIN}'`)}->>'kind'
when 'workspace' then exists (select 1 from ${executionWorkspaces} w where w.company_id = ${row.companyId}
and w.id = ${row.workspaceId} and w.metadata->'remoteWorkspace'->>'snapshotId' = ${row.id}::text)
when 'import' then ${exists(db.select({ id: pinSource.id }).from(pinSource).where(and(eq(pinSource.id, recordedPinId("snapshotId")), importPin(row))))}
when 'operation' then ${exists(db.select({ id: workspaceOperations.id }).from(workspaceOperations)
.where(and(eq(workspaceOperations.id, recordedPinId("operationId")), operationPin(row))))}
when 'child' then ${exists(db.select({ id: pinSource.id }).from(pinSource).where(and(eq(pinSource.id, recordedPinId("snapshotId")), childPin(row))))}
when 'run' then ${exists(db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, recordedPinId("runId")), runPin(row))))}
when 'recovery' then ${exists(db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
.where(and(eq(issueRecoveryActions.id, recordedPinId("actionId")), recoveryPin(row))))}
else false end)`;
}
function samePin(value: unknown, pin: Pin) {
if (!value || typeof value !== "object") return false;
const recorded = value as Record<string, unknown>, entries = Object.entries(pin);
return Object.keys(recorded).length === entries.length && entries.every(([key, item]) => recorded[key] === item);
}
function withoutPin(metadata: Snapshot["metadata"]) {
if (!metadata || !Object.hasOwn(metadata, RETENTION_PIN)) return {};
const { [RETENTION_PIN]: _released, ...rest } = metadata;
return { metadata: rest };
}
/** Producers wake this owner; pinned history carries no speculative retry. */
@@ -135,6 +193,7 @@ export function remoteWorkspaceSnapshotRetentionService(db: Db, options: {
or(isNull(executionWorkspaceSnapshots.deleteAfter),
lte(executionWorkspaceSnapshots.deleteAfter, at),
eq(executionWorkspaceSnapshots.status, "committed")),
sql`not (${executionWorkspaceSnapshots.deleteAfter} is null and ${recordedPinHolds(db)})`,
after ? sql`${executionWorkspaceSnapshots.id} > ${after}::uuid` : undefined,
)).orderBy(asc(executionWorkspaceSnapshots.id)).limit(batch + 1);
result.truncated = rows.length > batch;
@@ -144,32 +203,38 @@ export function remoteWorkspaceSnapshotRetentionService(db: Db, options: {
const claimed = await db.transaction(async tx => {
const [row] = await tx.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, candidate.id)).for("update");
if (!row || row.status === "deleted") return null;
if (await pinned(tx, row)) {
const pin = await pinned(tx, row);
if (pin) {
result.pinned++;
const metadata = { ...row.metadata };
const metadata: Record<string, unknown> = { ...row.metadata, [RETENTION_PIN]: pin };
if (row.status === "superseded" && row.deleteAfter) metadata.retentionDueAt = row.deleteAfter.toISOString();
if (row.status === "committed") delete metadata.retentionDueAt;
if (row.deleteAfter) await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: null, metadata }).where(eq(executionWorkspaceSnapshots.id, row.id));
if (row.deleteAfter || !samePin(row.metadata?.[RETENTION_PIN], pin)) {
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: null, metadata }).where(eq(executionWorkspaceSnapshots.id, row.id));
}
return null;
}
const deadline = row.deleteAfter ?? (row.status === "superseded" ? date(row.metadata?.retentionDueAt) : null)
?? new Date((row.status === "pending" || row.status === "uploaded" ? row.createdAt.getTime() : at.getTime()) + REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS);
if (row.status !== "delete_requested" && deadline > at) {
result.scheduled++;
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: deadline,
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: deadline, ...withoutPin(row.metadata),
...(row.status === "committed" ? { status: "superseded" as const } : {}) }).where(eq(executionWorkspaceSnapshots.id, row.id));
return null;
}
if (row.status === "delete_requested" && row.deleteAfter && row.deleteAfter > at) return null;
const [saved] = await tx.update(executionWorkspaceSnapshots).set({ status: "delete_requested", deleteAfter: at, lastError: null })
const [saved] = await tx.update(executionWorkspaceSnapshots).set({ status: "delete_requested", deleteAfter: at, lastError: null, ...withoutPin(row.metadata) })
.where(eq(executionWorkspaceSnapshots.id, row.id)).returning();
return saved!;
});
if (!claimed) continue;
// New readers cannot adopt a delete_requested record. Recheck all
// durable references immediately before any irreversible object I/O.
if (await pinned(db, claimed)) {
await db.update(executionWorkspaceSnapshots).set({ deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, claimed.id));
const pin = await pinned(db, claimed);
if (pin) {
await db.update(executionWorkspaceSnapshots).set({ deleteAfter: null,
metadata: sql`coalesce(${executionWorkspaceSnapshots.metadata}, '{}'::jsonb) || ${JSON.stringify({ [RETENTION_PIN]: pin })}::jsonb` })
.where(eq(executionWorkspaceSnapshots.id, claimed.id));
result.pinned++; continue;
}
const uploads = Array.isArray(claimed.metadata?.uploads) ? claimed.metadata.uploads : [];
@@ -226,7 +291,7 @@ export function registerRemoteWorkspaceSnapshotRetentionWork(
) {
let next: number | null = null;
return coordinator.register(DELIVERY_QUEUES.remoteWorkspaceSnapshots, {
retryMs: REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS, pausePassiveMaintenance: true,
retryMs: REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS, coalesceMs: REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS, pausePassiveMaintenance: true,
onSettled: () => service.resetCursor(),
run: async signal => { next = null; const result = await service.tick(signal); onResult(result);
next = result.truncated ? Date.now() + 1000 : await service.nextDeadlineAt(); },