mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-16 03:31:44 +01:00
fix(retention): stop rescanning unchanged pinned workspace snapshots (#21571)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Remote workspaces keep saved versions in the
`execution_workspace_snapshots` ledger, and a retention sweep retires
versions that no reference holds
> - Every execution notification wakes this sweep, and the sweep
examined every pinned version again on each pass
> - Some pins never change. A migration cohort supersession leaves
failed operations with `lastFailure` and no `retiredSave`, and these
hold their snapshots until a separate review releases them
> - On a busy server the sweep ran four statements per pinned row, many
times each second, on a main thread that was already saturated
> - This pull request records the reference that pins a version and
leaves that version out of the candidate query until that reference
releases
> - The benefit is that an execution notification costs one candidate
query, not one transaction per pinned row
## Linked Issues or Issue Description
No public issue exists. Bug report:
**What happened?**
A hosted instance ran four statements at 13.4 calls per second each: the
workspace read, the importing-parent read and the operation read in
`pinned()`, and the locked ledger row read. The sweep in
`server/src/services/remote-workspace-snapshot-retention.ts` caused
them. A pinned superseded row keeps `status = 'superseded'` with
`delete_after = null`, so the candidate query selects it on every pass.
Each pass examines up to 50 rows. A truncated page runs again after one
second. Each run-work notification (`notifyExecutionRunWork`) sets
`rescanRequested`, which starts the scan again. The sequence stopped at
the operation read because a failed operation with `lastFailure` and no
`retiredSave` pinned these rows.
`supersedeExpiredRemoteWorkspaceMigrationCohort` writes such operations,
and they hold their snapshots permanently by design (see
`doc/DATABASE.md`, "Workspace cohort admission proofs").
**Expected behavior**
A pinned version is examined again only when the reference that pins it
can have changed. An idle or busy server does not repeat reads for
unchanged pinned history. A released version is retired on its normal
deadline.
**Steps to reproduce**
1. Create 300 superseded snapshots. Pin each with a failed
`remote_workspace_migration` operation that has `lastFailure` and no
`retiredSave`.
2. Register the retention worker on a delivery coordinator and let the
first pass finish.
3. Send `notifyExecutionRunWork` about 13 times each second for 60
seconds.
4. Count statements with `pg_stat_statements`.
**Paperclip version or commit**
`master` at 6dd0349d52
**Additional context**
The local reproduction showed the same four statements at equal call
counts, as on the hosted instance (see Verification).
## What Changed
- `pinned()` returns the first reference that holds the version (current
pointer, importing parent, operation, unfinished child, run, or recovery
action) and its ID, not a boolean.
- The sweep stores that reference in `metadata.retentionPin` when it
pins a version. It writes again only when the reference changes or a
deadline must be cleared.
- The candidate query leaves out a version with no deadline while its
recorded reference still holds. Each pin condition is one SQL builder.
`pinned()` and the candidate query use the same builder, so the
exclusion can never hold a version that `pinned()` would release.
- When the recorded reference releases, the version comes back, and
`pinned()` checks every reference again. It records the next reference
that holds, or schedules or retires the version.
- The record is removed when the sweep schedules or claims a version.
- The worker coalesces wakes for up to one second
(`REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS`). Retention deadlines are days,
so this delay has no effect on retirement.
- `doc/DATABASE.md` describes the record.
- No schema migration. The record is in the existing `metadata` JSONB
column.
## Verification
Red/green regression tests in
`server/src/services/remote-workspace-snapshot-retention.test.ts`:
- `leaves %s-pinned history out of later passes, across restarts, until
that pin is released`. This runs once for each of the six pin kinds. It
pins the version and checks that a new service instance (a restart)
examines zero rows. Then it releases the pin and checks that the version
is deleted.
- `re-examines a pinned snapshot when its recorded pin is released but
another still holds it`.
- `revisits a failed save's recorded pin once its abandoned upload is
settled`.
- `does not re-examine unchanged pinned history on execution
notifications`. This uses the coordinator and `notifyExecutionRunWork`.
On the parent commit, the first eight new tests fail (6 pin kinds, the
second-pin case, and the notification case). With the fix, all 31 tests
pass.
Related suites pass locally (212 tests):
`remote-workspaces.integration.test.ts`,
`remote-workspace-snapshot-reuse.test.ts`,
`remote-workspace-cohort-migration.test.ts`,
`snapshot-attempt-upload-abort.test.ts`, and
`remote-workspace-snapshot-retention.test.ts`. `tsc --noEmit -p
server/tsconfig.json` is clean.
Load measurement. Local PostgreSQL 18 with `pg_stat_statements`. 300
pinned rows. About 13 run-work notifications each second for 60 seconds,
after the first pass finished:
| | Before | After |
| --- | --- | --- |
| Pinned rows examined again | 30,250 – 39,000 / min | 0 / min |
| Sweep statements | ~151,000 – 196,000 / min | 88 / min |
| All statements, including the test's own notification transactions |
153,327 – 197,205 / min | 1,647 / min |
| Sweep database execution time | ~1,285 ms / min | ~52 ms / min |
| Candidate query | 0.07 ms, ~10 / min | 0.88 ms, 58 / min |
Before: `begin`, the locked row read, the workspace read, the
importing-parent read and the operation read each ran 30,250 times in 60
seconds. This is the same signature as on the hosted instance. After:
the sweep ran 58 candidate queries and 30 deadline reads. With
per-notification wakes and no coalescing, the sweep ran 776 candidate
queries and 776 deadline reads (1,552 / min).
I did not verify the change on a hosted instance.
## Risks
- A recorded pin is checked against only its own reference. The check
uses the same SQL condition as `pinned()`, so a version cannot stay
excluded after that reference releases. The parametrized test covers
each pin kind.
- A version can be pinned by more than one reference. When the recorded
reference releases, the sweep checks all of them again and records the
next one.
- A producer that rewrites the row's metadata and drops the record
causes one extra examination. A producer that sets a deadline makes the
record inactive, because the exclusion applies only when `delete_after`
is null.
- Versions that have no record yet, which includes all existing pinned
rows, are examined once after deploy. The examination writes the record.
- The candidate query does one primary-key lookup for each pinned row.
It takes about 0.9 ms with 300 pinned rows, and the time grows linearly
with the number of pinned rows.
- Pinned history from failed migration cohorts stays pinned. This change
does not alter that hold.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, local PostgreSQL)
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
1 parent
b1f608296e
commit
3c3e6c2d75
3 files changed
+232
-64
No files matched your search
@@ -1723,6 +1723,17 @@ pinned history and an empty queue do not cause recurring scans. Deletion fences
|
||||
new adoption, rechecks references, and removes every recorded upload candidate
|
||||
and winning object within the company/workspace prefix.
|
||||
|
||||
When retention finds a pinned version, it records the first reference that holds
|
||||
it in `metadata.retentionPin`: the current pointer, an importing parent, an
|
||||
operation, an unfinished child, a run, or a recovery action. The candidate query
|
||||
leaves that version out while the recorded reference still holds, so execution
|
||||
notifications do not re-examine unchanged history. When the recorded reference
|
||||
releases, the next pass checks every reference again. It records the next
|
||||
reference that holds, or schedules or retires the version. The record is durable,
|
||||
so a restarted server keeps the exclusion. A version without a record, such as
|
||||
one saved before this change, is examined once. Wakes coalesce for up to one
|
||||
second.
|
||||
|
||||
Database backups contain snapshot receipts, not S3 payloads or encryption keys.
|
||||
Disaster recovery requires the database, object store, and original master key.
|
||||
Unsaved live sandbox edits remain outside that saved-version guarantee. See
|
||||
|
||||
@@ -6,8 +6,8 @@ import { createServer } from "node:http";
|
||||
import type { AddressInfo } from "node:net";
|
||||
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { eq, sql } from "drizzle-orm";
|
||||
import { applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots,
|
||||
executionWorkspaces, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
|
||||
import { agents, applyPendingMigrations, closeRegisteredClients, companies, createDb, executionWorkspaceSnapshots,
|
||||
executionWorkspaces, heartbeatRuns, issueRecoveryActions, issues, workspaceOperations } from "@paperclipai/db";
|
||||
import { createLocalDiskStorageProvider } from "../storage/local-disk-provider.js";
|
||||
import { createStorageService } from "../storage/service.js";
|
||||
import { createS3StorageProvider } from "../storage/s3-provider.js";
|
||||
@@ -16,6 +16,7 @@ import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } fro
|
||||
import { resolveExternalTestDatabaseUrl } from "../__tests__/helpers/external-test-database.js";
|
||||
import { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { idleWorkSnapshot } from "./task-admission.js";
|
||||
import { notifyExecutionRunWork } from "./execution-control-notifications.js";
|
||||
import { issueRecoveryActionService } from "./issue-recovery-actions.js";
|
||||
import { notifyRemoteWorkspaceSnapshots, registerRemoteWorkspaceSnapshotRetentionWork, remoteWorkspaceSnapshotRetentionService } from "./remote-workspace-snapshot-retention.js";
|
||||
|
||||
@@ -108,6 +109,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
|
||||
beforeEach(async () => { root = await fs.mkdtemp(path.join(os.tmpdir(), "pc-retention-test-")); storage = createStorageService(createLocalDiskStorageProvider(root)); });
|
||||
afterEach(async () => {
|
||||
await db.delete(issueRecoveryActions); await db.delete(workspaceOperations); await db.delete(executionWorkspaceSnapshots);
|
||||
await db.delete(heartbeatRuns); await db.delete(agents);
|
||||
await db.delete(issues); await db.delete(executionWorkspaces); await db.delete(companies);
|
||||
await fs.rm(root, { recursive: true, force: true });
|
||||
vi.unstubAllEnvs();
|
||||
@@ -340,6 +342,96 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
|
||||
ownerType: "board", cause: "workspace_recovery", fingerprint: randomUUID(), nextAction: "Recover saved files" });
|
||||
expect(await owner.tick()).toMatchObject({ pinned: 1, deleted: 0 }); expect(await owner.nextDeadlineAt()).toBeNull();
|
||||
});
|
||||
type Fixture = Awaited<ReturnType<typeof seed>>;
|
||||
const failedMigration = (f: Fixture) => db.insert(workspaceOperations).values({ id: f.operationId, companyId: f.companyId,
|
||||
executionWorkspaceId: f.workspaceId, phase: "remote_workspace_migration", status: "failed",
|
||||
metadata: { lastFailure: { code: "source_expired_during_import", cohortId: randomUUID() } } });
|
||||
const pins: Record<string, { hold: (f: Fixture) => Promise<() => Promise<unknown>> }> = {
|
||||
workspace: { hold: async f => {
|
||||
await db.update(executionWorkspaces).set({ metadata: { remoteWorkspace: { snapshotId: f.row.id } } }).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
return () => db.update(executionWorkspaces).set({ metadata: {} }).where(eq(executionWorkspaces.id, f.workspaceId));
|
||||
} },
|
||||
import: { hold: async f => {
|
||||
const [parent] = await db.insert(executionWorkspaceSnapshots).values({ companyId: f.companyId, executionWorkspaceId: randomUUID(),
|
||||
sourceOperationId: randomUUID(), generation: 1, epoch: 1, status: "superseded", deleteAfter: new Date(now.getTime() + 86400_000),
|
||||
metadata: { importedWorkspaceChildren: [{ workspaceId: f.workspaceId, snapshotId: f.row.id }] } }).returning();
|
||||
return () => db.update(executionWorkspaceSnapshots).set({ status: "deleted", deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, parent!.id));
|
||||
} },
|
||||
operation: { hold: async f => {
|
||||
await failedMigration(f);
|
||||
return () => db.update(workspaceOperations).set({ metadata: { lastFailure: { code: "source_expired_during_import" },
|
||||
retiredSave: { snapshotId: randomUUID() } } }).where(eq(workspaceOperations.id, f.operationId));
|
||||
} },
|
||||
child: { hold: async f => {
|
||||
const [child] = await db.insert(executionWorkspaceSnapshots).values({ companyId: f.companyId, executionWorkspaceId: f.workspaceId,
|
||||
sourceOperationId: randomUUID(), parentSnapshotId: f.row.id, generation: 2, epoch: 1, status: "pending",
|
||||
deleteAfter: new Date(now.getTime() + 86400_000) }).returning();
|
||||
return () => db.update(executionWorkspaceSnapshots).set({ status: "deleted", deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, child!.id));
|
||||
} },
|
||||
run: { hold: async f => {
|
||||
const [agent] = await db.insert(agents).values({ companyId: f.companyId, name: "Remote worker" }).returning();
|
||||
const [run] = await db.insert(heartbeatRuns).values({ companyId: f.companyId, agentId: agent!.id, status: "running",
|
||||
runnerProfileJson: { remoteWorkspace: { workspaceId: f.workspaceId, snapshotId: f.row.id } } }).returning();
|
||||
return () => db.update(heartbeatRuns).set({ status: "succeeded" }).where(eq(heartbeatRuns.id, run!.id));
|
||||
} },
|
||||
recovery: { hold: async f => {
|
||||
const [action] = await db.insert(issueRecoveryActions).values({ companyId: f.companyId, sourceIssueId: f.issueId, kind: "active_run_watchdog",
|
||||
ownerType: "board", cause: "workspace_recovery", fingerprint: randomUUID(), nextAction: "Recover saved files" }).returning();
|
||||
return () => db.update(issueRecoveryActions).set({ status: "resolved" }).where(eq(issueRecoveryActions.id, action!.id));
|
||||
} },
|
||||
};
|
||||
it.each(Object.keys(pins))("leaves %s-pinned history out of later passes, across restarts, until that pin is released", async kind => {
|
||||
const f = await seed(), release = await pins[kind]!.hold(f);
|
||||
expect(await service().tick()).toMatchObject({ pinned: 1, deleted: 0 });
|
||||
const owner = service();
|
||||
expect(await owner.tick()).toMatchObject({ checked: 0, pinned: 0, deleted: 0 });
|
||||
expect(await owner.tick()).toMatchObject({ checked: 0 });
|
||||
expect((await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id)))[0])
|
||||
.toMatchObject({ status: "superseded", deleteAfter: null });
|
||||
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(true);
|
||||
await release();
|
||||
expect(await owner.tick()).toMatchObject({ checked: 1, pinned: 0, deleted: 1 });
|
||||
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(false);
|
||||
});
|
||||
it("re-examines a pinned snapshot when its recorded pin is released but another still holds it", async () => {
|
||||
const f = await seed();
|
||||
const releaseRun = await pins.run!.hold(f);
|
||||
const owner = service();
|
||||
expect(await owner.tick()).toMatchObject({ pinned: 1 });
|
||||
await failedMigration(f);
|
||||
expect(await owner.tick()).toMatchObject({ checked: 0 });
|
||||
await releaseRun();
|
||||
expect(await owner.tick()).toMatchObject({ checked: 1, pinned: 1, deleted: 0 });
|
||||
expect(await owner.tick()).toMatchObject({ checked: 0 });
|
||||
for (const key of f.keys) expect((await storage.headObject(f.companyId, key)).exists).toBe(true);
|
||||
});
|
||||
it("revisits a failed save's recorded pin once its abandoned upload is settled", async () => {
|
||||
const f = await seed("pending");
|
||||
await db.insert(workspaceOperations).values({ id: f.operationId, companyId: f.companyId, executionWorkspaceId: f.workspaceId,
|
||||
phase: "remote_workspace_snapshot", status: "failed", metadata: { lastFailure: { code: "remote_workspace_save_abandoned" } } });
|
||||
const owner = service();
|
||||
expect(await owner.tick()).toMatchObject({ pinned: 1 });
|
||||
expect(await owner.tick()).toMatchObject({ checked: 0 });
|
||||
const [row] = await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id));
|
||||
await db.update(executionWorkspaceSnapshots).set({ status: "delete_requested", objectKey: null,
|
||||
metadata: { ...row!.metadata, abandonedUpload: { operationId: f.operationId } } }).where(eq(executionWorkspaceSnapshots.id, f.row.id));
|
||||
expect(await owner.tick()).toMatchObject({ checked: 1, deleted: 1 });
|
||||
});
|
||||
it("does not re-examine unchanged pinned history on execution notifications", async () => {
|
||||
for (let index = 0; index < 3; index++) await failedMigration(await seed());
|
||||
const results: Array<{ checked: number; pinned: number }> = [];
|
||||
const coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => true, onError: error => { throw error; } });
|
||||
const worker = registerRemoteWorkspaceSnapshotRetentionWork(coordinator, service(), result => { results.push(result); });
|
||||
try {
|
||||
await vi.waitFor(() => expect(results.reduce((sum, result) => sum + result.pinned, 0)).toBe(3));
|
||||
const settled = results.length;
|
||||
for (let index = 1; index <= 3; index++) {
|
||||
await db.transaction(async tx => { await notifyExecutionRunWork(tx); });
|
||||
await vi.waitFor(() => expect(results.length).toBeGreaterThanOrEqual(settled + index), { timeout: 4000 });
|
||||
}
|
||||
expect(results.slice(settled).reduce((sum, result) => sum + result.checked, 0)).toBe(0);
|
||||
} finally { await worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
it("rejects a foreign upload key without deleting another company's object", async () => {
|
||||
const f = await seed(); const foreignKey = `${randomUUID()}/workspace-snapshots/foreign`;
|
||||
await db.update(executionWorkspaceSnapshots).set({ metadata: { uploads: [{ objectKey: foreignKey }] } }).where(eq(executionWorkspaceSnapshots.id, f.row.id));
|
||||
@@ -355,7 +447,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
|
||||
await new Promise(resolve => setTimeout(resolve, 30)); expect(run).toHaveBeenCalledOnce();
|
||||
const f = await seed();
|
||||
await db.transaction(async tx => { await notifyRemoteWorkspaceSnapshots(tx, `workspace:${f.workspaceId}`); });
|
||||
await vi.waitFor(async () => expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false));
|
||||
await vi.waitFor(async () => expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false), { timeout: 4000 });
|
||||
expect(run).toHaveBeenCalledTimes(2);
|
||||
} finally { await worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
@@ -407,7 +499,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
|
||||
status: "resolved", outcome: "restored", resolutionNote: "Verified saved files" });
|
||||
expect(resolved?.status).toBe("resolved");
|
||||
try {
|
||||
await vi.waitFor(() => expect(secondDeleteHeld).toBe(true));
|
||||
await vi.waitFor(() => expect(secondDeleteHeld).toBe(true), { timeout: 4000 });
|
||||
expect((await storage.headObject(f.companyId, f.keys[0]!)).exists).toBe(false);
|
||||
expect((await storage.headObject(f.companyId, f.keys[1]!)).exists).toBe(true);
|
||||
expect((await db.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, f.row.id)))[0])
|
||||
@@ -422,7 +514,7 @@ describe.skipIf(!support.supported)("remote workspace snapshot retention", () =>
|
||||
expect(await owner.nextDeadlineAt()).toBeNull();
|
||||
expect(idleWorkSnapshot().active).toBe(0);
|
||||
expect(coordinator.nextWakeAt()).toBeNull();
|
||||
});
|
||||
}, { timeout: 4000 });
|
||||
} finally { releaseSecondDelete(); await worker.stop(); await coordinator.stop(); }
|
||||
});
|
||||
});
|
||||
@@ -1,4 +1,5 @@
|
||||
import { and, asc, eq, inArray, isNotNull, isNull, lte, ne, or, sql } from "drizzle-orm";
|
||||
import { and, asc, eq, exists, inArray, isNotNull, isNull, lte, ne, or, sql, type SQL } from "drizzle-orm";
|
||||
import { alias } from "drizzle-orm/pg-core";
|
||||
import { executionWorkspaceSnapshots, executionWorkspaces, heartbeatRuns, issueRecoveryActions, workspaceOperations, type Db } from "@paperclipai/db";
|
||||
import { readRemoteWorkspaceAuthority, type EnvironmentLease } from "@paperclipai/shared";
|
||||
import { hasRemoteTerminationReceipt } from "./remote-execution-termination.js";
|
||||
@@ -10,6 +11,7 @@ import { logger } from "../middleware/logger.js";
|
||||
|
||||
export const REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS = 7 * 86400_000;
|
||||
export const REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS = 15 * 60_000;
|
||||
export const REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS = 1_000;
|
||||
const SNAPSHOT_ABORT_CONCURRENCY = 8;
|
||||
type Snapshot = typeof executionWorkspaceSnapshots.$inferSelect;
|
||||
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
|
||||
@@ -58,56 +60,112 @@ function abandonedUploadOperationId(row: Snapshot) {
|
||||
return typeof abandoned?.operationId === "string" && abandoned.operationId === row.sourceOperationId ? abandoned.operationId : null;
|
||||
}
|
||||
|
||||
async function pinned(db: ReadDb, row: Snapshot) {
|
||||
const [workspace] = await db.select({ id: executionWorkspaces.id, sourceIssueId: executionWorkspaces.sourceIssueId,
|
||||
metadata: executionWorkspaces.metadata }).from(executionWorkspaces).where(and(
|
||||
eq(executionWorkspaces.companyId, row.companyId), eq(executionWorkspaces.id, row.executionWorkspaceId),
|
||||
));
|
||||
if ((workspace?.metadata?.remoteWorkspace as { snapshotId?: unknown } | undefined)?.snapshotId === row.id) return true;
|
||||
const [importedBy] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
|
||||
eq(executionWorkspaceSnapshots.companyId, row.companyId), ne(executionWorkspaceSnapshots.status, "deleted"),
|
||||
sql`${executionWorkspaceSnapshots.metadata} ? 'importedWorkspaceChildren'`,
|
||||
sql`${executionWorkspaceSnapshots.metadata}->'importedWorkspaceChildren' @> ${JSON.stringify([
|
||||
{ workspaceId: row.executionWorkspaceId, snapshotId: row.id },
|
||||
])}::jsonb`,
|
||||
)).limit(1);
|
||||
if (importedBy) return true;
|
||||
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(and(
|
||||
eq(workspaceOperations.companyId, row.companyId), eq(workspaceOperations.executionWorkspaceId, row.executionWorkspaceId),
|
||||
or(inArray(workspaceOperations.status, ["running", "pending"]),
|
||||
// A failed save keeps its snapshot pinned for its retry, as while it ran,
|
||||
// until the final save retires it.
|
||||
sql`(${workspaceOperations.status} = 'failed' and ${workspaceOperations.metadata} ? 'lastFailure'
|
||||
and ${workspaceOperations.metadata}->'retiredSave' is null
|
||||
and ${workspaceOperations.id}::text is distinct from ${abandonedUploadOperationId(row)})`,
|
||||
sql`(${workspaceOperations.metadata}->'saveWork'->>'schema' = 'paperclip.remote-workspace-save-work.v1'
|
||||
and (${workspaceOperations.metadata}#>'{saveWork,unknown}' is distinct from 'false'::jsonb
|
||||
or case when jsonb_typeof(${workspaceOperations.metadata}#>'{saveWork,activeIds}') = 'array'
|
||||
then jsonb_array_length(${workspaceOperations.metadata}#>'{saveWork,activeIds}') <> 0 else true end))`),
|
||||
sql`(${workspaceOperations.id} = ${row.sourceOperationId}::uuid
|
||||
or ${workspaceOperations.metadata}->>'snapshotId' = ${row.id}
|
||||
or ${workspaceOperations.metadata}->>'parentSnapshotId' = ${row.id})`,
|
||||
)).limit(1);
|
||||
if (operation) return true;
|
||||
const [child] = await db.select({ id: executionWorkspaceSnapshots.id }).from(executionWorkspaceSnapshots).where(and(
|
||||
eq(executionWorkspaceSnapshots.companyId, row.companyId), eq(executionWorkspaceSnapshots.executionWorkspaceId, row.executionWorkspaceId),
|
||||
eq(executionWorkspaceSnapshots.parentSnapshotId, row.id), inArray(executionWorkspaceSnapshots.status, ["pending", "uploaded"]),
|
||||
)).limit(1);
|
||||
if (child) return true;
|
||||
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.companyId, row.companyId),
|
||||
inArray(heartbeatRuns.status, ["queued", "running"]),
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'workspaceId' = ${row.executionWorkspaceId}`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'snapshotId' = ${row.id}`,
|
||||
)).limit(1);
|
||||
if (run) return true;
|
||||
const [recovery] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(and(
|
||||
eq(issueRecoveryActions.companyId, row.companyId), inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||||
sql`(${workspace?.sourceIssueId ? sql`${issueRecoveryActions.sourceIssueId} = ${workspace.sourceIssueId}::uuid` : sql`false`}
|
||||
or ${issueRecoveryActions.evidence}->>'snapshotId' = ${row.id}
|
||||
or ${issueRecoveryActions.evidence}->>'workspaceId' = ${row.executionWorkspaceId}
|
||||
or ${issueRecoveryActions.evidence}->'remoteWorkspace'->>'snapshotId' = ${row.id})`,
|
||||
)).limit(1);
|
||||
return Boolean(recovery);
|
||||
type Pin = { kind: "workspace" } | { kind: "import" | "child"; snapshotId: string } | { kind: "operation"; operationId: string }
|
||||
| { kind: "run"; runId: string } | { kind: "recovery"; actionId: string };
|
||||
type PinRow = { companyId: SQL; id: SQL; workspaceId: SQL; sourceOperationId: SQL; abandonedOperationId: SQL; sourceIssueId: SQL; importedChild: SQL };
|
||||
const RETENTION_PIN = "retentionPin";
|
||||
const UUID_PATTERN = "^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$";
|
||||
const pinSource = alias(executionWorkspaceSnapshots, "retention_pin_source");
|
||||
const snapshot = executionWorkspaceSnapshots;
|
||||
|
||||
function literalPinRow(row: Snapshot, sourceIssueId: string | null | undefined): PinRow {
|
||||
return { companyId: sql`${row.companyId}::uuid`, id: sql`${row.id}::uuid`, workspaceId: sql`${row.executionWorkspaceId}::uuid`,
|
||||
sourceOperationId: sql`${row.sourceOperationId}::uuid`, abandonedOperationId: sql`${abandonedUploadOperationId(row)}::text`,
|
||||
sourceIssueId: sourceIssueId ? sql`${sourceIssueId}::uuid` : sql`null::uuid`,
|
||||
importedChild: sql`${JSON.stringify([{ workspaceId: row.executionWorkspaceId, snapshotId: row.id }])}::jsonb` };
|
||||
}
|
||||
|
||||
const correlatedPinRow: PinRow = {
|
||||
companyId: sql`${snapshot.companyId}`, id: sql`${snapshot.id}`, workspaceId: sql`${snapshot.executionWorkspaceId}`,
|
||||
sourceOperationId: sql`${snapshot.sourceOperationId}`,
|
||||
abandonedOperationId: sql`(case when ${snapshot.status} = 'delete_requested'
|
||||
and ${snapshot.metadata}->'abandonedUpload'->>'operationId' = ${snapshot.sourceOperationId}::text
|
||||
then ${snapshot.sourceOperationId}::text end)`,
|
||||
sourceIssueId: sql`(select w.source_issue_id from ${executionWorkspaces} w
|
||||
where w.company_id = ${snapshot.companyId} and w.id = ${snapshot.executionWorkspaceId})`,
|
||||
importedChild: sql`jsonb_build_array(jsonb_build_object('workspaceId', ${snapshot.executionWorkspaceId}::text, 'snapshotId', ${snapshot.id}::text))`,
|
||||
};
|
||||
|
||||
const importPin = (row: PinRow) => and(sql`${pinSource.companyId} = ${row.companyId}`, ne(pinSource.status, "deleted"),
|
||||
sql`${pinSource.metadata} ? 'importedWorkspaceChildren'`,
|
||||
sql`${pinSource.metadata}->'importedWorkspaceChildren' @> ${row.importedChild}`);
|
||||
|
||||
const operationPin = (row: PinRow) => and(sql`${workspaceOperations.companyId} = ${row.companyId}`,
|
||||
sql`${workspaceOperations.executionWorkspaceId} = ${row.workspaceId}`,
|
||||
or(inArray(workspaceOperations.status, ["running", "pending"]),
|
||||
sql`(${workspaceOperations.status} = 'failed' and ${workspaceOperations.metadata} ? 'lastFailure'
|
||||
and ${workspaceOperations.metadata}->'retiredSave' is null
|
||||
and ${workspaceOperations.id}::text is distinct from ${row.abandonedOperationId})`,
|
||||
sql`(${workspaceOperations.metadata}->'saveWork'->>'schema' = 'paperclip.remote-workspace-save-work.v1'
|
||||
and (${workspaceOperations.metadata}#>'{saveWork,unknown}' is distinct from 'false'::jsonb
|
||||
or case when jsonb_typeof(${workspaceOperations.metadata}#>'{saveWork,activeIds}') = 'array'
|
||||
then jsonb_array_length(${workspaceOperations.metadata}#>'{saveWork,activeIds}') <> 0 else true end))`),
|
||||
sql`(${workspaceOperations.id} = ${row.sourceOperationId}
|
||||
or ${workspaceOperations.metadata}->>'snapshotId' = ${row.id}::text
|
||||
or ${workspaceOperations.metadata}->>'parentSnapshotId' = ${row.id}::text)`);
|
||||
|
||||
const childPin = (row: PinRow) => and(sql`${pinSource.companyId} = ${row.companyId}`,
|
||||
sql`${pinSource.executionWorkspaceId} = ${row.workspaceId}`, sql`${pinSource.parentSnapshotId} = ${row.id}`,
|
||||
inArray(pinSource.status, ["pending", "uploaded"]));
|
||||
|
||||
const runPin = (row: PinRow) => and(sql`${heartbeatRuns.companyId} = ${row.companyId}`, inArray(heartbeatRuns.status, ["queued", "running"]),
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'workspaceId' = ${row.workspaceId}::text`,
|
||||
sql`${heartbeatRuns.runnerProfileJson}->'remoteWorkspace'->>'snapshotId' = ${row.id}::text`);
|
||||
|
||||
const recoveryPin = (row: PinRow) => and(sql`${issueRecoveryActions.companyId} = ${row.companyId}`,
|
||||
inArray(issueRecoveryActions.status, ["active", "escalated"]),
|
||||
sql`(${issueRecoveryActions.sourceIssueId} = ${row.sourceIssueId}
|
||||
or ${issueRecoveryActions.evidence}->>'snapshotId' = ${row.id}::text
|
||||
or ${issueRecoveryActions.evidence}->>'workspaceId' = ${row.workspaceId}::text
|
||||
or ${issueRecoveryActions.evidence}->'remoteWorkspace'->>'snapshotId' = ${row.id}::text)`);
|
||||
|
||||
async function pinned(db: ReadDb, row: Snapshot): Promise<Pin | null> {
|
||||
const [workspace] = await db.select({ sourceIssueId: executionWorkspaces.sourceIssueId, metadata: executionWorkspaces.metadata })
|
||||
.from(executionWorkspaces).where(and(eq(executionWorkspaces.companyId, row.companyId), eq(executionWorkspaces.id, row.executionWorkspaceId)));
|
||||
if ((workspace?.metadata?.remoteWorkspace as { snapshotId?: unknown } | undefined)?.snapshotId === row.id) return { kind: "workspace" };
|
||||
const ref = literalPinRow(row, workspace?.sourceIssueId);
|
||||
const [importedBy] = await db.select({ id: pinSource.id }).from(pinSource).where(importPin(ref)).limit(1);
|
||||
if (importedBy) return { kind: "import", snapshotId: importedBy.id };
|
||||
const [operation] = await db.select({ id: workspaceOperations.id }).from(workspaceOperations).where(operationPin(ref)).limit(1);
|
||||
if (operation) return { kind: "operation", operationId: operation.id };
|
||||
const [child] = await db.select({ id: pinSource.id }).from(pinSource).where(childPin(ref)).limit(1);
|
||||
if (child) return { kind: "child", snapshotId: child.id };
|
||||
const [run] = await db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(runPin(ref)).limit(1);
|
||||
if (run) return { kind: "run", runId: run.id };
|
||||
const [recovery] = await db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions).where(recoveryPin(ref)).limit(1);
|
||||
return recovery ? { kind: "recovery", actionId: recovery.id } : null;
|
||||
}
|
||||
|
||||
function recordedPinId(key: "snapshotId" | "operationId" | "runId" | "actionId") {
|
||||
const value = sql`${snapshot.metadata}->${sql.raw(`'${RETENTION_PIN}'`)}->>${sql.raw(`'${key}'`)}`;
|
||||
return sql`(case when ${value} ~* ${UUID_PATTERN} then (${value})::uuid end)`;
|
||||
}
|
||||
|
||||
function recordedPinHolds(db: Db) {
|
||||
const row = correlatedPinRow;
|
||||
return sql`(case ${snapshot.metadata}->${sql.raw(`'${RETENTION_PIN}'`)}->>'kind'
|
||||
when 'workspace' then exists (select 1 from ${executionWorkspaces} w where w.company_id = ${row.companyId}
|
||||
and w.id = ${row.workspaceId} and w.metadata->'remoteWorkspace'->>'snapshotId' = ${row.id}::text)
|
||||
when 'import' then ${exists(db.select({ id: pinSource.id }).from(pinSource).where(and(eq(pinSource.id, recordedPinId("snapshotId")), importPin(row))))}
|
||||
when 'operation' then ${exists(db.select({ id: workspaceOperations.id }).from(workspaceOperations)
|
||||
.where(and(eq(workspaceOperations.id, recordedPinId("operationId")), operationPin(row))))}
|
||||
when 'child' then ${exists(db.select({ id: pinSource.id }).from(pinSource).where(and(eq(pinSource.id, recordedPinId("snapshotId")), childPin(row))))}
|
||||
when 'run' then ${exists(db.select({ id: heartbeatRuns.id }).from(heartbeatRuns).where(and(eq(heartbeatRuns.id, recordedPinId("runId")), runPin(row))))}
|
||||
when 'recovery' then ${exists(db.select({ id: issueRecoveryActions.id }).from(issueRecoveryActions)
|
||||
.where(and(eq(issueRecoveryActions.id, recordedPinId("actionId")), recoveryPin(row))))}
|
||||
else false end)`;
|
||||
}
|
||||
|
||||
function samePin(value: unknown, pin: Pin) {
|
||||
if (!value || typeof value !== "object") return false;
|
||||
const recorded = value as Record<string, unknown>, entries = Object.entries(pin);
|
||||
return Object.keys(recorded).length === entries.length && entries.every(([key, item]) => recorded[key] === item);
|
||||
}
|
||||
|
||||
function withoutPin(metadata: Snapshot["metadata"]) {
|
||||
if (!metadata || !Object.hasOwn(metadata, RETENTION_PIN)) return {};
|
||||
const { [RETENTION_PIN]: _released, ...rest } = metadata;
|
||||
return { metadata: rest };
|
||||
}
|
||||
|
||||
/** Producers wake this owner; pinned history carries no speculative retry. */
|
||||
@@ -135,6 +193,7 @@ export function remoteWorkspaceSnapshotRetentionService(db: Db, options: {
|
||||
or(isNull(executionWorkspaceSnapshots.deleteAfter),
|
||||
lte(executionWorkspaceSnapshots.deleteAfter, at),
|
||||
eq(executionWorkspaceSnapshots.status, "committed")),
|
||||
sql`not (${executionWorkspaceSnapshots.deleteAfter} is null and ${recordedPinHolds(db)})`,
|
||||
after ? sql`${executionWorkspaceSnapshots.id} > ${after}::uuid` : undefined,
|
||||
)).orderBy(asc(executionWorkspaceSnapshots.id)).limit(batch + 1);
|
||||
result.truncated = rows.length > batch;
|
||||
@@ -144,32 +203,38 @@ export function remoteWorkspaceSnapshotRetentionService(db: Db, options: {
|
||||
const claimed = await db.transaction(async tx => {
|
||||
const [row] = await tx.select().from(executionWorkspaceSnapshots).where(eq(executionWorkspaceSnapshots.id, candidate.id)).for("update");
|
||||
if (!row || row.status === "deleted") return null;
|
||||
if (await pinned(tx, row)) {
|
||||
const pin = await pinned(tx, row);
|
||||
if (pin) {
|
||||
result.pinned++;
|
||||
const metadata = { ...row.metadata };
|
||||
const metadata: Record<string, unknown> = { ...row.metadata, [RETENTION_PIN]: pin };
|
||||
if (row.status === "superseded" && row.deleteAfter) metadata.retentionDueAt = row.deleteAfter.toISOString();
|
||||
if (row.status === "committed") delete metadata.retentionDueAt;
|
||||
if (row.deleteAfter) await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: null, metadata }).where(eq(executionWorkspaceSnapshots.id, row.id));
|
||||
if (row.deleteAfter || !samePin(row.metadata?.[RETENTION_PIN], pin)) {
|
||||
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: null, metadata }).where(eq(executionWorkspaceSnapshots.id, row.id));
|
||||
}
|
||||
return null;
|
||||
}
|
||||
const deadline = row.deleteAfter ?? (row.status === "superseded" ? date(row.metadata?.retentionDueAt) : null)
|
||||
?? new Date((row.status === "pending" || row.status === "uploaded" ? row.createdAt.getTime() : at.getTime()) + REMOTE_WORKSPACE_SNAPSHOT_GRACE_MS);
|
||||
if (row.status !== "delete_requested" && deadline > at) {
|
||||
result.scheduled++;
|
||||
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: deadline,
|
||||
await tx.update(executionWorkspaceSnapshots).set({ deleteAfter: deadline, ...withoutPin(row.metadata),
|
||||
...(row.status === "committed" ? { status: "superseded" as const } : {}) }).where(eq(executionWorkspaceSnapshots.id, row.id));
|
||||
return null;
|
||||
}
|
||||
if (row.status === "delete_requested" && row.deleteAfter && row.deleteAfter > at) return null;
|
||||
const [saved] = await tx.update(executionWorkspaceSnapshots).set({ status: "delete_requested", deleteAfter: at, lastError: null })
|
||||
const [saved] = await tx.update(executionWorkspaceSnapshots).set({ status: "delete_requested", deleteAfter: at, lastError: null, ...withoutPin(row.metadata) })
|
||||
.where(eq(executionWorkspaceSnapshots.id, row.id)).returning();
|
||||
return saved!;
|
||||
});
|
||||
if (!claimed) continue;
|
||||
// New readers cannot adopt a delete_requested record. Recheck all
|
||||
// durable references immediately before any irreversible object I/O.
|
||||
if (await pinned(db, claimed)) {
|
||||
await db.update(executionWorkspaceSnapshots).set({ deleteAfter: null }).where(eq(executionWorkspaceSnapshots.id, claimed.id));
|
||||
const pin = await pinned(db, claimed);
|
||||
if (pin) {
|
||||
await db.update(executionWorkspaceSnapshots).set({ deleteAfter: null,
|
||||
metadata: sql`coalesce(${executionWorkspaceSnapshots.metadata}, '{}'::jsonb) || ${JSON.stringify({ [RETENTION_PIN]: pin })}::jsonb` })
|
||||
.where(eq(executionWorkspaceSnapshots.id, claimed.id));
|
||||
result.pinned++; continue;
|
||||
}
|
||||
const uploads = Array.isArray(claimed.metadata?.uploads) ? claimed.metadata.uploads : [];
|
||||
@@ -226,7 +291,7 @@ export function registerRemoteWorkspaceSnapshotRetentionWork(
|
||||
) {
|
||||
let next: number | null = null;
|
||||
return coordinator.register(DELIVERY_QUEUES.remoteWorkspaceSnapshots, {
|
||||
retryMs: REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS, pausePassiveMaintenance: true,
|
||||
retryMs: REMOTE_WORKSPACE_SNAPSHOT_RETRY_MS, coalesceMs: REMOTE_WORKSPACE_SNAPSHOT_COALESCE_MS, pausePassiveMaintenance: true,
|
||||
onSettled: () => service.resetCursor(),
|
||||
run: async signal => { next = null; const result = await service.tick(signal); onResult(result);
|
||||
next = result.truncated ? Date.now() + 1000 : await service.nextDeadlineAt(); },
|
||||
|
||||
Reference in new issue
Block a user