mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-10-23 17:33:51 +02:00
feat: track git workspace restore points and apply their retention (#17241)
## Thinking Path
> - Paperclip is the open source app people use to manage AI agents for
work
> - Closed issues' git worktrees pile up on disk because they can only
be removed once the remote reproduces them exactly
> - #17238 adds restore points: encrypted saves of the unpushed and
uncommitted parts of a worktree, so the worktree itself can be removed
> - Those saves live in file storage and must be deleted on a schedule,
and promptly when their issue or company is deleted
> - This pull request adds the table that tracks restore points and a
deadline-driven retention worker, with nothing creating restore points
yet
> - The benefit is a reviewable lifecycle foundation before capture is
wired into expiration
## Linked Issues or Issue Description
Refs #17238 (restore point library). Bug report described in-PR (bug
report template).
**What happened?**
Closed issues' worktrees accumulate without bound. For example, one
instance's last 111 git cache expiration checks removed nothing, because
each worktree held unpushed commits or uncommitted files. Restore points
(#17238) will let those worktrees be removed safely. That needs a
durable record of every stored restore point and a retention policy for
deleting them.
**Expected behavior**
- A restore point is kept while its work can still resume.
- It is deleted a fixed time after its issue tree closes.
- It is deleted promptly when its issue or company is deleted.
**Steps to reproduce**
Not applicable on its own: this adds the lifecycle that the capture
integration will use.
**Paperclip version or commit:** `master` at def4726bea.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).
## What Changed
**New table `execution_workspace_restore_points`** (migration `0372`,
schema `packages/db/src/schema/execution_workspace_restore_points.ts`):
- **Columns:** storage key, sha256, size, generation, source issue,
`status`, closing time, review/deletion time (`delete_after`), a
`review_generation` counter and last error.
- **Statuses:** `pending`, `active`, `expiring`, `superseded`,
`delete_requested`, `deleted`.
- **No foreign keys:** a row outlives the deletion of its workspace,
issue or company long enough to remove the stored object.
- **Indexes:**
- the storage key is unique;
- live generations are unique per workspace;
- an index on status and deadline serves the worker, and a partial index
on active rows' source issue serves the trigger.
**Issue tree trigger** (same migration):
`paperclip_restore_point_tree_changed` runs after an issue's `status` or
`parent_id` changes, and after an issue is deleted. It walks up from the
issue and from its old and new parents (at most 64 levels). For every
active restore point whose source issue is in that set, it makes the row
due now (keeping an earlier due time) and increments
`review_generation`. This happens in the same transaction, so a child's
deletion or move still reaches its former parent's tree.
**New `server/src/services/git-restore-point-ledger.ts`:**
- **`gitRestorePointRetentionService(db, { storage, retentionDays,
beforeExpire })`.** Each `tick` handles only due work, in batches of 50:
- **Review:** an active restore point is checked when its `delete_after`
time arrives. The due check compares against the later of the worker's
clock and the database clock, so a row the trigger just marked is never
skipped. At review:
- **Tree still open:** clear `delete_after`. The trigger marks it due
again on the next relevant issue change, so open trees cause no
recurring checks.
- **Tree `done` or `cancelled`:** expire
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS` (default `90`, minimum `1`)
after the tree's latest close. Expiry moves the row to `expiring`, then
runs an optional `beforeExpire` hook, which the capture integration will
use to mark the workspace no longer restorable. Only then does it
request deletion. An `expiring` row's object is never deleted: after a
restart, or if the hook fails, the hook is retried (after 15 minutes
when it failed).
- **Source issue gone:** delete.
- **Concurrency:** a review result is written only if
`review_generation` is unchanged since the worker read the row. If the
trigger fired in between (for example, the tree closed or reopened), the
write is skipped and the row stays due for the next run. Expiry and
orphan deletion claim the row under the same check before the expiry
hook runs.
- **Abandoned captures:** a `pending` row older than one hour is
deleted.
- **Deletion:** `superseded` and `delete_requested` rows have their
object removed. They become `deleted` only once `headObject` reports the
object gone. Failures, including a local unlink error the provider
swallows, are recorded in `last_error` and retried after 15 minutes.
- **`nextDeadlineAt`** returns the earliest real deadline, or `null`
when nothing is due.
- **`requestRestorePointDeletion(tx, { sourceIssueId } | { companyId
})`** registers a delivery-work notification and marks the matching
`pending`/`active` rows `delete_requested` in the caller's transaction.
**Deletion hooks:**
- `issueService.remove` and `deleteCompany` call
`requestRestorePointDeletion` in their transactions, so a deleted
issue's or company's objects are removed promptly.
- Child issues cannot be deleted with their parent (the parent link
restricts deletion), so each issue removal goes through this hook.
**Scheduling** follows `rules/sleep-aware-background-work.mdc`:
- `registerGitRestorePointWork` registers a `git-restore-points` queue
with the shared delivery-work coordinator.
- The worker keeps no in-memory inputs. Due work is recorded in the
table, by the trigger or by deletion requests. Each run handles it and
schedules only `nextDeadlineAt`, so a failed run loses nothing and is
retried by the coordinator.
- It runs at startup, at real deadlines, and when terminal-workspace
notifications settle after issue changes, but only while restore points
exist. With nothing due, it schedules nothing. A full batch continues
immediately. There is no timer.
**Config and docs:**
- `server/src/config.ts` adds
`PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`.
- `doc/recoverable-disk-cleanup.md` gains a "Git workspace restore
points" section; `doc/DEVELOPING.md` documents the variable.
- `drizzle-kit generate` pruned `meta/0367_snapshot.json`, following the
repository's snapshot-retention convention.
- No code comments were added.
## Verification
- `pnpm exec vitest run src/__tests__/git-restore-point-ledger.test.ts`
(in `server/`): 26 passed against embedded Postgres, stable across
repeated runs. The cases:
- **Review outcomes:** an open tree clears its deadline and schedules
nothing; a closed tree is scheduled 90 days after its close; a
long-closed tree expires and is deleted; a missing source issue is
deleted.
- **Expiry durability:** a failing hook keeps the row `expiring` with
its object until a retry succeeds; an `expiring` row left by a restart
resumes the hook before deletion; an `expiring` row's object is never
deleted while its hook fails.
- **Trigger:** the last open child closing, a child being deleted, a
child moving to another parent, and the source issue's own status change
all mark the restore point due. Unrelated issues, inactive rows and
already-due rows are untouched. End to end, an open tree's restore point
waits without a deadline, then is scheduled 90 days after its tree
closes.
- **Races:** closing the tree, or reopening it, between the worker's
read and write leaves the row due, and the next run resolves it
correctly. A reopened tree's restore point is not deleted.
- **No rescanning:** 60 retained restore points with future review times
are not evaluated. A backlog of 60 due ones is processed as 50, then 10,
then nothing.
- **Captures and deletion:** abandoned pending rows are deleted while
recent ones are kept; superseded rows are deleted; an object still
present after deletion is not marked deleted; a failed deletion is
retried only after its 15-minute deadline.
- **Deadlines:** the next deadline is correct for each obligation, and
`null` when idle.
- **Deletion hooks:** `issueService.remove` marks only that issue's
restore points, and `deleteCompany` marks all of the company's; the
worker then deletes their objects.
- **Worker:** schedules nothing when idle, and only the next deadline
otherwise; wakes on issue changes only while restore points exist.
- **Mutation checks:** each of these fails at least one case:
- removing the issue hook or the company hook;
- making the trigger ignore former parents;
- removing the worker's generation check or the trigger's generation
bump;
- letting storage cleanup delete `expiring` rows, or not resuming the
hook;
- ignoring open trees;
- removing the deletion verification;
- evaluating rows that are not due;
- dropping the retry deadline.
- `pnpm exec vitest run company-deletion issue-delete issues-service
delivery-work issue-tree` (in `server/`): 190 passed.
- **CI note:** one earlier `Typecheck + Release Registry` run was
cancelled when the GitHub runner received a shutdown signal; it is not a
code failure.
- `pnpm exec vitest run` (in `packages/db`): 211 passed. One run had a
single failure in `issue-comment-derived-attribution-migration.test.ts`,
an older migration's test unrelated to this change. It passed alone and
on the full re-run.
- `pnpm exec vitest run
src/__tests__/git-restore-point-ledger-config.test.ts`: 5 passed.
- `pnpm run check:migrations` (in `packages/db`): numbering and safety
checks pass.
- `pnpm run typecheck` (in `server/`): pass.
## Risks
- **No behavior change today:** nothing creates restore points yet. The
worker runs once at startup, finds no rows and schedules nothing. The
capture integration follows in a separate PR.
- **Trigger cost:** every issue status change, parent change or deletion
now runs a short ancestor walk, plus an `UPDATE` on the new table that
matches only active restore points through a partial index. With no
restore points, the `UPDATE` matches nothing.
- **Missed wakes:** if a wake is ever missed, for example because the
process exits after commit, the due row stays in the table. The worker's
startup run or next deadline processes it.
- **Deletion paths:** issue and company deletion each gain one `UPDATE`
on the new table inside their existing transactions.
- **Migration:** `0372_mature_the_initiative.sql` creates one empty
table, its indexes, and the trigger function with two triggers on
`issues`; nothing is rewritten.
- **Rollback:** revert. The table can stay, since nothing else reads it.
## Model Used
- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, read-only measurements on live
instances).
## Checklist
- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge
---------
Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
1 parent
341b622604
commit
12bb7e1976
15 files changed
+6096
-4701
No files matched your search
@@ -2580,6 +2580,10 @@ Environment overrides:
|
||||
negative, empty, or non-numeric value falls back to the default.
|
||||
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED=false` turns expiry off, and
|
||||
`PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN=true` only reports candidates.
|
||||
- `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS=<days>` sets how long a git workspace
|
||||
restore point is kept after its issue tree closes (see
|
||||
`doc/recoverable-disk-cleanup.md`). The default is `90` and the minimum is `1`. A
|
||||
negative, empty, or non-numeric value falls back to the default.
|
||||
|
||||
Without `PAPERCLIP_DB_BACKUP_ALERT_FILE`, health checks look for
|
||||
`db-backup-to-s3.failure` in the backup directory, beside the backup directory,
|
||||
|
||||
@@ -92,3 +92,42 @@ The rest of the session directory, including its control-plane state, is kept.
|
||||
|
||||
Each batch that finds candidates logs them with candidate and reclaimed allocated
|
||||
bytes, cleared task sessions, and skip reasons.
|
||||
|
||||
## Git workspace restore points
|
||||
|
||||
Restore points are encrypted saves of the parts of a git worktree that its remote
|
||||
cannot reproduce: unpushed commits and uncommitted files. They let a closed issue's
|
||||
worktree be removed and restored later. The table `execution_workspace_restore_points`
|
||||
tracks each one with its storage key, hash, size and status
|
||||
(`pending`, `active`, `expiring`, `superseded`, `delete_requested`, `deleted`). The objects live in
|
||||
the instance's configured file storage (`local_disk` or `s3`).
|
||||
|
||||
An active restore point's `delete_after` is either its expiry time or empty while its
|
||||
source issue tree is open. A database trigger on `issues` keeps this current: when an
|
||||
issue's status or parent changes, or an issue is deleted, it marks every active restore
|
||||
point whose source issue is that issue, its old or new parent, or any of their ancestors
|
||||
as due now, in the same transaction. A retention worker on the shared delivery-work
|
||||
scheduler runs at startup, when issue changes settle while restore points exist, and at
|
||||
real deadlines: the earliest expiry, the end of an abandoned capture's grace period, or
|
||||
a failed deletion's retry. With nothing due, it schedules nothing, and open trees cause
|
||||
no recurring checks. At review it checks the source issue tree:
|
||||
|
||||
- Tree still open: clear `delete_after` and wait for the next issue change.
|
||||
- Tree `done` or `cancelled`: expire `PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS`
|
||||
(default `90`, minimum `1`) after the tree's latest close, immediately if that time
|
||||
has passed. Expiry first moves the row to `expiring`, runs the expiry hook, and only
|
||||
then requests deletion. An `expiring` row's object is never deleted; after a restart
|
||||
or a hook failure the hook is retried, after 15 minutes when it failed.
|
||||
- Source issue gone: delete.
|
||||
|
||||
Deleting an issue or a company marks its restore points `delete_requested` in the same
|
||||
transaction and notifies the worker, so their objects are removed promptly. A
|
||||
`pending` restore point older than one hour was abandoned mid-capture and is deleted.
|
||||
`superseded` and `delete_requested` restore points have their stored object removed,
|
||||
then become `deleted` once the storage provider confirms the object is gone. A
|
||||
storage failure is recorded in `last_error` and retried after 15 minutes. Work is
|
||||
processed in batches of 50; a full batch continues immediately.
|
||||
|
||||
Rows have no foreign keys, so they outlive a deleted workspace, issue or company long
|
||||
enough to remove the stored object. Each run that changes anything logs its counts.
|
||||
|
||||
@@ -0,0 +1,59 @@
|
||||
CREATE TABLE "execution_workspace_restore_points" (
|
||||
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
|
||||
"company_id" uuid NOT NULL,
|
||||
"execution_workspace_id" uuid NOT NULL,
|
||||
"source_issue_id" uuid,
|
||||
"generation" integer NOT NULL,
|
||||
"object_key" text NOT NULL,
|
||||
"sha256" text NOT NULL,
|
||||
"byte_size" bigint NOT NULL,
|
||||
"status" text DEFAULT 'pending' NOT NULL,
|
||||
"activated_at" timestamp with time zone,
|
||||
"tree_closed_at" timestamp with time zone,
|
||||
"delete_after" timestamp with time zone,
|
||||
"review_generation" integer DEFAULT 0 NOT NULL,
|
||||
"deleted_at" timestamp with time zone,
|
||||
"last_error" text,
|
||||
"created_at" timestamp with time zone DEFAULT now() NOT NULL,
|
||||
"updated_at" timestamp with time zone DEFAULT now() NOT NULL
|
||||
);
|
||||
--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX "execution_workspace_restore_points_object_key_uniq" ON "execution_workspace_restore_points" USING btree ("object_key");--> statement-breakpoint
|
||||
CREATE UNIQUE INDEX "execution_workspace_restore_points_live_generation_uniq" ON "execution_workspace_restore_points" USING btree ("execution_workspace_id","generation") WHERE "execution_workspace_restore_points"."status" <> 'deleted';--> statement-breakpoint
|
||||
CREATE INDEX "execution_workspace_restore_points_status_delete_after_idx" ON "execution_workspace_restore_points" USING btree ("status","delete_after");--> statement-breakpoint
|
||||
CREATE INDEX "execution_workspace_restore_points_company_idx" ON "execution_workspace_restore_points" USING btree ("company_id");--> statement-breakpoint
|
||||
CREATE INDEX "execution_workspace_restore_points_source_issue_idx" ON "execution_workspace_restore_points" USING btree ("company_id","source_issue_id");--> statement-breakpoint
|
||||
CREATE INDEX "execution_workspace_restore_points_active_source_idx" ON "execution_workspace_restore_points" USING btree ("source_issue_id") WHERE "execution_workspace_restore_points"."status" = 'active';--> statement-breakpoint
|
||||
CREATE OR REPLACE FUNCTION paperclip_restore_point_tree_changed() RETURNS trigger
|
||||
LANGUAGE plpgsql SET search_path = public, pg_temp AS $$
|
||||
DECLARE starts uuid[];
|
||||
BEGIN
|
||||
IF TG_OP = 'DELETE' THEN
|
||||
starts := ARRAY[OLD.parent_id];
|
||||
ELSE
|
||||
starts := ARRAY[NEW.id, OLD.parent_id, NEW.parent_id];
|
||||
END IF;
|
||||
UPDATE execution_workspace_restore_points
|
||||
SET delete_after = CASE WHEN delete_after IS NULL OR delete_after > now() THEN now() ELSE delete_after END,
|
||||
review_generation = review_generation + 1,
|
||||
updated_at = now()
|
||||
WHERE status = 'active'
|
||||
AND source_issue_id IN (
|
||||
WITH RECURSIVE ancestors(id, parent_id, depth) AS (
|
||||
SELECT i.id, i.parent_id, 0 FROM issues i WHERE i.id = ANY(starts)
|
||||
UNION
|
||||
SELECT parent.id, parent.parent_id, ancestors.depth + 1 FROM issues parent
|
||||
JOIN ancestors ON parent.id = ancestors.parent_id WHERE ancestors.depth < 64
|
||||
)
|
||||
SELECT id FROM ancestors
|
||||
);
|
||||
RETURN NULL;
|
||||
END;
|
||||
$$;
|
||||
--> statement-breakpoint
|
||||
CREATE TRIGGER paperclip_restore_point_tree_update AFTER UPDATE OF status, parent_id ON issues
|
||||
FOR EACH ROW WHEN (OLD.status IS DISTINCT FROM NEW.status OR OLD.parent_id IS DISTINCT FROM NEW.parent_id)
|
||||
EXECUTE FUNCTION paperclip_restore_point_tree_changed();
|
||||
--> statement-breakpoint
|
||||
CREATE TRIGGER paperclip_restore_point_tree_delete AFTER DELETE ON issues
|
||||
FOR EACH ROW EXECUTE FUNCTION paperclip_restore_point_tree_changed();
|
||||
+5271
-4701
File diff suppressed because it is too large.
Load diff
+7
@@ -2591,6 +2591,13 @@
|
||||
"when": 1792597792969,
|
||||
"tag": "0371_furry_red_shift",
|
||||
"breakpoints": true
|
||||
},
|
||||
{
|
||||
"idx": 372,
|
||||
"version": "7",
|
||||
"when": 1792707356696,
|
||||
"tag": "0372_mature_the_initiative",
|
||||
"breakpoints": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
import { sql } from "drizzle-orm";
|
||||
import { bigint, index, integer, pgTable, text, timestamp, uniqueIndex, uuid } from "drizzle-orm/pg-core";
|
||||
|
||||
export const EXECUTION_WORKSPACE_RESTORE_POINT_STATUSES = [
|
||||
"pending",
|
||||
"active",
|
||||
"expiring",
|
||||
"superseded",
|
||||
"delete_requested",
|
||||
"deleted",
|
||||
] as const;
|
||||
export type ExecutionWorkspaceRestorePointStatus = (typeof EXECUTION_WORKSPACE_RESTORE_POINT_STATUSES)[number];
|
||||
|
||||
export const executionWorkspaceRestorePoints = pgTable(
|
||||
"execution_workspace_restore_points",
|
||||
{
|
||||
id: uuid("id").primaryKey().defaultRandom(),
|
||||
companyId: uuid("company_id").notNull(),
|
||||
executionWorkspaceId: uuid("execution_workspace_id").notNull(),
|
||||
sourceIssueId: uuid("source_issue_id"),
|
||||
generation: integer("generation").notNull(),
|
||||
objectKey: text("object_key").notNull(),
|
||||
sha256: text("sha256").notNull(),
|
||||
byteSize: bigint("byte_size", { mode: "number" }).notNull(),
|
||||
status: text("status").$type<ExecutionWorkspaceRestorePointStatus>().notNull().default("pending"),
|
||||
activatedAt: timestamp("activated_at", { withTimezone: true }),
|
||||
treeClosedAt: timestamp("tree_closed_at", { withTimezone: true }),
|
||||
deleteAfter: timestamp("delete_after", { withTimezone: true }),
|
||||
reviewGeneration: integer("review_generation").notNull().default(0),
|
||||
deletedAt: timestamp("deleted_at", { withTimezone: true }),
|
||||
lastError: text("last_error"),
|
||||
createdAt: timestamp("created_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
updatedAt: timestamp("updated_at", { withTimezone: true }).notNull().defaultNow(),
|
||||
},
|
||||
(table) => ({
|
||||
objectKeyUniqueIdx: uniqueIndex("execution_workspace_restore_points_object_key_uniq").on(table.objectKey),
|
||||
liveGenerationUniqueIdx: uniqueIndex("execution_workspace_restore_points_live_generation_uniq")
|
||||
.on(table.executionWorkspaceId, table.generation)
|
||||
.where(sql`${table.status} <> 'deleted'`),
|
||||
statusDeleteAfterIdx: index("execution_workspace_restore_points_status_delete_after_idx").on(table.status, table.deleteAfter),
|
||||
companyIdx: index("execution_workspace_restore_points_company_idx").on(table.companyId),
|
||||
sourceIssueIdx: index("execution_workspace_restore_points_source_issue_idx").on(table.companyId, table.sourceIssueId),
|
||||
activeSourceIdx: index("execution_workspace_restore_points_active_source_idx").on(table.sourceIssueId)
|
||||
.where(sql`${table.status} = 'active'`),
|
||||
}),
|
||||
);
|
||||
@@ -32,6 +32,11 @@ export { projectAccessMembers } from "./project_access_members.js";
|
||||
export { documentMemberships } from "./document_memberships.js";
|
||||
export { projectWorkspaces } from "./project_workspaces.js";
|
||||
export { executionWorkspaces } from "./execution_workspaces.js";
|
||||
export {
|
||||
executionWorkspaceRestorePoints,
|
||||
EXECUTION_WORKSPACE_RESTORE_POINT_STATUSES,
|
||||
type ExecutionWorkspaceRestorePointStatus,
|
||||
} from "./execution_workspace_restore_points.js";
|
||||
export { executionWorkspaceRuntimeLeases } from "./execution_workspace_runtime_leases.js";
|
||||
export { environments } from "./environments.js";
|
||||
export { environmentLeases } from "./environment_leases.js";
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
import { afterEach, describe, expect, it, vi } from "vitest";
|
||||
import { loadConfig } from "../config.ts";
|
||||
|
||||
describe("git restore point retention config parsing", () => {
|
||||
afterEach(() => {
|
||||
vi.unstubAllEnvs();
|
||||
});
|
||||
|
||||
it("keeps restore points 90 days after their issue tree closes by default", () => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", undefined);
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointRetentionDays: 90 });
|
||||
});
|
||||
|
||||
it("accepts an explicit retention and never goes below one day", () => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", "30");
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointRetentionDays: 30 });
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", "0");
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointRetentionDays: 1 });
|
||||
});
|
||||
|
||||
it.each(["", "-1", "many"])("falls back to 90 days for %j", (value) => {
|
||||
vi.stubEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", value);
|
||||
expect(loadConfig()).toMatchObject({ gitRestorePointRetentionDays: 90 });
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,381 @@
|
||||
import { randomUUID } from "node:crypto";
|
||||
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
|
||||
import { eq } from "drizzle-orm";
|
||||
import { agents, companies, createDb, executionWorkspaceRestorePoints, issues } from "@paperclipai/db";
|
||||
import {
|
||||
getEmbeddedPostgresTestSupport,
|
||||
startEmbeddedPostgresTestDatabase,
|
||||
} from "./helpers/embedded-postgres.js";
|
||||
import {
|
||||
GIT_RESTORE_POINT_RETRY_MS,
|
||||
gitRestorePointRetentionService,
|
||||
registerGitRestorePointWork,
|
||||
requestRestorePointDeletion,
|
||||
} from "../services/git-restore-point-ledger.ts";
|
||||
import { issueService } from "../services/issues.ts";
|
||||
import { deleteCompany } from "../services/company-deletion.ts";
|
||||
import type { StorageService } from "../storage/types.ts";
|
||||
|
||||
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
|
||||
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
|
||||
|
||||
if (!embeddedPostgresSupport.supported) {
|
||||
console.warn(
|
||||
`Skipping embedded Postgres git restore point ledger tests on this host: ${embeddedPostgresSupport.reason ?? "unsupported environment"}`,
|
||||
);
|
||||
}
|
||||
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
|
||||
describeEmbeddedPostgres("git restore point retention", () => {
|
||||
let db!: ReturnType<typeof createDb>;
|
||||
let tempDb: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | null = null;
|
||||
const now = new Date();
|
||||
let deleted: string[] = [];
|
||||
let failDeletes = false;
|
||||
let silentlyKeep = false;
|
||||
const storage = {
|
||||
deleteObject: async (_companyId: string, objectKey: string) => {
|
||||
if (failDeletes) throw new Error("storage unavailable");
|
||||
if (!silentlyKeep) deleted.push(objectKey);
|
||||
},
|
||||
headObject: async (_companyId: string, objectKey: string) => ({ exists: silentlyKeep && !deleted.includes(objectKey) }),
|
||||
} as unknown as StorageService;
|
||||
|
||||
beforeAll(async () => {
|
||||
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-git-restore-point-ledger-");
|
||||
db = createDb(tempDb.connectionString);
|
||||
}, 20_000);
|
||||
|
||||
afterEach(async () => {
|
||||
deleted = [];
|
||||
failDeletes = false;
|
||||
silentlyKeep = false;
|
||||
await db.delete(executionWorkspaceRestorePoints);
|
||||
await db.delete(issues);
|
||||
await db.delete(agents);
|
||||
await db.delete(companies);
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await tempDb?.cleanup();
|
||||
});
|
||||
|
||||
async function seed(input: {
|
||||
status?: "pending" | "active" | "superseded" | "delete_requested";
|
||||
issueStatus?: string;
|
||||
closedDaysAgo?: number;
|
||||
openChild?: boolean;
|
||||
createdHoursAgo?: number;
|
||||
withoutIssue?: boolean;
|
||||
deleteAfter?: Date | null;
|
||||
companyId?: string;
|
||||
} = {}) {
|
||||
const companyId = input.companyId ?? randomUUID();
|
||||
if (!input.companyId) {
|
||||
await db.insert(companies).values({ id: companyId, name: "Restore", issuePrefix: `R${companyId.slice(0, 6)}` });
|
||||
}
|
||||
const issueStatus = input.issueStatus ?? "done";
|
||||
const closedAt = new Date(now.getTime() - (input.closedDaysAgo ?? 10) * DAY_MS);
|
||||
const [issue] = await db.insert(issues).values({
|
||||
companyId,
|
||||
title: "Work",
|
||||
status: issueStatus,
|
||||
completedAt: issueStatus === "done" ? closedAt : null,
|
||||
cancelledAt: issueStatus === "cancelled" ? closedAt : null,
|
||||
}).returning();
|
||||
if (input.openChild) {
|
||||
await db.insert(issues).values({ companyId, title: "Child", status: "blocked", parentId: issue!.id });
|
||||
}
|
||||
const workspaceId = randomUUID();
|
||||
const [row] = await db.insert(executionWorkspaceRestorePoints).values({
|
||||
companyId,
|
||||
executionWorkspaceId: workspaceId,
|
||||
sourceIssueId: input.withoutIssue ? randomUUID() : issue!.id,
|
||||
generation: 1,
|
||||
objectKey: `${companyId}/git-restore-points/${workspaceId}/1/${randomUUID()}.pcrp`,
|
||||
sha256: "a".repeat(64),
|
||||
byteSize: 1024,
|
||||
status: input.status ?? "active",
|
||||
deleteAfter: input.deleteAfter ?? null,
|
||||
createdAt: new Date(now.getTime() - (input.createdHoursAgo ?? 0) * 60 * 60 * 1000),
|
||||
}).returning();
|
||||
return row!;
|
||||
}
|
||||
|
||||
function service(options: { at?: Date; beforeExpire?: (row: typeof executionWorkspaceRestorePoints.$inferSelect) => Promise<void> } = {}) {
|
||||
return gitRestorePointRetentionService(db, { storage, now: () => options.at ?? now, beforeExpire: options.beforeExpire });
|
||||
}
|
||||
|
||||
async function reload(id: string) {
|
||||
const [row] = await db.select().from(executionWorkspaceRestorePoints).where(eq(executionWorkspaceRestorePoints.id, id));
|
||||
return row!;
|
||||
}
|
||||
|
||||
const due = () => new Date(now.getTime() - 1000);
|
||||
|
||||
it("waits without a deadline while its tree is open", async () => {
|
||||
const row = await seed({ openChild: true, deleteAfter: due() });
|
||||
expect(await service().tick()).toMatchObject({ scheduled: 1, deleted: 0 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "active", treeClosedAt: null, deleteAfter: null });
|
||||
expect(await service().nextDeadlineAt()).toBeNull();
|
||||
});
|
||||
|
||||
it("schedules deletion 90 days after the tree closes", async () => {
|
||||
const row = await seed({ closedDaysAgo: 10, deleteAfter: due() });
|
||||
await service().tick();
|
||||
const reloaded = await reload(row.id);
|
||||
expect(reloaded.status).toBe("active");
|
||||
expect(reloaded.deleteAfter!.getTime() - reloaded.treeClosedAt!.getTime()).toBe(90 * DAY_MS);
|
||||
expect(deleted).toEqual([]);
|
||||
});
|
||||
|
||||
it("expires and deletes a restore point at its deadline once the tree has been closed long enough", async () => {
|
||||
const row = await seed({ closedDaysAgo: 91, deleteAfter: new Date(now.getTime() - 1000) });
|
||||
const beforeExpire = vi.fn(async () => {});
|
||||
expect(await service({ beforeExpire }).tick()).toMatchObject({ expired: 1, deleted: 1 });
|
||||
expect(beforeExpire).toHaveBeenCalledOnce();
|
||||
expect(deleted).toEqual([row.objectKey]);
|
||||
expect(await reload(row.id)).toMatchObject({ status: "deleted" });
|
||||
});
|
||||
|
||||
it("keeps a restore point expiring, and its object, until the expiry hook succeeds", async () => {
|
||||
const row = await seed({ closedDaysAgo: 91, deleteAfter: due() });
|
||||
expect(await service({ beforeExpire: async () => { throw new Error("recipe busy"); } }).tick()).toMatchObject({ failed: 1, expired: 0, deleted: 0 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "expiring", lastError: "recipe busy" });
|
||||
expect(await service().tick()).toMatchObject({ expired: 0, deleted: 0 });
|
||||
expect(await service().nextDeadlineAt()).toBe((await reload(row.id)).updatedAt.getTime() + GIT_RESTORE_POINT_RETRY_MS);
|
||||
const beforeExpire = vi.fn(async () => {});
|
||||
const later = service({ at: new Date(now.getTime() + GIT_RESTORE_POINT_RETRY_MS + 1000), beforeExpire });
|
||||
expect(await later.tick()).toMatchObject({ expired: 1, deleted: 1 });
|
||||
expect(beforeExpire).toHaveBeenCalledOnce();
|
||||
expect(deleted).toEqual([row.objectKey]);
|
||||
});
|
||||
|
||||
it("resumes the expiry hook after a restart instead of deleting the object directly", async () => {
|
||||
const row = await seed({ closedDaysAgo: 91, status: "active" });
|
||||
await db.update(executionWorkspaceRestorePoints).set({ status: "expiring" }).where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
const beforeExpire = vi.fn(async () => {});
|
||||
expect(await service().nextDeadlineAt()).toBe(now.getTime());
|
||||
expect(await service({ beforeExpire }).tick()).toMatchObject({ expired: 1, deleted: 1 });
|
||||
expect(beforeExpire).toHaveBeenCalledOnce();
|
||||
expect(await reload(row.id)).toMatchObject({ status: "deleted" });
|
||||
});
|
||||
|
||||
it("never deletes the object of an expiring restore point whose hook has not run", async () => {
|
||||
const row = await seed({ closedDaysAgo: 91, status: "active" });
|
||||
await db.update(executionWorkspaceRestorePoints).set({ status: "expiring" }).where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
await service({ beforeExpire: async () => { throw new Error("recipe busy"); } }).tick();
|
||||
expect(deleted).toEqual([]);
|
||||
});
|
||||
|
||||
it("deletes restore points whose source issue no longer exists at review", async () => {
|
||||
const row = await seed({ withoutIssue: true, deleteAfter: due() });
|
||||
expect(await service().tick()).toMatchObject({ orphaned: 1, deleted: 1 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "deleted" });
|
||||
});
|
||||
|
||||
it("evaluates only restore points that are due, so retained ones cause no rescans", async () => {
|
||||
for (let index = 0; index < 60; index += 1) await seed({ closedDaysAgo: 10, deleteAfter: new Date(now.getTime() + DAY_MS) });
|
||||
expect(await service().tick()).toMatchObject({ scheduled: 0, expired: 0, truncated: false });
|
||||
expect(await service().nextDeadlineAt()).toBe(now.getTime() + DAY_MS);
|
||||
});
|
||||
|
||||
it("works through a backlog of due restore points in bounded batches", async () => {
|
||||
for (let index = 0; index < 60; index += 1) await seed({ closedDaysAgo: 10, deleteAfter: due() });
|
||||
expect(await service().tick()).toMatchObject({ scheduled: 50, truncated: true });
|
||||
expect(await service().tick()).toMatchObject({ scheduled: 10, truncated: false });
|
||||
expect(await service().tick()).toMatchObject({ scheduled: 0, truncated: false });
|
||||
});
|
||||
|
||||
it("cleans up abandoned pending restore points but leaves recent ones", async () => {
|
||||
const stale = await seed({ status: "pending", createdHoursAgo: 2 });
|
||||
const recent = await seed({ status: "pending", createdHoursAgo: 0 });
|
||||
await service().tick();
|
||||
expect(await reload(stale.id)).toMatchObject({ status: "deleted" });
|
||||
expect(await reload(recent.id)).toMatchObject({ status: "pending" });
|
||||
});
|
||||
|
||||
it("deletes superseded restore points", async () => {
|
||||
const row = await seed({ status: "superseded" });
|
||||
await service().tick();
|
||||
expect(await reload(row.id)).toMatchObject({ status: "deleted" });
|
||||
});
|
||||
|
||||
it("does not mark an object deleted when it is still present after deletion", async () => {
|
||||
const row = await seed({ status: "delete_requested" });
|
||||
silentlyKeep = true;
|
||||
expect(await service().tick()).toMatchObject({ failed: 1, deleted: 0 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "delete_requested", lastError: "restore_point_object_still_present" });
|
||||
});
|
||||
|
||||
it("retries a failed deletion only after its retry deadline", async () => {
|
||||
const row = await seed({ status: "delete_requested" });
|
||||
failDeletes = true;
|
||||
expect(await service().tick()).toMatchObject({ failed: 1 });
|
||||
failDeletes = false;
|
||||
expect(await service().tick()).toMatchObject({ failed: 0, deleted: 0 });
|
||||
expect(await service().nextDeadlineAt()).toBe((await reload(row.id)).updatedAt.getTime() + GIT_RESTORE_POINT_RETRY_MS);
|
||||
expect(await service({ at: new Date(now.getTime() + GIT_RESTORE_POINT_RETRY_MS + 1000) }).tick()).toMatchObject({ deleted: 1 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "deleted", lastError: null });
|
||||
});
|
||||
|
||||
it("reports the real next deadline, and none when idle", async () => {
|
||||
expect(await service().nextDeadlineAt()).toBeNull();
|
||||
await seed({ openChild: true });
|
||||
expect(await service().nextDeadlineAt()).toBeNull();
|
||||
const reviewed = await seed({ closedDaysAgo: 10, deleteAfter: due() });
|
||||
expect(await service().nextDeadlineAt()).toBe(due().getTime());
|
||||
await service().tick();
|
||||
expect(await service().nextDeadlineAt()).toBe((await reload(reviewed.id)).deleteAfter!.getTime());
|
||||
const pending = await seed({ status: "pending", createdHoursAgo: 0 });
|
||||
expect(await service().nextDeadlineAt()).toBe((await reload(pending.id)).createdAt.getTime() + 60 * 60 * 1000);
|
||||
await seed({ status: "delete_requested" });
|
||||
expect(await service().nextDeadlineAt()).toBe(now.getTime());
|
||||
});
|
||||
|
||||
it("marks only the deleted issue's restore points when an issue is removed", async () => {
|
||||
const removed = await seed({ deleteAfter: new Date(now.getTime() + 80 * DAY_MS) });
|
||||
const kept = await seed({ deleteAfter: new Date(now.getTime() + 80 * DAY_MS) });
|
||||
await issueService(db).remove(removed.sourceIssueId!);
|
||||
expect(await reload(removed.id)).toMatchObject({ status: "delete_requested" });
|
||||
expect(await reload(kept.id)).toMatchObject({ status: "active" });
|
||||
await service().tick();
|
||||
expect(deleted).toEqual([removed.objectKey]);
|
||||
});
|
||||
|
||||
it("marks every restore point of a deleted company", async () => {
|
||||
const first = await seed({ deleteAfter: new Date(now.getTime() + 80 * DAY_MS) });
|
||||
const second = await seed({ companyId: first.companyId, status: "pending" });
|
||||
const other = await seed({ deleteAfter: new Date(now.getTime() + 80 * DAY_MS) });
|
||||
await deleteCompany(db, first.companyId);
|
||||
expect(await reload(first.id)).toMatchObject({ status: "delete_requested" });
|
||||
expect(await reload(second.id)).toMatchObject({ status: "delete_requested" });
|
||||
expect(await reload(other.id)).toMatchObject({ status: "active" });
|
||||
await service().tick();
|
||||
expect(deleted.sort()).toEqual([first.objectKey, second.objectKey].sort());
|
||||
});
|
||||
|
||||
it("requests deletion by issue without touching superseded rows", async () => {
|
||||
const row = await seed({ status: "superseded" });
|
||||
await db.transaction(async (tx) => requestRestorePointDeletion(tx, { sourceIssueId: row.sourceIssueId! }));
|
||||
expect(await reload(row.id)).toMatchObject({ status: "superseded" });
|
||||
});
|
||||
|
||||
it("schedules nothing when idle, and only the next deadline when restore points exist", async () => {
|
||||
let task!: { run: (signal: AbortSignal) => Promise<unknown>; nextRunAt?: () => number | null };
|
||||
const coordinator = { register: (_queue: string, registered: typeof task) => { task = registered; return { ready: Promise.resolve(), wake: vi.fn() }; } };
|
||||
registerGitRestorePointWork(coordinator as never, service(), () => {});
|
||||
await task.run(new AbortController().signal);
|
||||
expect(task.nextRunAt?.()).toBeNull();
|
||||
const row = await seed({ closedDaysAgo: 10, deleteAfter: due() });
|
||||
await task.run(new AbortController().signal);
|
||||
expect(task.nextRunAt?.()).toBe((await reload(row.id)).deleteAfter!.getTime());
|
||||
});
|
||||
|
||||
it("wakes on issue changes only while restore points exist", async () => {
|
||||
let task!: { run: (signal: AbortSignal) => Promise<unknown>; nextRunAt?: () => number | null };
|
||||
const wake = vi.fn();
|
||||
const coordinator = { register: (_queue: string, registered: typeof task) => { task = registered; return { ready: Promise.resolve(), wake }; } };
|
||||
const worker = registerGitRestorePointWork(coordinator as never, service(), () => {});
|
||||
await task.run(new AbortController().signal);
|
||||
worker.issuesChanged();
|
||||
expect(wake).not.toHaveBeenCalled();
|
||||
await seed({ openChild: true });
|
||||
await task.run(new AbortController().signal);
|
||||
worker.issuesChanged();
|
||||
expect(wake).toHaveBeenCalledOnce();
|
||||
});
|
||||
|
||||
describe("issue tree trigger", () => {
|
||||
async function child(parentId: string, companyId: string, status = "blocked") {
|
||||
const [row] = await db.insert(issues).values({ companyId, title: "Child", status, parentId }).returning();
|
||||
return row!;
|
||||
}
|
||||
|
||||
it("marks a parent's restore point due when its last open child closes", async () => {
|
||||
const row = await seed();
|
||||
const open = await child(row.sourceIssueId!, row.companyId);
|
||||
await db.update(executionWorkspaceRestorePoints).set({ deleteAfter: null }).where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, open.id));
|
||||
expect((await reload(row.id)).deleteAfter!.getTime()).toBeLessThanOrEqual(Date.now());
|
||||
});
|
||||
|
||||
it("marks the former parent when a child is deleted or moved", async () => {
|
||||
const deletedFrom = await seed();
|
||||
const movedFrom = await seed();
|
||||
const deleting = await child(deletedFrom.sourceIssueId!, deletedFrom.companyId);
|
||||
const moving = await child(movedFrom.sourceIssueId!, movedFrom.companyId);
|
||||
const [elsewhere] = await db.insert(issues).values({ companyId: movedFrom.companyId, title: "Other", status: "todo" }).returning();
|
||||
await db.update(executionWorkspaceRestorePoints).set({ deleteAfter: null });
|
||||
await db.delete(issues).where(eq(issues.id, deleting.id));
|
||||
await db.update(issues).set({ parentId: elsewhere!.id }).where(eq(issues.id, moving.id));
|
||||
expect((await reload(deletedFrom.id)).deleteAfter).not.toBeNull();
|
||||
expect((await reload(movedFrom.id)).deleteAfter).not.toBeNull();
|
||||
});
|
||||
|
||||
it("marks a restore point when its own source issue changes status", async () => {
|
||||
const row = await seed({ issueStatus: "in_progress" });
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, row.sourceIssueId!));
|
||||
expect((await reload(row.id)).deleteAfter).not.toBeNull();
|
||||
});
|
||||
|
||||
it("leaves unrelated, inactive and already due restore points alone", async () => {
|
||||
const unrelated = await seed({ issueStatus: "in_progress" });
|
||||
const superseded = await seed({ status: "superseded", issueStatus: "in_progress" });
|
||||
const alreadyDue = await seed({ issueStatus: "in_progress", deleteAfter: new Date(now.getTime() - DAY_MS) });
|
||||
const [other] = await db.insert(issues).values({ companyId: unrelated.companyId, title: "Other", status: "todo" }).returning();
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, other!.id));
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, superseded.sourceIssueId!));
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, alreadyDue.sourceIssueId!));
|
||||
expect((await reload(unrelated.id)).deleteAfter).toBeNull();
|
||||
expect((await reload(superseded.id)).deleteAfter).toBeNull();
|
||||
expect((await reload(alreadyDue.id)).deleteAfter!.getTime()).toBe(now.getTime() - DAY_MS);
|
||||
});
|
||||
|
||||
it("keeps a restore point due when its tree closes between the worker's read and write", async () => {
|
||||
const row = await seed({ deleteAfter: due() });
|
||||
const open = await child(row.sourceIssueId!, row.companyId);
|
||||
const racing = gitRestorePointRetentionService(db, {
|
||||
storage,
|
||||
afterTreeRead: async () => {
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, open.id));
|
||||
},
|
||||
});
|
||||
expect(await racing.tick()).toMatchObject({ raced: 1, scheduled: 0 });
|
||||
const stillDue = await reload(row.id);
|
||||
expect(stillDue.deleteAfter!.getTime()).toBeLessThanOrEqual(Date.now());
|
||||
await gitRestorePointRetentionService(db, { storage }).tick();
|
||||
const reviewed = await reload(row.id);
|
||||
expect(reviewed.deleteAfter!.getTime() - reviewed.treeClosedAt!.getTime()).toBe(90 * DAY_MS);
|
||||
});
|
||||
|
||||
it("does not expire a restore point whose tree reopens between the worker's read and write", async () => {
|
||||
const row = await seed({ closedDaysAgo: 120, deleteAfter: due() });
|
||||
const racing = gitRestorePointRetentionService(db, {
|
||||
storage,
|
||||
afterTreeRead: async () => {
|
||||
await db.update(issues).set({ status: "todo", completedAt: null }).where(eq(issues.id, row.sourceIssueId!));
|
||||
},
|
||||
});
|
||||
expect(await racing.tick()).toMatchObject({ raced: 1, expired: 0, deleted: 0 });
|
||||
expect(await reload(row.id)).toMatchObject({ status: "active" });
|
||||
await gitRestorePointRetentionService(db, { storage }).tick();
|
||||
expect(await reload(row.id)).toMatchObject({ status: "active", deleteAfter: null });
|
||||
expect(deleted).toEqual([]);
|
||||
});
|
||||
|
||||
it("lets the worker schedule expiry once an open tree closes", async () => {
|
||||
const row = await seed();
|
||||
const open = await child(row.sourceIssueId!, row.companyId);
|
||||
await db.update(executionWorkspaceRestorePoints).set({ deleteAfter: due() }).where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
const live = gitRestorePointRetentionService(db, { storage });
|
||||
await live.tick();
|
||||
expect((await reload(row.id)).deleteAfter).toBeNull();
|
||||
await db.update(issues).set({ status: "done", completedAt: new Date() }).where(eq(issues.id, open.id));
|
||||
await live.tick();
|
||||
const reviewed = await reload(row.id);
|
||||
expect(reviewed.deleteAfter!.getTime() - reviewed.treeClosedAt!.getTime()).toBe(90 * DAY_MS);
|
||||
});
|
||||
});
|
||||
});
|
||||
@@ -91,6 +91,7 @@ export interface Config {
|
||||
nativeHarnessBackupExpirationEnabled: boolean;
|
||||
nativeHarnessBackupExpirationDryRun: boolean;
|
||||
nativeHarnessBackupExpirationDays: number;
|
||||
gitRestorePointRetentionDays: number;
|
||||
serveUi: boolean;
|
||||
uiDevMiddleware: boolean;
|
||||
secretsProvider: SecretProvider;
|
||||
@@ -337,6 +338,7 @@ export function loadConfig(): Config {
|
||||
const skillVersionRetentionKeepNewest = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_KEEP", 0);
|
||||
const skillVersionRetentionGraceMinutes = nonNegativeIntegerEnv("PAPERCLIP_SKILL_VERSION_RETENTION_GRACE_MINUTES", 60);
|
||||
const nativeHarnessBackupExpirationDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DAYS", 7));
|
||||
const gitRestorePointRetentionDays = Math.max(1, nonNegativeIntegerEnv("PAPERCLIP_GIT_RESTORE_POINT_RETENTION_DAYS", 90));
|
||||
const bindValidationErrors = validateConfiguredBindMode({
|
||||
deploymentMode,
|
||||
deploymentExposure,
|
||||
@@ -394,6 +396,7 @@ export function loadConfig(): Config {
|
||||
nativeHarnessBackupExpirationEnabled: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_ENABLED !== "false",
|
||||
nativeHarnessBackupExpirationDryRun: process.env.PAPERCLIP_NATIVE_HARNESS_BACKUP_EXPIRATION_DRY_RUN === "true",
|
||||
nativeHarnessBackupExpirationDays,
|
||||
gitRestorePointRetentionDays,
|
||||
serveUi:
|
||||
process.env.SERVE_UI !== undefined
|
||||
? process.env.SERVE_UI === "true"
|
||||
|
||||
@@ -12,6 +12,7 @@ import {
|
||||
NATIVE_HARNESS_BACKUP_EXPIRATION_STARTUP_DELAY_MS,
|
||||
nativeHarnessBackupExpirationService,
|
||||
} from "./services/native-runtime/native-harness-backup-expiration.js";
|
||||
import { gitRestorePointRetentionService, registerGitRestorePointWork } from "./services/git-restore-point-ledger.js";
|
||||
import { subscribeAllCompanyLiveEvents } from "./services/live-events.js";
|
||||
import { DELIVERY_QUEUES } from "./services/delivery-work-notifications.js";
|
||||
import { registerBudgetEnforcementWork, registerCostAccountingWork } from "./services/budget-enforcement-worker.js";
|
||||
@@ -1497,9 +1498,21 @@ async function startServerWithDatabaseTeardown(
|
||||
logger.error({ err }, "merged pull-request confirmation sweep failed");
|
||||
}));
|
||||
};
|
||||
const gitRestorePointWorker = registerGitRestorePointWork(deliveryWork, gitRestorePointRetentionService(db as any, {
|
||||
storage: storageService,
|
||||
retentionDays: config.gitRestorePointRetentionDays,
|
||||
}), (result) => {
|
||||
if (result.abandoned > 0 || result.expired > 0 || result.orphaned > 0 || result.deleted > 0 || result.failed > 0) {
|
||||
logger.info(result, "git restore point retention");
|
||||
}
|
||||
});
|
||||
let lastTerminalWorkspaceSkipLogAt = 0;
|
||||
const terminalWorkspaceWorker = registerTerminalWorkspaceWork(deliveryWork, {
|
||||
...terminalWorkspaces,
|
||||
terminalWorkspaceWorkSettled: (hints) => {
|
||||
terminalWorkspaces.terminalWorkspaceWorkSettled?.(hints);
|
||||
gitRestorePointWorker.issuesChanged();
|
||||
},
|
||||
sweepTerminalWorkspaces: async (limit, signal, options) => {
|
||||
const result = await terminalWorkspaces.sweepTerminalWorkspaces(limit, signal, options);
|
||||
if (result.archived > 0 || result.cleanupFailed > 0 || result.compacted > 0 || result.compactionFailed > 0) {
|
||||
|
||||
@@ -4,6 +4,7 @@ import { withAccountHomeSecretMutationLock } from "@paperclipai/adapter-codex-lo
|
||||
import type { Db } from "@paperclipai/db";
|
||||
import { eq, inArray } from "drizzle-orm";
|
||||
import { createAgentLifecycle, processAgentLifecycle } from "./agent-lifecycle.js";
|
||||
import { requestRestorePointDeletion } from "./git-restore-point-ledger.js";
|
||||
import { agentLifecycleCompanyDeletion } from "../modules/agent-lifecycle/company-deletion.js";
|
||||
import {
|
||||
companies,
|
||||
@@ -109,6 +110,7 @@ export async function deleteCompany(db: Db, id: string) {
|
||||
await tx.delete(companyLogos).where(eq(companyLogos.companyId, id));
|
||||
await tx.delete(assets).where(eq(assets.companyId, id));
|
||||
await tx.delete(goals).where(eq(goals.companyId, id));
|
||||
await requestRestorePointDeletion(tx, { companyId: id });
|
||||
await tx.delete(executionWorkspaces).where(eq(executionWorkspaces.companyId, id));
|
||||
await tx.delete(projects).where(eq(projects.companyId, id));
|
||||
await agentLifecycleCompanyDeletion.deleteCompanyData(tx, id);
|
||||
|
||||
@@ -38,6 +38,7 @@ export const DELIVERY_QUEUES = {
|
||||
feedback: "feedback-exports",
|
||||
chatCompletion: "chat-completions",
|
||||
connection: "connection-continuations",
|
||||
gitRestorePoints: "git-restore-points",
|
||||
question: "question-responses",
|
||||
pluginLifecycle: "plugin-resource-lifecycle",
|
||||
toolAction: "tool-action-receipts",
|
||||
|
||||
@@ -0,0 +1,238 @@
|
||||
import { and, asc, eq, inArray, isNull, lt, lte, or, sql } from "drizzle-orm";
|
||||
import { executionWorkspaceRestorePoints, type Db } from "@paperclipai/db";
|
||||
import type { StorageService } from "../storage/types.js";
|
||||
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
|
||||
export const GIT_RESTORE_POINT_RETRY_MS = 15 * 60 * 1000;
|
||||
export const GIT_RESTORE_POINT_DEFAULT_RETENTION_DAYS = 90;
|
||||
const DAY_MS = 24 * 60 * 60 * 1000;
|
||||
const DEFAULT_PENDING_GRACE_MS = 60 * 60 * 1000;
|
||||
const DEFAULT_BATCH = 50;
|
||||
|
||||
type RestorePoint = typeof executionWorkspaceRestorePoints.$inferSelect;
|
||||
|
||||
export type GitRestorePointRetentionResult = {
|
||||
abandoned: number;
|
||||
scheduled: number;
|
||||
expired: number;
|
||||
orphaned: number;
|
||||
deleted: number;
|
||||
failed: number;
|
||||
raced: number;
|
||||
truncated: boolean;
|
||||
};
|
||||
|
||||
export type GitRestorePointRetentionOptions = {
|
||||
storage: StorageService;
|
||||
retentionDays?: number;
|
||||
pendingGraceMs?: number;
|
||||
batchSize?: number;
|
||||
now?: () => Date;
|
||||
beforeExpire?: (restorePoint: RestorePoint) => Promise<void>;
|
||||
afterTreeRead?: (restorePoint: RestorePoint) => Promise<void>;
|
||||
};
|
||||
|
||||
type TreeState = { exists: boolean; open: boolean; closedAt: Date | null };
|
||||
|
||||
export function gitRestorePointRetentionService(db: Db, options: GitRestorePointRetentionOptions) {
|
||||
const retentionMs = Math.max(1, options.retentionDays ?? GIT_RESTORE_POINT_DEFAULT_RETENTION_DAYS) * DAY_MS;
|
||||
const pendingGraceMs = options.pendingGraceMs ?? DEFAULT_PENDING_GRACE_MS;
|
||||
const batchSize = options.batchSize ?? DEFAULT_BATCH;
|
||||
const now = options.now ?? (() => new Date());
|
||||
|
||||
async function issueTreeState(row: RestorePoint): Promise<TreeState> {
|
||||
if (!row.sourceIssueId) return { exists: false, open: false, closedAt: null };
|
||||
const result = await db.execute<{ total: number; open: number; closed_at: string | null }>(sql`
|
||||
WITH RECURSIVE tree AS (
|
||||
SELECT id, status, completed_at, cancelled_at, updated_at FROM issues
|
||||
WHERE company_id = ${row.companyId} AND id = ${row.sourceIssueId}
|
||||
UNION
|
||||
SELECT c.id, c.status, c.completed_at, c.cancelled_at, c.updated_at FROM issues c
|
||||
JOIN tree p ON c.parent_id = p.id WHERE c.company_id = ${row.companyId}
|
||||
)
|
||||
SELECT count(*)::int AS total,
|
||||
count(*) FILTER (WHERE status NOT IN ('done', 'cancelled'))::int AS open,
|
||||
max(CASE WHEN status = 'done' THEN coalesce(completed_at, updated_at)
|
||||
WHEN status = 'cancelled' THEN coalesce(cancelled_at, updated_at) END) AS closed_at
|
||||
FROM tree`);
|
||||
const state = result[0];
|
||||
if (!state || state.total === 0) return { exists: false, open: false, closedAt: null };
|
||||
return { exists: true, open: state.open > 0, closedAt: state.closed_at ? new Date(state.closed_at) : null };
|
||||
}
|
||||
|
||||
async function requestDeletion(ids: string[], at: Date) {
|
||||
if (ids.length === 0) return;
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "delete_requested", updatedAt: at })
|
||||
.where(and(inArray(executionWorkspaceRestorePoints.id, ids),
|
||||
inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
|
||||
}
|
||||
|
||||
async function evaluateActive(row: RestorePoint, at: Date, result: GitRestorePointRetentionResult) {
|
||||
const tree = await issueTreeState(row);
|
||||
await options.afterTreeRead?.(row);
|
||||
const unchanged = and(
|
||||
eq(executionWorkspaceRestorePoints.id, row.id),
|
||||
eq(executionWorkspaceRestorePoints.status, "active"),
|
||||
eq(executionWorkspaceRestorePoints.reviewGeneration, row.reviewGeneration),
|
||||
);
|
||||
const deleteAfter = !tree.exists || tree.open || !tree.closedAt ? null : new Date(tree.closedAt.getTime() + retentionMs);
|
||||
if (!tree.exists || (deleteAfter && deleteAfter.getTime() <= at.getTime())) {
|
||||
const claimed = await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: tree.exists ? "expiring" : "delete_requested", lastError: null, updatedAt: at })
|
||||
.where(unchanged)
|
||||
.returning();
|
||||
if (claimed.length === 0) {
|
||||
result.raced += 1;
|
||||
return;
|
||||
}
|
||||
if (!tree.exists) {
|
||||
result.orphaned += 1;
|
||||
return;
|
||||
}
|
||||
await finishExpiry(claimed[0]!, at, result);
|
||||
return;
|
||||
}
|
||||
const updated = await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ treeClosedAt: tree.open ? null : tree.closedAt, deleteAfter, updatedAt: at })
|
||||
.where(unchanged)
|
||||
.returning({ id: executionWorkspaceRestorePoints.id });
|
||||
if (updated.length === 0) result.raced += 1;
|
||||
else result.scheduled += 1;
|
||||
}
|
||||
|
||||
async function finishExpiry(row: RestorePoint, at: Date, result: GitRestorePointRetentionResult) {
|
||||
try {
|
||||
await options.beforeExpire?.(row);
|
||||
} catch (error) {
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ lastError: error instanceof Error ? error.message.slice(0, 500) : "expire_failed", updatedAt: at })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.id, row.id), eq(executionWorkspaceRestorePoints.status, "expiring")));
|
||||
result.failed += 1;
|
||||
return;
|
||||
}
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "delete_requested", lastError: null, updatedAt: at })
|
||||
.where(and(eq(executionWorkspaceRestorePoints.id, row.id), eq(executionWorkspaceRestorePoints.status, "expiring")));
|
||||
result.expired += 1;
|
||||
}
|
||||
|
||||
function retryable(status: "expiring" | "superseded" | "delete_requested" | Array<"superseded" | "delete_requested">, at: Date) {
|
||||
return and(
|
||||
Array.isArray(status) ? inArray(executionWorkspaceRestorePoints.status, status) : eq(executionWorkspaceRestorePoints.status, status),
|
||||
or(isNull(executionWorkspaceRestorePoints.lastError),
|
||||
lte(executionWorkspaceRestorePoints.updatedAt, new Date(at.getTime() - GIT_RESTORE_POINT_RETRY_MS))),
|
||||
);
|
||||
}
|
||||
|
||||
async function tick() {
|
||||
const at = now();
|
||||
const result: GitRestorePointRetentionResult = { abandoned: 0, scheduled: 0, expired: 0, orphaned: 0, deleted: 0, failed: 0, raced: 0, truncated: false };
|
||||
|
||||
const abandoned = await db.select({ id: executionWorkspaceRestorePoints.id }).from(executionWorkspaceRestorePoints)
|
||||
.where(and(eq(executionWorkspaceRestorePoints.status, "pending"),
|
||||
lt(executionWorkspaceRestorePoints.createdAt, new Date(at.getTime() - pendingGraceMs))))
|
||||
.orderBy(asc(executionWorkspaceRestorePoints.createdAt)).limit(batchSize);
|
||||
await requestDeletion(abandoned.map((row) => row.id), at);
|
||||
result.abandoned = abandoned.length;
|
||||
|
||||
const due = await db.select().from(executionWorkspaceRestorePoints)
|
||||
.where(and(eq(executionWorkspaceRestorePoints.status, "active"),
|
||||
sql`${executionWorkspaceRestorePoints.deleteAfter} <= greatest(${at.toISOString()}::timestamptz, clock_timestamp())`))
|
||||
.orderBy(asc(executionWorkspaceRestorePoints.deleteAfter), asc(executionWorkspaceRestorePoints.id)).limit(batchSize);
|
||||
for (const row of due) await evaluateActive(row, at, result);
|
||||
|
||||
const expiring = await db.select().from(executionWorkspaceRestorePoints)
|
||||
.where(retryable("expiring", at))
|
||||
.orderBy(asc(executionWorkspaceRestorePoints.updatedAt), asc(executionWorkspaceRestorePoints.id)).limit(batchSize);
|
||||
for (const row of expiring) await finishExpiry(row, at, result);
|
||||
|
||||
const doomed = await db.select().from(executionWorkspaceRestorePoints)
|
||||
.where(retryable(["superseded", "delete_requested"], at))
|
||||
.orderBy(asc(executionWorkspaceRestorePoints.updatedAt), asc(executionWorkspaceRestorePoints.id)).limit(batchSize);
|
||||
for (const row of doomed) {
|
||||
try {
|
||||
await options.storage.deleteObject(row.companyId, row.objectKey);
|
||||
if ((await options.storage.headObject(row.companyId, row.objectKey)).exists) throw new Error("restore_point_object_still_present");
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "deleted", deletedAt: at, lastError: null, updatedAt: at })
|
||||
.where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
result.deleted += 1;
|
||||
} catch (error) {
|
||||
await db.update(executionWorkspaceRestorePoints)
|
||||
.set({ lastError: error instanceof Error ? error.message.slice(0, 500) : "delete_failed", updatedAt: at })
|
||||
.where(eq(executionWorkspaceRestorePoints.id, row.id));
|
||||
result.failed += 1;
|
||||
}
|
||||
}
|
||||
result.truncated = abandoned.length === batchSize || due.length === batchSize
|
||||
|| expiring.length === batchSize || doomed.length === batchSize;
|
||||
return result;
|
||||
}
|
||||
|
||||
async function nextDeadlineAt(): Promise<number | null> {
|
||||
const [row] = await db.select({
|
||||
deleteNow: sql<number>`count(*) FILTER (WHERE ${executionWorkspaceRestorePoints.status} IN ('expiring', 'superseded', 'delete_requested') AND ${executionWorkspaceRestorePoints.lastError} IS NULL)::int`,
|
||||
retryFrom: sql<string | null>`min(${executionWorkspaceRestorePoints.updatedAt}) FILTER (WHERE ${executionWorkspaceRestorePoints.status} IN ('expiring', 'superseded', 'delete_requested') AND ${executionWorkspaceRestorePoints.lastError} IS NOT NULL)`,
|
||||
pendingFrom: sql<string | null>`min(${executionWorkspaceRestorePoints.createdAt}) FILTER (WHERE ${executionWorkspaceRestorePoints.status} = 'pending')`,
|
||||
expiresAt: sql<string | null>`min(${executionWorkspaceRestorePoints.deleteAfter}) FILTER (WHERE ${executionWorkspaceRestorePoints.status} = 'active')`,
|
||||
}).from(executionWorkspaceRestorePoints);
|
||||
const at = now().getTime();
|
||||
const deadlines: number[] = [];
|
||||
if (row && row.deleteNow > 0) deadlines.push(at);
|
||||
if (row?.retryFrom) deadlines.push(new Date(row.retryFrom).getTime() + GIT_RESTORE_POINT_RETRY_MS);
|
||||
if (row?.pendingFrom) deadlines.push(new Date(row.pendingFrom).getTime() + pendingGraceMs);
|
||||
if (row?.expiresAt) deadlines.push(new Date(row.expiresAt).getTime());
|
||||
return deadlines.length > 0 ? Math.min(...deadlines) : null;
|
||||
}
|
||||
|
||||
async function hasActive(): Promise<boolean> {
|
||||
const [row] = await db.select({ id: executionWorkspaceRestorePoints.id }).from(executionWorkspaceRestorePoints)
|
||||
.where(eq(executionWorkspaceRestorePoints.status, "active")).limit(1);
|
||||
return Boolean(row);
|
||||
}
|
||||
|
||||
return { tick, nextDeadlineAt, hasActive };
|
||||
}
|
||||
|
||||
export function notifyGitRestorePointWork(transaction: object, hint?: string) {
|
||||
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.gitRestorePoints, hint);
|
||||
}
|
||||
|
||||
export async function requestRestorePointDeletion(
|
||||
tx: Pick<Db, "update">,
|
||||
scope: { sourceIssueId: string } | { companyId: string },
|
||||
) {
|
||||
await notifyGitRestorePointWork(tx, "sourceIssueId" in scope ? `issue:${scope.sourceIssueId}` : `company:${scope.companyId}`);
|
||||
await tx.update(executionWorkspaceRestorePoints)
|
||||
.set({ status: "delete_requested", updatedAt: new Date() })
|
||||
.where(and(
|
||||
"sourceIssueId" in scope
|
||||
? eq(executionWorkspaceRestorePoints.sourceIssueId, scope.sourceIssueId)
|
||||
: eq(executionWorkspaceRestorePoints.companyId, scope.companyId),
|
||||
inArray(executionWorkspaceRestorePoints.status, ["pending", "active", "expiring"]),
|
||||
));
|
||||
}
|
||||
|
||||
export function registerGitRestorePointWork(
|
||||
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
|
||||
service: ReturnType<typeof gitRestorePointRetentionService>,
|
||||
onResult: (result: GitRestorePointRetentionResult) => void,
|
||||
) {
|
||||
let next: number | null = null;
|
||||
let tracksActive = true;
|
||||
const worker = coordinator.register(DELIVERY_QUEUES.gitRestorePoints, {
|
||||
retryMs: GIT_RESTORE_POINT_RETRY_MS,
|
||||
run: async () => {
|
||||
next = null;
|
||||
const result = await service.tick();
|
||||
onResult(result);
|
||||
next = result.truncated ? Date.now() + 1000 : await service.nextDeadlineAt();
|
||||
tracksActive = await service.hasActive();
|
||||
},
|
||||
hasPending: async () => false,
|
||||
nextRunAt: () => next,
|
||||
});
|
||||
return { ...worker, issuesChanged: () => { if (tracksActive) worker.wake(); } };
|
||||
}
|
||||
@@ -12,6 +12,7 @@ import { type TaskWorkspaceSelection, type TaskWorkspaceIntent } from "@papercli
|
||||
import { executionWorkspaceService } from "./execution-workspaces.js";
|
||||
import { notifyChatPublicationWork } from "./chat-work-notifications.js";
|
||||
import { notifyTerminalWorkspaceWork } from "./terminal-workspace-work.js";
|
||||
import { requestRestorePointDeletion } from "./git-restore-point-ledger.js";
|
||||
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
|
||||
import { recordChatHandoff, recordChatCompletion, existingChatCompletionReply, acknowledgeChatCompletionReply } from "./chat-completion-delivery.js";
|
||||
import { mirrorSlackBoardComment, slackBoardReplyBindings } from "./slack-board-messages.js";
|
||||
@@ -11696,6 +11697,7 @@ export function issueService(db: Db) {
|
||||
let removedIssue;
|
||||
try {
|
||||
await notifyTerminalWorkspaceWork(tx, `issue:${id}`);
|
||||
await requestRestorePointDeletion(tx, { sourceIssueId: id });
|
||||
removedIssue = await tx
|
||||
.delete(issues)
|
||||
.where(eq(issues.id, id))
|
||||
|
||||
Reference in new issue
Block a user