Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-23 17:33:51 +02:00
Code Issues Packages Projects Releases Wiki Activity

feat: give nested managed checkouts their own restore points (#17257)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #17248 lets git cache expiration remove a closed issue's worktree
with unsaved work after saving an encrypted restore point
> - Workspaces with nested `.paperclip-repositories` checkouts were
still kept whenever a checkout had unpushed or uncommitted work
> - On one instance that was the second most common reason expiration
skipped a workspace
> - This pull request gives each nested checkout its own restore point
and restores it when the workspace comes back
> - The benefit is that multi-repository workspaces can be cleaned up as
safely as single-repository ones

## Linked Issues or Issue Description

Refs #17238, #17241, #17248. Bug report described in-PR (bug report
template).

**What happened?**
With restore points enabled, a workspace whose nested managed checkout
has unpushed commits or uncommitted files is still skipped with
`unpushed_managed_checkout` or `dirty_or_untracked_files`. On one
instance, `unpushed_managed_checkout` accounted for 32 of the last 111
expiration checks.

**Expected behavior**
A nested checkout's unsaved work is saved like the main worktree's, and
the workspace can expire and later be restored exactly.

**Steps to reproduce**
1. Use a project with an extra repository, so the task worktree contains
`.paperclip-repositories/<name>`.
2. Commit in that checkout without pushing, then close the issue.
3. With restore points enabled, the workspace is never expired.

**Paperclip version or commit:** `master` at 6d44b80a5e.
**Deployment mode:** any.
**Agent adapter(s) involved:** not adapter-specific (core).

## What Changed

**Relaxed checkout check** (`assertReproducibleManagedCheckout(..., {
restorePoint: true })`, used only in restore point mode):
- **Allowed:** the checked-out branch's unpushed commits, plus its
uncommitted, untracked and history-only work.
- **Still blocks:**
- another local branch or tag whose tip the remote lacks and `HEAD` does
not contain (`unpushed_managed_checkout_branches`), because restore
points do not recreate other branches;
- commits in any reflog that are neither on the remote nor in the
bundled history (`HEAD`'s reflog and `ORIG_HEAD`), such as work left in
another branch's reflog;
  - objects reachable from nothing;
  - a `COMMIT_EDITMSG` that matches no commit in the checkout's history.
- **`COMMIT_EDITMSG`:** a nested checkout is a full clone, so its base
git storage check now accepts a `COMMIT_EDITMSG` that matches a commit
in its history, in strict and relaxed mode. It is read only if it is a
regular file of at most 64 KB, opened without following symlinks and
without blocking; anything else keeps the workspace
(`unsupported_commit_message`). The main worktree's existing
`COMMIT_EDITMSG` read uses the same bounded reader. A clean checkout
with only an unpushed commit therefore reports
`unpushed_managed_checkout`, which restore point mode can relax, instead
of `unknown_base_git_files`.

**Capture** (`git-workspace-expiration.ts`):
- **Expiration flow:** strict inspection still runs first.
`unpushed_managed_checkout` is now relaxable. The relaxed prover records
which checkouts need restore points.
- **Storage:** each checkout's restore point is stored under
`.../<generation>/checkouts/<name>/<id>.pcrp`, against a remote-verified
boundary from that checkout. It is recorded on its
`contents.checkouts[]` entry.
- **Activation:** all restore points of the generation are activated
together in the archive transaction. Any nested restore point makes the
recipe `paperclip.git-cache-restoration.v2`, so older servers reject it.
- **Schema:** migration `0374` adds `checkout_name` to
`execution_workspace_restore_points` and includes it in the
live-generation unique index.
- **Final check:** under the expiration locks, each checkout with a
restore point gets the relaxed check and must match its own snapshot.
Checkouts without one get the strict check and must still be clean.
Every restore point, the main worktree's included, records a digest of
its `HEAD` reflog and `ORIG_HEAD` at capture, and that history must be
unchanged, so a commit followed by a reset after capture rolls back.
Rollback drops every restore point reference, returns the recipe to v1,
and supersedes the stored restore points.

**Restore** (`git-workspace-contents.ts`):
- **Re-cloned checkouts:** re-cloning a checkout imports its bundle
before the saved commit is needed. After checking out the saved commit,
it applies and verifies the restore point.
- **Existing checkouts:** the same helper as the main worktree marks a
matching checkout applied, and fails closed instead of overwriting a
diverged one.

**Validation:** `readGitCacheRestoration` and
`savedGitWorkspaceRestoration` validate nested restore point references
against their checkout names. Contents comparison ignores the reference.

**Docs:** `doc/recoverable-disk-cleanup.md` describes nested checkouts.

No code comments were added.

## Verification

- `pnpm exec vitest run src/__tests__/git-workspace-expiration.test.ts
-t "restore points for nested checkouts"` (in `server/`): 10 passed,
with a real nested checkout and embedded Postgres. The cases:
- **Round trip:** an unpushed commit and an untracked file in the nested
checkout survive expiration and reopen. The recipe is v2, the restore
point lives on the checkout entry, and a ledger row carries the checkout
name.
- **Another branch:** unpushed work on another local branch keeps the
workspace.
- **Change after capture:** a nested change after capture rolls back,
keeps the files, returns the recipe to v1, and supersedes the restore
point.
- **Only an unpushed commit:** a clean nested checkout with only an
unpushed commit (and its `COMMIT_EDITMSG`) expires and restores.
- **Reflog-only history:** unpushed history held only in another
branch's reflog keeps the workspace.
- **History after capture:** a commit followed by a reset after capture
rolls back, in a nested checkout and in the main worktree.
- **Unsafe `COMMIT_EDITMSG`:** a named pipe, an oversized file, or a
symlink keeps the workspace without reading it unsafely.
- **Mutation checks:** each of these fails a case:
- removing the other-branch check, the per-checkout snapshot check, the
rollback cleanup, or the checkout apply;
  - rejecting a saved `COMMIT_EDITMSG`;
  - dropping reflog coverage;
  - dropping the history digest check;
  - removing the size cap.

Not covered by a dedicated case: a workspace with two nested checkouts
where only one has a restore point.
- `pnpm exec vitest run git-workspace-expiration git-workspace-contents
git-restore-point execution-workspaces` (in `server/`): 232 passed.
- `pnpm run check:migrations` (in `packages/db`): pass.
- `pnpm run typecheck` (in `server/`): pass.
- The server builds, the runtime bundle builds, and
`scripts/server-boot-smoke.sh node server/dist-runtime/index.mjs`
reaches healthy.

## Risks

- **Off by default:** nothing changes unless
`PAPERCLIP_GIT_RESTORE_POINTS_ENABLED=true`.
- **Other local branches:** a nested checkout's other local branches are
not saved. Any of them with commits the remote lacks keeps the workspace
instead.
- **Migration:** adds a defaulted column and replaces one partial unique
index on a table only written when restore points are enabled.
- **Rollback:** disable the setting. Workspaces already expired with
nested restore points keep v2 recipes that require this code to restore.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), 1M context, through Claude Code
with tool use (shell, file edits, tests, local server builds and boots,
read-only measurements on live instances).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
Devin FoleyandPaperclip authored and GitHub committed 2026-10-22 18:42:22 -07:00
1 parent 989ee668e0
commit 650581b2f0
11 files changed
+861 -70
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/recoverable-disk-cleanup.md
+17 -2
View File
@@ -114,8 +114,23 @@ the same transaction that archives the workspace with a versioned recipe
(`paperclip.git-cache-restoration.v2`). Older servers reject that recipe instead of
restoring without the saved work. Under the expiration locks, the worktree must still
match the captured snapshot exactly; otherwise expiration rolls back, keeps the
worktree, and drops the restore point. Nested managed checkouts, submodules, special
git state and ignored files other than `.env` still keep a worktree from expiring.
worktree, and drops the restore point.
Nested managed checkouts under `.paperclip-repositories/` get their own restore points
when their checked-out branch has unpushed commits or uncommitted, untracked or
history-only work. Each is stored under its checkout name, recorded in the recipe's
contents, and verified against its own snapshot under the expiration locks. A nested
checkout still blocks expiration when another local branch or tag has commits its
remote lacks, when any reflog holds commits that are neither on the remote nor part of
the checked-out history the restore point bundles, when it has objects reachable from
nothing, or when its `COMMIT_EDITMSG` holds a draft that matches no commit in its
history. A `COMMIT_EDITMSG` that matches a commit is accepted for every nested checkout.
Each restore point also records a digest of its `HEAD` reflog and `ORIG_HEAD`; under the
expiration locks that history must be unchanged too, so a commit followed by a reset
after capture rolls expiration back. Reopening
re-clones the checkout, imports its bundle, checks out the saved commit and applies the
restore point. Submodules, special git state and ignored files other than `.env`
still keep a worktree from expiring.
Restore points are refused when the file storage is `s3` and the secrets master key
does not come from `PAPERCLIP_SECRETS_MASTER_KEY`.
packages/db/src/migrations/0374_rare_wolfpack.sql
+3
View File
@@ -0,0 +1,3 @@
DROP INDEX "execution_workspace_restore_points_live_generation_uniq";--> statement-breakpoint
ALTER TABLE "execution_workspace_restore_points" ADD COLUMN "checkout_name" text DEFAULT '' NOT NULL;--> statement-breakpoint
CREATE UNIQUE INDEX "execution_workspace_restore_points_live_checkout_uniq" ON "execution_workspace_restore_points" USING btree ("execution_workspace_id","generation","checkout_name") WHERE "execution_workspace_restore_points"."status" <> 'deleted';
packages/db/src/migrations/meta/0369_snapshot.json → packages/db/src/migrations/meta/0374_snapshot.json
Generated
+479 -3
View File
@@ -1,6 +1,6 @@
{
"id": "9fa514b5-5cd1-4350-9b77-f60924655298",
"prevId": "a9c12cbc-ed4b-4e5d-9d33-36f9a7fb746c",
"id": "945c31ed-f39f-4272-a91d-85de925f5361",
"prevId": "406521f8-5352-415b-805d-786854381e03",
"version": "7",
"dialect": "postgresql",
"tables": {
@@ -8252,6 +8252,24 @@
"name": "case_handoffs",
"schema": "",
"columns": {
"workflow_transition": {
"name": "workflow_transition",
"type": "jsonb",
"primaryKey": false,
"notNull": false
},
"workflow_prompt": {
"name": "workflow_prompt",
"type": "text",
"primaryKey": false,
"notNull": false
},
"base_workflow_document_revision_id": {
"name": "base_workflow_document_revision_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"id": {
"name": "id",
"type": "uuid",
@@ -9067,6 +9085,12 @@
"primaryKey": false,
"notNull": false
},
"workflow_id": {
"name": "workflow_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"handoff_version": {
"name": "handoff_version",
"type": "integer",
@@ -9410,6 +9434,19 @@
"onDelete": "restrict",
"onUpdate": "no action"
},
"cases_workflow_id_workflows_id_fk": {
"name": "cases_workflow_id_workflows_id_fk",
"tableFrom": "cases",
"tableTo": "workflows",
"columnsFrom": [
"workflow_id"
],
"columnsTo": [
"id"
],
"onDelete": "restrict",
"onUpdate": "no action"
},
"cases_parent_case_id_cases_id_fk": {
"name": "cases_parent_case_id_cases_id_fk",
"tableFrom": "cases",
@@ -17136,6 +17173,12 @@
"primaryKey": false,
"notNull": false
},
"owner_project_id": {
"name": "owner_project_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"owner_plugin_id": {
"name": "owner_plugin_id",
"type": "uuid",
@@ -17370,6 +17413,40 @@
"method": "btree",
"with": {}
},
"company_secrets_project_instance_owner_key_uq": {
"name": "company_secrets_project_instance_owner_key_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "owner_project_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "owner_plugin_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"company_secrets\".\"scope\" = 'instance' and \"company_secrets\".\"deleted_at\" is null",
"concurrently": false,
"method": "btree",
"with": {}
},
"company_secrets_instance_owner_key_uq": {
"name": "company_secrets_instance_owner_key_uq",
"columns": [
@@ -17572,6 +17649,21 @@
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"company_secrets_company_id_owner_project_id_projects_company_id_id_fk": {
"name": "company_secrets_company_id_owner_project_id_projects_company_id_id_fk",
"tableFrom": "company_secrets",
"tableTo": "projects",
"columnsFrom": [
"company_id",
"owner_project_id"
],
"columnsTo": [
"company_id",
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
@@ -17580,7 +17672,7 @@
"checkConstraints": {
"company_secrets_scope_shape_check": {
"name": "company_secrets_scope_shape_check",
"value": "(\n \"company_secrets\".\"scope\" = 'company'\n and \"company_secrets\".\"owner_user_id\" is null\n and \"company_secrets\".\"user_secret_definition_id\" is null\n and \"company_secrets\".\"owner_agent_id\" is null and \"company_secrets\".\"owner_plugin_id\" is null\n ) or (\n \"company_secrets\".\"scope\" = 'user'\n and \"company_secrets\".\"owner_user_id\" is not null\n and \"company_secrets\".\"user_secret_definition_id\" is not null\n and \"company_secrets\".\"owner_agent_id\" is null and \"company_secrets\".\"owner_plugin_id\" is null\n ) or (\n \"company_secrets\".\"scope\" = 'instance'\n and \"company_secrets\".\"owner_user_id\" is null and \"company_secrets\".\"user_secret_definition_id\" is null\n and \"company_secrets\".\"owner_agent_id\" is not null and \"company_secrets\".\"owner_plugin_id\" is not null\n )"
"value": "(\n \"company_secrets\".\"scope\" = 'company'\n and \"company_secrets\".\"owner_user_id\" is null\n and \"company_secrets\".\"user_secret_definition_id\" is null\n and \"company_secrets\".\"owner_agent_id\" is null and \"company_secrets\".\"owner_project_id\" is null and \"company_secrets\".\"owner_plugin_id\" is null\n ) or (\n \"company_secrets\".\"scope\" = 'user'\n and \"company_secrets\".\"owner_user_id\" is not null\n and \"company_secrets\".\"user_secret_definition_id\" is not null\n and \"company_secrets\".\"owner_agent_id\" is null and \"company_secrets\".\"owner_project_id\" is null and \"company_secrets\".\"owner_plugin_id\" is null\n ) or (\n \"company_secrets\".\"scope\" = 'instance'\n and \"company_secrets\".\"owner_user_id\" is null and \"company_secrets\".\"user_secret_definition_id\" is null\n and (\"company_secrets\".\"owner_agent_id\" is not null) <> (\"company_secrets\".\"owner_project_id\" is not null)\n and \"company_secrets\".\"owner_plugin_id\" is not null\n )"
}
},
"isRLSEnabled": false
@@ -28381,6 +28473,250 @@
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_restore_points": {
"name": "execution_workspace_restore_points",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"execution_workspace_id": {
"name": "execution_workspace_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"source_issue_id": {
"name": "source_issue_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"generation": {
"name": "generation",
"type": "integer",
"primaryKey": false,
"notNull": true
},
"checkout_name": {
"name": "checkout_name",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "''"
},
"object_key": {
"name": "object_key",
"type": "text",
"primaryKey": false,
"notNull": true
},
"sha256": {
"name": "sha256",
"type": "text",
"primaryKey": false,
"notNull": true
},
"byte_size": {
"name": "byte_size",
"type": "bigint",
"primaryKey": false,
"notNull": true
},
"status": {
"name": "status",
"type": "text",
"primaryKey": false,
"notNull": true,
"default": "'pending'"
},
"activated_at": {
"name": "activated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"tree_closed_at": {
"name": "tree_closed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"delete_after": {
"name": "delete_after",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"review_generation": {
"name": "review_generation",
"type": "integer",
"primaryKey": false,
"notNull": true,
"default": 0
},
"deleted_at": {
"name": "deleted_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"last_error": {
"name": "last_error",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"execution_workspace_restore_points_object_key_uniq": {
"name": "execution_workspace_restore_points_object_key_uniq",
"columns": [
{
"expression": "object_key",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_restore_points_live_checkout_uniq": {
"name": "execution_workspace_restore_points_live_checkout_uniq",
"columns": [
{
"expression": "execution_workspace_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "generation",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "checkout_name",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"where": "\"execution_workspace_restore_points\".\"status\" <> 'deleted'",
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_restore_points_status_delete_after_idx": {
"name": "execution_workspace_restore_points_status_delete_after_idx",
"columns": [
{
"expression": "status",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "delete_after",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_restore_points_company_idx": {
"name": "execution_workspace_restore_points_company_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_restore_points_source_issue_idx": {
"name": "execution_workspace_restore_points_source_issue_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "source_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
},
"execution_workspace_restore_points_active_source_idx": {
"name": "execution_workspace_restore_points_active_source_idx",
"columns": [
{
"expression": "source_issue_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"where": "\"execution_workspace_restore_points\".\"status\" = 'active'",
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.execution_workspace_runtime_leases": {
"name": "execution_workspace_runtime_leases",
"schema": "",
@@ -38535,6 +38871,18 @@
"primaryKey": false,
"notNull": false
},
"dismissed_at": {
"name": "dismissed_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": false
},
"dismissed_by_user_id": {
"name": "dismissed_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
@@ -47474,6 +47822,27 @@
}
},
"indexes": {
"projects_company_id_uq": {
"name": "projects_company_id_uq",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
},
{
"expression": "id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": true,
"concurrently": false,
"method": "btree",
"with": {}
},
"projects_company_idx": {
"name": "projects_company_idx",
"columns": [
@@ -59203,6 +59572,113 @@
},
"isRLSEnabled": false
},
"public.workflows": {
"name": "workflows",
"schema": "",
"columns": {
"id": {
"name": "id",
"type": "uuid",
"primaryKey": true,
"notNull": true,
"default": "gen_random_uuid()"
},
"company_id": {
"name": "company_id",
"type": "uuid",
"primaryKey": false,
"notNull": true
},
"name": {
"name": "name",
"type": "text",
"primaryKey": false,
"notNull": true
},
"prompt": {
"name": "prompt",
"type": "text",
"primaryKey": false,
"notNull": true
},
"created_by_agent_id": {
"name": "created_by_agent_id",
"type": "uuid",
"primaryKey": false,
"notNull": false
},
"created_by_user_id": {
"name": "created_by_user_id",
"type": "text",
"primaryKey": false,
"notNull": false
},
"created_at": {
"name": "created_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
},
"updated_at": {
"name": "updated_at",
"type": "timestamp with time zone",
"primaryKey": false,
"notNull": true,
"default": "now()"
}
},
"indexes": {
"workflows_company_idx": {
"name": "workflows_company_idx",
"columns": [
{
"expression": "company_id",
"isExpression": false,
"asc": true,
"nulls": "last"
}
],
"isUnique": false,
"concurrently": false,
"method": "btree",
"with": {}
}
},
"foreignKeys": {
"workflows_company_id_companies_id_fk": {
"name": "workflows_company_id_companies_id_fk",
"tableFrom": "workflows",
"tableTo": "companies",
"columnsFrom": [
"company_id"
],
"columnsTo": [
"id"
],
"onDelete": "cascade",
"onUpdate": "no action"
},
"workflows_created_by_agent_id_agents_id_fk": {
"name": "workflows_created_by_agent_id_agents_id_fk",
"tableFrom": "workflows",
"tableTo": "agents",
"columnsFrom": [
"created_by_agent_id"
],
"columnsTo": [
"id"
],
"onDelete": "set null",
"onUpdate": "no action"
}
},
"compositePrimaryKeys": {},
"uniqueConstraints": {},
"policies": {},
"checkConstraints": {},
"isRLSEnabled": false
},
"public.workspace_operations": {
"name": "workspace_operations",
"schema": "",
packages/db/src/migrations/meta/_journal.json
Generated
+7
View File
@@ -2605,6 +2605,13 @@
"when": 1792710084082,
"tag": "0373_wandering_mongu",
"breakpoints": true
},
{
"idx": 374,
"version": "7",
"when": 1792713110842,
"tag": "0374_rare_wolfpack",
"breakpoints": true
}
]
}
packages/db/src/schema/execution_workspace_restore_points.ts
+3 -2
View File
@@ -19,6 +19,7 @@ export const executionWorkspaceRestorePoints = pgTable(
executionWorkspaceId: uuid("execution_workspace_id").notNull(),
sourceIssueId: uuid("source_issue_id"),
generation: integer("generation").notNull(),
checkoutName: text("checkout_name").notNull().default(""),
objectKey: text("object_key").notNull(),
sha256: text("sha256").notNull(),
byteSize: bigint("byte_size", { mode: "number" }).notNull(),
@@ -34,8 +35,8 @@ export const executionWorkspaceRestorePoints = pgTable(
},
(table) => ({
objectKeyUniqueIdx: uniqueIndex("execution_workspace_restore_points_object_key_uniq").on(table.objectKey),
liveGenerationUniqueIdx: uniqueIndex("execution_workspace_restore_points_live_generation_uniq")
.on(table.executionWorkspaceId, table.generation)
liveGenerationUniqueIdx: uniqueIndex("execution_workspace_restore_points_live_checkout_uniq")
.on(table.executionWorkspaceId, table.generation, table.checkoutName)
.where(sql`${table.status} <> 'deleted'`),
statusDeleteAfterIdx: index("execution_workspace_restore_points_status_delete_after_idx").on(table.status, table.deleteAfter),
companyIdx: index("execution_workspace_restore_points_company_idx").on(table.companyId),
server/src/__tests__/git-workspace-expiration.test.ts
+136 -1
View File
@@ -7,7 +7,8 @@ import { pathToFileURL } from "node:url";
import { createRequire } from "node:module";
import { afterAll, afterEach, beforeAll, describe, expect, it, vi } from "vitest";
import { eq, sql } from "drizzle-orm";
import { companies, projects, projectWorkspaces, issues, executionWorkspaces, agents, heartbeatRuns, nativeRunFinalizations, workspaceRuntimeServices, createDb } from "@paperclipai/db";
import { companies, projects, projectWorkspaces, issues, executionWorkspaces, agents, heartbeatRuns, nativeRunFinalizations, workspaceRuntimeServices, executionWorkspaceRestorePoints, createDb } from "@paperclipai/db";
import { getStorageService } from "../storage/index.js";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveManagedProjectWorkspaceDir } from "../home-paths.js";
import { materializeManagedProjectWorkspace } from "../services/managed-repository-checkout.js";
@@ -197,6 +198,140 @@ describeEmbeddedPostgres("reconstructable Git expiration", () => {
expect(git(f.nested.cwd, "rev-parse", "HEAD")).not.toBe(f.nestedA);
});
describe("restore points for nested checkouts", () => {
const restorePoint = () => ({ storage: getStorageService(), notBefore: new Date(Date.now() - 24 * 3_600_000), keyDurable: true });
const ledger = (workspaceId: string) => db.select().from(executionWorkspaceRestorePoints)
.where(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId));
async function unpushedNestedWork(nestedCwd: string) {
await fs.writeFile(path.join(nestedCwd, "nested.txt"), "nested B (unpushed)\n");
git(nestedCwd, "commit", "-am", "Nested B unpushed");
const unpushed = git(nestedCwd, "rev-parse", "HEAD");
await fs.writeFile(path.join(nestedCwd, "draft.md"), "nested untracked work\n");
return unpushed;
}
it("expires a workspace whose nested checkout has unsaved work and restores the checkout exactly", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
const unpushed = await unpushedNestedWork(f.nested.cwd);
expect(await f.expire({ expireBase: false })).toMatchObject({ expired: false, skipReason: "dirty_or_untracked_files" });
const result = await f.expire({ expireBase: false, restorePoint: restorePoint() });
expect(result).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(recipe.schema).toBe("paperclip.git-cache-restoration.v2");
expect(recipe.restorePoint).toBeUndefined();
expect(recipe.contents.checkouts[0].restorePoint).toMatchObject({ head: unpushed, headOnRemote: false });
expect(await ledger(f.workspaceId)).toMatchObject([{ status: "active", checkoutName: recipe.contents.checkouts[0].name }]);
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.nested.cwd, "rev-parse", "HEAD")).toBe(unpushed);
expect(await fs.readFile(path.join(f.nested.cwd, "draft.md"), "utf8")).toBe("nested untracked work\n");
expect(await fs.readFile(f.nestedSettingsPath)).toEqual(f.settings);
vi.unstubAllEnvs();
});
it("expires and restores a nested checkout that only has an unpushed commit", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await fs.writeFile(path.join(f.nested.cwd, "nested.txt"), "nested B (unpushed)\n");
git(f.nested.cwd, "commit", "-am", "Nested B unpushed");
const unpushed = git(f.nested.cwd, "rev-parse", "HEAD");
expect(await exists(path.join(f.nested.cwd, ".git", "COMMIT_EDITMSG"))).toBe(true);
expect(await f.expire({ expireBase: false })).toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout" });
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() })).toMatchObject({ expired: true, restorePoint: true, skipReason: null });
expect(await f.reopen()).toMatchObject({ ok: true, reopened: true });
expect(git(f.nested.cwd, "rev-parse", "HEAD")).toBe(unpushed);
vi.unstubAllEnvs();
});
it("keeps a workspace whose nested checkout holds unpushed history only in another branch's reflog", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
const tree = git(f.nested.cwd, "rev-parse", "HEAD^{tree}");
const orphan = execFileSync("git", ["commit-tree", tree, "-p", "HEAD", "-m", "Only in a reflog"],
{ cwd: f.nested.cwd, env: gitEnv, encoding: "utf8" }).trim();
git(f.nested.cwd, "update-ref", "--create-reflog", "refs/heads/side", orphan);
git(f.nested.cwd, "update-ref", "refs/heads/side", f.nestedA);
await fs.writeFile(path.join(f.nested.cwd, "draft.md"), "untracked\n");
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() }))
.toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout_branches" });
expect(await exists(f.cwd)).toBe(true);
vi.unstubAllEnvs();
});
it("rolls back when new history appears after capture without changing files", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await unpushedNestedWork(f.nested.cwd);
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
git(f.nested.cwd, "commit", "--allow-empty", "-q", "-m", "Made after capture");
git(f.nested.cwd, "reset", "-q", "--soft", "HEAD~1");
} });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await exists(f.cwd)).toBe(true);
vi.unstubAllEnvs();
});
it("rolls back when the main worktree gains history after capture without changing files", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await fs.writeFile(path.join(f.cwd, "draft.md"), "untracked main work\n");
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(), beforeDelete: async () => {
git(f.cwd, "commit", "--allow-empty", "-q", "-m", "Made after capture");
git(f.cwd, "reset", "-q", "--soft", "HEAD~1");
} });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await exists(f.cwd)).toBe(true);
vi.unstubAllEnvs();
});
it.each([
["a named pipe", async (file: string) => { execFileSync("mkfifo", [file]); }],
["an oversized file", async (file: string) => { await fs.writeFile(file, "x".repeat(65 * 1024)); }],
["a symlink", async (file: string) => { await fs.symlink("/dev/zero", file); }],
])("keeps a workspace whose nested COMMIT_EDITMSG is %s without reading it unsafely", async (_name, plant) => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await unpushedNestedWork(f.nested.cwd);
const file = path.join(f.nested.cwd, ".git", "COMMIT_EDITMSG");
await fs.rm(file, { force: true });
await plant(file);
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() }))
.toMatchObject({ expired: false, skipReason: "unsupported_commit_message" });
expect(await exists(f.cwd)).toBe(true);
vi.unstubAllEnvs();
});
it("keeps a workspace whose nested checkout has unpushed work on another branch", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
git(f.nested.cwd, "checkout", "-q", "-b", "side");
await fs.writeFile(path.join(f.nested.cwd, "side.txt"), "side branch work\n");
git(f.nested.cwd, "add", "side.txt"); git(f.nested.cwd, "commit", "-q", "-m", "Side work");
git(f.nested.cwd, "checkout", "-q", "main");
await fs.writeFile(path.join(f.nested.cwd, "draft.md"), "untracked\n");
expect(await f.expire({ expireBase: false, restorePoint: restorePoint() }))
.toMatchObject({ expired: false, skipReason: "unpushed_managed_checkout_branches" });
expect(await exists(f.cwd)).toBe(true);
expect(await ledger(f.workspaceId)).toEqual([]);
vi.unstubAllEnvs();
});
it("keeps the workspace and drops every restore point when a nested checkout changes after capture", async () => {
vi.stubEnv("PAPERCLIP_SECRETS_MASTER_KEY", "31".repeat(32));
const f = await nestedFixture(false);
await unpushedNestedWork(f.nested.cwd);
const result = await f.expire({ expireBase: false, restorePoint: restorePoint(),
beforeDelete: async () => { await fs.writeFile(path.join(f.nested.cwd, "draft.md"), "changed after capture\n"); } });
expect(result).toMatchObject({ expired: false, skipReason: "restore_point_contents_changed" });
expect(await fs.readFile(path.join(f.nested.cwd, "draft.md"), "utf8")).toBe("changed after capture\n");
const recipe = (await f.row()).metadata?.gitCacheRestoration as Record<string, any>;
expect(recipe.schema).toBe("paperclip.git-cache-restoration.v1");
expect(recipe.contents.checkouts[0].restorePoint).toBeUndefined();
expect(await ledger(f.workspaceId)).toMatchObject([{ status: "superseded" }]);
vi.unstubAllEnvs();
});
});
it("retains custom base ignore patterns after expiring the nested workspace", async () => {
const f = await nestedFixture(false);
const exclude = path.join(f.baseCwd, ".git", "info", "exclude");
server/src/services/git-restore-point-storage.ts
+33 -12
View File
@@ -1,5 +1,5 @@
import { createHash, randomUUID } from "node:crypto";
import { and, eq, inArray, ne, sql } from "drizzle-orm";
import { and, eq, inArray, notInArray, sql } from "drizzle-orm";
import { executionWorkspaceRestorePoints, executionWorkspaces, type Db } from "@paperclipai/db";
import type { StoredSecretVersionMaterial } from "../secrets/types.js";
import type { StorageService } from "../storage/types.js";
@@ -10,6 +10,7 @@ import {
decodeRestorePointPayload,
importRestorePoint,
openRestorePoint,
restorePointHistoryDigest,
RestorePointRestoreFailure,
sealRestorePoint,
type RestorePointLimits,
@@ -34,6 +35,7 @@ export type GitRestorePointRef = {
index: string | null;
worktree: string | null;
modes: Array<{ path: string; mode: number }>;
history?: string;
headOnRemote: boolean;
appliedAt?: string | null;
expiredAt?: string;
@@ -41,20 +43,28 @@ export type GitRestorePointRef = {
type Tx = Parameters<Parameters<Db["transaction"]>[0]>[0];
function restorePointObjectKey(companyId: string, workspaceId: string, generation: number, id: string) {
return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/${id}.pcrp`;
const CHECKOUT_NAME = /^[A-Za-z0-9._-]{1,200}$/;
function restorePointObjectKey(companyId: string, workspaceId: string, generation: number, id: string, checkout?: string) {
if (checkout !== undefined && (!CHECKOUT_NAME.test(checkout) || checkout === "." || checkout === "..")) {
throw new RestorePointRestoreFailure("invalid_restore_point_reference");
}
return `${companyId}/${NAMESPACE}/${workspaceId}/${generation}/${checkout ? `checkouts/${checkout}/` : ""}${id}.pcrp`;
}
export function validateGitRestorePointRef(value: unknown, workspace: { companyId: string; id: string }): GitRestorePointRef {
export function validateGitRestorePointRef(value: unknown, workspace: { companyId: string; id: string }, checkout?: string): GitRestorePointRef {
const ref = value as GitRestorePointRef | null;
const sha = /^[0-9a-f]{40}$/;
let expectedKey: string | null = null;
try { expectedKey = ref ? restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id, checkout) : null; } catch { expectedKey = null; }
if (!ref || ref.schema !== GIT_RESTORE_POINT_SCHEMA || typeof ref.id !== "string"
|| ref.objectKey !== restorePointObjectKey(workspace.companyId, workspace.id, ref.generation, ref.id)
|| expectedKey === null || ref.objectKey !== expectedKey
|| !/^[0-9a-f]{64}$/.test(ref.sha256) || !Number.isSafeInteger(ref.byteSize) || ref.byteSize <= 0 || ref.byteSize > MAX_RESTORE_POINT_BYTES
|| !ref.key || typeof ref.key !== "object" || Array.isArray(ref.key)
|| !Number.isSafeInteger(ref.generation) || ref.generation < 1
|| !sha.test(ref.head) || (ref.index !== null && !sha.test(ref.index)) || (ref.worktree !== null && !sha.test(ref.worktree))
|| (ref.index === null) !== (ref.worktree === null) || typeof ref.headOnRemote !== "boolean"
|| (ref.history !== undefined && !/^[0-9a-f]{64}$/.test(ref.history))
|| !Array.isArray(ref.modes) || ref.modes.length > 20_000 || ref.modes.some((entry) => !entry || typeof entry.path !== "string"
|| entry.path.startsWith("/") || entry.path.split("/").includes("..") || !Number.isInteger(entry.mode) || entry.mode < 0 || entry.mode > 0o777)
|| (ref.appliedAt != null && !Number.isFinite(Date.parse(ref.appliedAt)))
@@ -70,17 +80,20 @@ export async function storeGitRestorePoint(input: {
workspace: { companyId: string; id: string; sourceIssueId: string | null };
cwd: string;
generation: number;
checkout?: string;
upstreamCommits: readonly string[];
headOnRemote: boolean;
limits?: Partial<RestorePointLimits>;
signal?: AbortSignal;
}): Promise<{ ref: GitRestorePointRef; ledgerId: string; manifest: RestorePointManifest } | null> {
const history = await restorePointHistoryDigest(input.cwd, input.signal);
const captured = await captureRestorePoint({ cwd: input.cwd, upstreamCommits: input.upstreamCommits, limits: input.limits, signal: input.signal });
if (!captured) return null;
if (await restorePointHistoryDigest(input.cwd, input.signal) !== history) throw new RestorePointRestoreFailure("restore_point_history_changed");
const id = randomUUID();
const identity = { companyId: input.workspace.companyId, workspaceId: input.workspace.id, restorePointId: id, generation: input.generation };
const sealed = await sealRestorePoint(captured.payload, identity);
const objectKey = restorePointObjectKey(input.workspace.companyId, input.workspace.id, input.generation, id);
const objectKey = restorePointObjectKey(input.workspace.companyId, input.workspace.id, input.generation, id, input.checkout);
const [ledger] = await input.db.transaction(async (tx) => {
await notifyGitRestorePointWork(tx, `workspace:${input.workspace.id}`);
return tx.insert(executionWorkspaceRestorePoints).values({
@@ -88,6 +101,7 @@ export async function storeGitRestorePoint(input: {
executionWorkspaceId: input.workspace.id,
sourceIssueId: input.workspace.sourceIssueId,
generation: input.generation,
checkoutName: input.checkout ?? "",
objectKey,
sha256: sealed.sha256,
byteSize: sealed.ciphertext.length,
@@ -127,24 +141,30 @@ export async function storeGitRestorePoint(input: {
index: manifest.index,
worktree: manifest.worktree,
modes: manifest.modes,
history,
headOnRemote: input.headOnRemote,
appliedAt: null,
},
};
}
export async function activateGitRestorePoint(tx: Tx, input: { ledgerId: string; workspaceId: string; at: Date }) {
export async function activateGitRestorePoints(tx: Tx, input: { ledgerIds: string[]; workspaceId: string; at: Date }) {
if (input.ledgerIds.length === 0) return;
await notifyGitRestorePointWork(tx, `workspace:${input.workspaceId}`);
await tx.update(executionWorkspaceRestorePoints)
.set({ status: "superseded", updatedAt: input.at })
.where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, input.workspaceId),
ne(executionWorkspaceRestorePoints.id, input.ledgerId),
notInArray(executionWorkspaceRestorePoints.id, input.ledgerIds),
inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
const activated = await tx.update(executionWorkspaceRestorePoints)
.set({ status: "active", activatedAt: input.at, deleteAfter: input.at, updatedAt: input.at })
.where(and(eq(executionWorkspaceRestorePoints.id, input.ledgerId), eq(executionWorkspaceRestorePoints.status, "pending")))
.where(and(inArray(executionWorkspaceRestorePoints.id, input.ledgerIds), eq(executionWorkspaceRestorePoints.status, "pending")))
.returning({ id: executionWorkspaceRestorePoints.id });
if (activated.length !== 1) throw new RestorePointRestoreFailure("restore_point_not_pending");
if (activated.length !== input.ledgerIds.length) throw new RestorePointRestoreFailure("restore_point_not_pending");
}
export async function restorePointHistoryUnchanged(ref: GitRestorePointRef, cwd: string, signal?: AbortSignal) {
return ref.history === undefined || await restorePointHistoryDigest(cwd, signal) === ref.history;
}
export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: { head: string; index: string | null; worktree: string | null; modes: Array<{ path: string; mode: number }> }) {
@@ -152,10 +172,11 @@ export function restorePointSnapshotMatches(ref: GitRestorePointRef, snapshot: {
&& JSON.stringify(snapshot.modes) === JSON.stringify(ref.modes);
}
export async function abandonGitRestorePointById(db: Db, workspaceId: string, objectKey: string) {
export async function abandonGitRestorePointsByKey(db: Db, workspaceId: string, objectKeys: string[]) {
if (objectKeys.length === 0) return;
await db.update(executionWorkspaceRestorePoints)
.set({ status: "superseded", updatedAt: new Date() })
.where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId), eq(executionWorkspaceRestorePoints.objectKey, objectKey),
.where(and(eq(executionWorkspaceRestorePoints.executionWorkspaceId, workspaceId), inArray(executionWorkspaceRestorePoints.objectKey, objectKeys),
inArray(executionWorkspaceRestorePoints.status, ["pending", "active"])));
}
server/src/services/git-restore-point.ts
+7
View File
@@ -446,3 +446,10 @@ export async function openRestorePoint(input: {
}
return payload;
}
export async function restorePointHistoryDigest(input: string, signal?: AbortSignal): Promise<string> {
const cwd = await fs.realpath(input);
const reflog = await gitText(cwd, ["reflog", "show", "--format=%H", "HEAD"], { signal }).catch(() => "");
const original = await gitText(cwd, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], { signal }).catch(() => "");
return createHash("sha256").update(JSON.stringify([reflog.split("\n").filter((line) => SHA.test(line)), SHA.test(original) ? original : null])).digest("hex");
}
server/src/services/git-workspace-contents.ts
+15 -2
View File
@@ -11,6 +11,9 @@ import { localRepositoryPath } from "./local-repository-path.js";
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
import { materializeManagedProjectWorkspace } from "./managed-repository-checkout.js";
import { ensureSavedGitCommit, gitCacheCommand, GitCacheRestorationFailure } from "./git-workspace-restoration.js";
import { applyGitRestorePointToWorktree, ensureGitRestorePointCommit, type GitRestorePointRef } from "./git-restore-point-storage.js";
import { getStorageService } from "../storage/index.js";
import type { StorageService } from "../storage/types.js";
export const CLAUDE_SETTINGS_PATH = ".claude/settings.local.json";
const SETTINGS_LIMIT = 64 * 1024;
@@ -20,6 +23,7 @@ type Settings = { material: StoredSecretVersionMaterial } | null;
export type ManagedCheckoutSource = { id: string; repoUrl: string; repoRef: string | null; cwd: string | null };
export type ManagedCheckoutContents = {
name: string; source: ManagedCheckoutSource; commit: string; branchName: string | null;
restorePoint?: GitRestorePointRef;
};
export type GitWorkspaceContents = {
schema: "paperclip.git-workspace-contents.v1";
@@ -141,8 +145,9 @@ export async function captureGitWorkspaceContents(cwd: string, identity: Identit
export async function assertGitWorkspaceContentsUnchanged(cwd: string, identity: Identity, contents: GitWorkspaceContents | undefined, checkouts: ManagedCheckoutContents[]) {
validateGitWorkspaceContents(contents);
const recorded = contents?.checkouts.map(({ settings: _settings, ...checkout }) => checkout) ?? [];
if (!isDeepStrictEqual(recorded, checkouts.sort((a, b) => a.name.localeCompare(b.name)))) throw new Error("managed_checkouts_changed");
const recorded = contents?.checkouts.map(({ settings: _settings, restorePoint: _restorePoint, ...checkout }) => checkout) ?? [];
const actual = checkouts.map(({ restorePoint: _restorePoint, ...checkout }) => checkout).sort((a, b) => a.name.localeCompare(b.name));
if (!isDeepStrictEqual(recorded, actual)) throw new Error("managed_checkouts_changed");
const assertSettings = async (root: string, saved: Settings, relative: string) => {
if (!isDeepStrictEqual(await readSettings(root), await savedSettings(saved, identity, relative))) throw new Error("local_settings_changed");
};
@@ -188,9 +193,11 @@ async function restoreSettings(cwd: string, settings: Settings, identity: Identi
export async function restoreGitWorkspaceContents(input: Identity & { cwd: string; contents?: GitWorkspaceContents;
prepareLocalSource?: (sourceCwd: string) => Promise<() => Promise<void>>;
storage?: StorageService;
resolveGitAuth: GitRemoteAuthProvider; authorizeSource: (source: ManagedCheckoutSource) => Promise<void> }) {
if (!input.contents) return;
validateGitWorkspaceContents(input.contents);
const workspace = { companyId: input.companyId, id: input.workspaceId };
try {
for (const checkout of input.contents.checkouts) {
await input.authorizeSource(checkout.source);
@@ -208,15 +215,21 @@ export async function restoreGitWorkspaceContents(input: Identity & { cwd: strin
|| (await gitCacheCommand(destination, ["config", "--local", "--null", "--get-all", "remote.origin.url"])) !== `${checkout.source.repoUrl}\0`) throw new GitCacheRestorationFailure();
const actual = await readManagedCheckoutContents(destination, checkout.name, checkout.source);
if (actual.commit !== checkout.commit || actual.branchName !== checkout.branchName) throw new GitCacheRestorationFailure();
if (checkout.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage ?? getStorageService(), workspace,
ref: checkout.restorePoint, repositoryCwd: destination, worktreePath: destination });
await restoreSettings(destination, checkout.settings, input, relative);
continue;
}
const result = await materializeManagedProjectWorkspace(destination, { repoUrl: checkout.source.repoUrl,
resolveGitAuth: input.resolveGitAuth, beforePublish: async temporary => {
if (checkout.restorePoint) await ensureGitRestorePointCommit({ storage: input.storage ?? getStorageService(), workspace,
ref: checkout.restorePoint, repositoryCwd: temporary });
await ensureSavedGitCommit(temporary, checkout.source.repoUrl, checkout.commit, input.resolveGitAuth);
if (checkout.branchName) await gitCacheCommand(temporary, ["check-ref-format", "--branch", checkout.branchName]);
await gitCacheCommand(temporary, checkout.branchName
? ["checkout", "-B", checkout.branchName, checkout.commit] : ["checkout", "--detach", checkout.commit]);
if (checkout.restorePoint) await applyGitRestorePointToWorktree({ storage: input.storage ?? getStorageService(), workspace,
ref: checkout.restorePoint, repositoryCwd: temporary, worktreePath: temporary });
await gitCacheCommand(temporary, ["reflog", "expire", "--expire=now", "--all"]);
await restoreSettings(temporary, checkout.settings, input, relative);
await input.authorizeSource(checkout.source);
server/src/services/git-workspace-expiration.ts
+71 -27
View File
@@ -9,7 +9,7 @@ import { executionWorkspaces, issues, heartbeatRuns, nativeRunFinalizations, env
import { createGitRemoteAuthProvider } from "./git-credentials.js";
import type { StorageService } from "../storage/types.js";
import { currentRestorePointSnapshot, RestorePointSkip, type RestorePointLimits } from "./git-restore-point.js";
import { abandonGitRestorePoint, abandonGitRestorePointById, activateGitRestorePoint, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js";
import { abandonGitRestorePoint, abandonGitRestorePointsByKey, activateGitRestorePoints, restorePointHistoryUnchanged, restorePointSnapshotMatches, storeGitRestorePoint } from "./git-restore-point-storage.js";
import { withIsolatedRepositorySourceLock, verifiedRecordedIsolatedRemoteSource } from "./isolated-remote-workspace-source.js";
import { managedProjectCheckoutName } from "./managed-project-checkout-name.js";
import type { GitRemoteAuthProvider } from "./git-credentials.js";
@@ -54,17 +54,22 @@ export type GitRestorePointExpirationOptions = {
};
const RESTORE_POINT_RELAXABLE = new Set(["dirty_or_untracked_files", "unique_ignored_files", "unique_original_head",
"unique_reflog_commits", "upstream_commit_unverified"]);
"unique_reflog_commits", "upstream_commit_unverified", "unpushed_managed_checkout"]);
const CHECKOUT_RESTORE_POINT_RELAXABLE = new Set(["dirty_or_untracked_files", "unpushed_managed_checkout", "unique_ignored_files",
"unique_original_head", "unique_reflog_commits"]);
async function restorePointBoundary(recipe: GitCacheRestoration, headOnRemote: boolean, auth: GitRemoteAuthProvider, signal?: AbortSignal) {
if (headOnRemote) return [recipe.commit];
const upstream = await gitCacheCommand(recipe.worktreePath, ["rev-parse", "--verify", "-q", "refs/remotes/origin/HEAD"], signal)
type CheckoutRestorePointNeed = { headOnRemote: boolean; path: string; repoUrl: string };
async function restorePointBoundary(cwd: string, repoUrl: string, headOnRemote: boolean, auth: GitRemoteAuthProvider, signal?: AbortSignal) {
const head = (await gitCacheCommand(cwd, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
if (headOnRemote) return [head];
const upstream = await gitCacheCommand(cwd, ["rev-parse", "--verify", "-q", "refs/remotes/origin/HEAD"], signal)
.catch(() => "");
const tip = upstream.trim();
if (!tip) return [];
const base = (await gitCacheCommand(recipe.worktreePath, ["merge-base", "HEAD", tip], signal).catch(() => "")).trim();
const base = (await gitCacheCommand(cwd, ["merge-base", "HEAD", tip], signal).catch(() => "")).trim();
if (!/^[a-f0-9]{40}$/.test(base)) return [];
try { await verifyRemoteGitCommit(recipe.repoUrl, base, auth, signal); return [base]; }
try { await verifyRemoteGitCommit(repoUrl, base, auth, signal); return [base]; }
catch (error) { if (signal?.aborted) throw error; return []; }
}
@@ -163,7 +168,8 @@ async function inspectBase(db: Db, workspace: Workspace, signal?: AbortSignal, a
return allocatedBytes(recipe.sourcePath, () => { signal?.throwIfAborted(); return true; });
}
function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuthProvider, checkouts: Map<string, ManagedCheckoutContents>): ManagedCheckoutProver {
function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuthProvider, checkouts: Map<string, ManagedCheckoutContents>,
restorePoints?: Map<string, CheckoutRestorePointNeed>): ManagedCheckoutProver {
let rows: Promise<Array<{ id: string; repoUrl: string | null; repoRef: string | null; cwd: string | null }>> | undefined;
return async (checkoutPath, name, signal) => {
if (!workspace.projectId) throw new Error("unknown_managed_checkout");
@@ -173,7 +179,12 @@ function managedCheckoutProver(db: Db, workspace: Workspace, auth: GitRemoteAuth
const source = selectRestoredProjectRepositories(await rows, readGitCacheRestoration(workspace)?.contents).find(row => row.repoUrl
&& managedProjectCheckoutName({ id: row.id, repoUrl: row.repoUrl, repoRef: row.repoRef, cwd: row.cwd }) === name);
if (!source?.repoUrl) throw new Error("unknown_managed_checkout");
await assertReproducibleManagedCheckout(checkoutPath, { repoUrl: source.repoUrl, resolveGitAuth: auth, signal });
try { await assertReproducibleManagedCheckout(checkoutPath, { repoUrl: source.repoUrl, resolveGitAuth: auth, signal }); }
catch (error) {
if (!restorePoints || !CHECKOUT_RESTORE_POINT_RELAXABLE.has(skipReason(error))) throw error;
const { headOnRemote } = await assertReproducibleManagedCheckout(checkoutPath, { repoUrl: source.repoUrl, resolveGitAuth: auth, signal, restorePoint: true });
restorePoints.set(name, { headOnRemote, path: checkoutPath, repoUrl: source.repoUrl });
}
checkouts.set(name, await readManagedCheckoutContents(checkoutPath, name, { ...source, repoUrl: source.repoUrl }, signal));
};
}
@@ -185,7 +196,7 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
}): Promise<GitExpirationResult> {
const result: GitExpirationResult = { candidate: false, expired: false, reclaimedBytes: 0,
candidateBytes: 0, baseExpired: false, skipReason: null, restorePoint: false };
let pendingRestorePoint: string | null = null;
let pendingRestorePoints: string[] = [];
if (!snapshot.sourceIssueId) return { ...result, skipReason: "missing_source_issue" };
try {
return await withIsolatedRepositorySourceLock(db, snapshot.companyId, snapshot.sourceIssueId, () =>
@@ -203,6 +214,7 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
const checkouts = new Map<string, ManagedCheckoutContents>();
const identity = { companyId: workspace.companyId, workspaceId: workspace.id };
const proveManagedCheckout = managedCheckoutProver(db, workspace, auth, checkouts);
const checkoutRestorePoints = new Map<string, CheckoutRestorePointNeed>();
let headOnRemote = true;
let needsRestorePoint = false;
let recipe: GitCacheRestoration;
@@ -214,7 +226,9 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
if (options.now.getTime() < options.restorePoint.notBefore.getTime()) throw new Error("restore_point_not_due");
if (!options.restorePoint.keyDurable) throw new Error("restore_point_key_not_durable");
checkouts.clear();
const inspected = await inspectGitCacheRecipeForRestorePoint(workspace, auth, options.signal, { proveManagedCheckout });
checkoutRestorePoints.clear();
const inspected = await inspectGitCacheRecipeForRestorePoint(workspace, auth, options.signal, {
proveManagedCheckout: managedCheckoutProver(db, workspace, auth, checkouts, checkoutRestorePoints) });
recipe = inspected.recipe;
headOnRemote = inspected.headOnRemote;
needsRestorePoint = true;
@@ -242,17 +256,28 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
if (await fs.realpath(worktreeAdmin) !== canonicalAdmin || await fs.realpath(rawAdmin) !== canonicalAdmin) {
throw new Error("unproven_worktree_admin");
}
let activateLedger: string | null = null;
if (needsRestorePoint) {
const stored = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage,
workspace: { companyId: workspace.companyId, id: workspace.id, sourceIssueId: workspace.sourceIssueId },
const owner = { companyId: workspace.companyId, id: workspace.id, sourceIssueId: workspace.sourceIssueId };
const stored = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, workspace: owner,
cwd: recipe.worktreePath, generation, headOnRemote, limits: options.restorePoint!.limits, signal: options.signal,
upstreamCommits: await restorePointBoundary(recipe, headOnRemote, auth, options.signal) });
upstreamCommits: await restorePointBoundary(recipe.worktreePath, recipe.repoUrl, headOnRemote, auth, options.signal) });
if (stored) {
pendingRestorePoint = stored.ledgerId;
activateLedger = stored.ledgerId;
recipe = { ...recipe, schema: GIT_CACHE_RESTORATION_SCHEMA_V2, restorePoint: stored.ref };
pendingRestorePoints.push(stored.ledgerId);
recipe = { ...recipe, restorePoint: stored.ref };
} else if (!headOnRemote) throw new Error("upstream_commit_unverified");
for (const [name, need] of [...checkoutRestorePoints.entries()].sort(([a], [b]) => a.localeCompare(b))) {
const storedCheckout = await storeGitRestorePoint({ db, storage: options.restorePoint!.storage, workspace: owner,
cwd: need.path, checkout: name, generation, headOnRemote: need.headOnRemote, limits: options.restorePoint!.limits,
signal: options.signal, upstreamCommits: await restorePointBoundary(need.path, need.repoUrl, need.headOnRemote, auth, options.signal) });
if (!storedCheckout) {
if (!need.headOnRemote) throw new Error("unpushed_managed_checkout");
continue;
}
pendingRestorePoints.push(storedCheckout.ledgerId);
recipe = { ...recipe, contents: { ...recipe.contents!, checkouts: recipe.contents!.checkouts.map(checkout =>
checkout.name === name ? { ...checkout, restorePoint: storedCheckout.ref } : checkout) } };
}
if (pendingRestorePoints.length > 0) recipe = { ...recipe, schema: GIT_CACHE_RESTORATION_SCHEMA_V2 };
}
state = { schema: "paperclip.git-cache-expiration.v1", generation, workspaceRemoved: false,
worktreeTrash: path.join(path.dirname(recipe.worktreePath), `.expired-git-worktree-${workspace.id}-${generation}`),
@@ -266,10 +291,10 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
};
[workspace] = await db.transaction(async (archiveTx) => {
const rows = await archiveTx.update(executionWorkspaces).set(archive).where(eq(executionWorkspaces.id, workspace!.id)).returning();
if (activateLedger) await activateGitRestorePoint(archiveTx, { ledgerId: activateLedger, workspaceId: workspace!.id, at: options.now });
await activateGitRestorePoints(archiveTx, { ledgerIds: pendingRestorePoints, workspaceId: workspace!.id, at: options.now });
return rows;
});
pendingRestorePoint = null;
pendingRestorePoints = [];
}
if (!workspace) throw new Error("workspace_changed");
const persist = async () => {
@@ -289,15 +314,30 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
try {
try {
checkouts.clear();
await assertReproducibleGitFiles(recipe.worktreePath, options.signal, { proveManagedCheckout, preserveSettings: true,
const savedCheckoutRestorePoints = new Map((recipe.contents?.checkouts ?? []).filter(checkout => checkout.restorePoint)
.map(checkout => [checkout.name, checkout.restorePoint!]));
const relaxedChecks = new Map<string, CheckoutRestorePointNeed>();
const relaxedProver = managedCheckoutProver(db, workspace, auth, checkouts, relaxedChecks);
await assertReproducibleGitFiles(recipe.worktreePath, options.signal, { preserveSettings: true,
proveManagedCheckout: async (checkoutPath, name, signal) => {
if (!savedCheckoutRestorePoints.has(name)) return proveManagedCheckout(checkoutPath, name, signal);
await relaxedProver(checkoutPath, name, signal);
if (!await restorePointHistoryUnchanged(savedCheckoutRestorePoints.get(name)!, checkoutPath, signal)) {
throw new Error("restore_point_contents_changed");
}
},
restorePoint: Boolean(recipe.restorePoint),
verifyFetchedCommit: (commit, signal) => verifyRemoteGitCommit(recipe.repoUrl, commit, auth, signal) });
if (recipe.restorePoint && !restorePointSnapshotMatches(recipe.restorePoint,
await currentRestorePointSnapshot(recipe.worktreePath, options.signal))) throw new Error("restore_point_contents_changed");
if (recipe.restorePoint && (!restorePointSnapshotMatches(recipe.restorePoint,
await currentRestorePointSnapshot(recipe.worktreePath, options.signal))
|| !await restorePointHistoryUnchanged(recipe.restorePoint, recipe.worktreePath, options.signal))) throw new Error("restore_point_contents_changed");
for (const checkout of checkouts.values()) {
const cwd = path.join(recipe.worktreePath, ".paperclip-repositories", checkout.name);
locks.push(await acquireGitExpirationLocks(lockLocations.filter(location => location.checkout === checkout.name), state.lockToken));
if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], options.signal)).length) throw new Error("managed_checkouts_changed");
const savedCheckout = savedCheckoutRestorePoints.get(checkout.name);
if (savedCheckout) {
if (!restorePointSnapshotMatches(savedCheckout, await currentRestorePointSnapshot(cwd, options.signal))) throw new Error("restore_point_contents_changed");
} else if ((await gitCacheCommand(cwd, ["status", "--porcelain=v1", "--untracked-files=all", "-z"], options.signal)).length) throw new Error("managed_checkouts_changed");
checkouts.set(checkout.name, await readManagedCheckoutContents(cwd, checkout.name, checkout.source, options.signal));
}
await assertGitWorkspaceContentsUnchanged(recipe.worktreePath, identity, recipe.contents, [...checkouts.values()]);
@@ -306,12 +346,16 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
} catch (error) {
for (const lock of [...locks].reverse()) await lock.release();
const { restorePoint: _abandoned, ...reproducible } = recipe;
const strippedContents = reproducible.contents && { ...reproducible.contents,
checkouts: reproducible.contents.checkouts.map(({ restorePoint: _checkoutRestorePoint, ...checkout }) => checkout) };
const metadata: Record<string, unknown> = { ...workspace.metadata, lifecycleGeneration: state.generation + 1,
[GIT_CACHE_RESTORATION_KEY]: { ...reproducible, schema: "paperclip.git-cache-restoration.v1", restoredAt: new Date().toISOString() } };
[GIT_CACHE_RESTORATION_KEY]: { ...reproducible, ...(strippedContents ? { contents: strippedContents } : {}),
schema: "paperclip.git-cache-restoration.v1", restoredAt: new Date().toISOString() } };
delete metadata[EXPIRATION_KEY];
await db.update(executionWorkspaces).set({ status: "active", closedAt: null, cleanupEligibleAt: null,
cleanupReason: null, metadata, updatedAt: new Date() }).where(eq(executionWorkspaces.id, workspace.id));
if (recipe.restorePoint) await abandonGitRestorePointById(db, workspace.id, recipe.restorePoint.objectKey);
await abandonGitRestorePointsByKey(db, workspace.id, [recipe.restorePoint?.objectKey,
...(recipe.contents?.checkouts ?? []).map(checkout => checkout.restorePoint?.objectKey)].filter((key): key is string => Boolean(key)));
throw error;
}
await fs.rename(recipe.worktreePath, state.worktreeTrash);
@@ -350,7 +394,7 @@ export async function expireReconstructableGitWorkspace(db: Db, snapshot: Worksp
return result;
})), true);
} catch (error) {
if (pendingRestorePoint) await abandonGitRestorePoint(db, pendingRestorePoint).catch(() => undefined);
for (const ledgerId of pendingRestorePoints) await abandonGitRestorePoint(db, ledgerId).catch(() => undefined);
options.signal?.throwIfAborted();
result.skipReason = error instanceof RestorePointSkip ? error.reason : skipReason(error);
return result;
server/src/services/git-workspace-restoration.ts
+90 -21
View File
@@ -101,16 +101,27 @@ export function readGitCacheRestoration(workspace: Workspace): GitCacheRestorati
throw new GitCacheRestorationFailure();
}
validateGitWorkspaceContents(recipe.contents);
if (recipe.restorePoint) {
try { validateGitRestorePointRef(recipe.restorePoint, workspace); } catch { throw new GitCacheRestorationFailure(); }
}
validateRestorePoints(recipe, workspace);
return recipe;
}
function hasRestorePoints(recipe: GitCacheRestoration) {
return recipe.restorePoint !== undefined || (recipe.contents?.checkouts ?? []).some(checkout => checkout?.restorePoint !== undefined);
}
function restorationSchemaMatches(recipe: GitCacheRestoration) {
return recipe.schema === "paperclip.git-cache-restoration.v1"
? recipe.restorePoint === undefined
: recipe.schema === GIT_CACHE_RESTORATION_SCHEMA_V2 && recipe.restorePoint !== undefined;
? !hasRestorePoints(recipe)
: recipe.schema === GIT_CACHE_RESTORATION_SCHEMA_V2 && hasRestorePoints(recipe);
}
function validateRestorePoints(recipe: GitCacheRestoration, workspace: { companyId: string; id: string }) {
try {
if (recipe.restorePoint) validateGitRestorePointRef(recipe.restorePoint, workspace);
for (const checkout of recipe.contents?.checkouts ?? []) {
if (checkout.restorePoint) validateGitRestorePointRef(checkout.restorePoint, workspace, checkout.name);
}
} catch { throw new GitCacheRestorationFailure(); }
}
export async function gitCacheCommand(cwd: string, args: string[], signal?: AbortSignal) {
@@ -154,9 +165,10 @@ async function stockGitFiles() {
return templateFiles;
}
export async function assertReproducibleBaseGitStorage(cwd: string, signal?: AbortSignal) {
export async function assertReproducibleBaseGitStorage(cwd: string, signal?: AbortSignal, options: { savedCommitMessage?: boolean } = {}) {
const root = path.join(cwd, ".git");
const allowed = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "config", "description", "hooks", "info", "refs", "objects", "logs", "packed-refs", "worktrees", "branches"]);
const allowed = new Set(["HEAD", "ORIG_HEAD", "FETCH_HEAD", "index", "config", "description", "hooks", "info", "refs", "objects", "logs", "packed-refs", "worktrees", "branches",
...(options.savedCommitMessage ? ["COMMIT_EDITMSG"] : [])]);
if ((await fs.readdir(root)).some(name => !allowed.has(name))) throw new Error("unknown_base_git_files");
const branches = path.join(root, "branches");
const branchesStat = await fs.lstat(branches).catch((error: NodeJS.ErrnoException) => {
@@ -201,9 +213,43 @@ const PAPERCLIP_CLAUDE_SETTINGS_FILE = ".claude/settings.local.json";
export type ManagedCheckoutProver = (checkoutPath: string, name: string, signal?: AbortSignal) => Promise<void>;
const COMMIT_MESSAGE_LIMIT = 64 * 1024;
async function readCommitMessage(gitDir: string): Promise<string | null> {
const target = path.join(gitDir, "COMMIT_EDITMSG");
const entry = await fs.lstat(target).catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null;
throw error;
});
if (!entry) return null;
if (!entry.isFile() || entry.size > COMMIT_MESSAGE_LIMIT) throw new Error("unsupported_commit_message");
const handle = await fs.open(target, constants.O_RDONLY | constants.O_NOFOLLOW | constants.O_NONBLOCK);
try {
const stat = await handle.stat();
if (!stat.isFile() || stat.size > COMMIT_MESSAGE_LIMIT) throw new Error("unsupported_commit_message");
const buffer = Buffer.alloc(COMMIT_MESSAGE_LIMIT + 1);
let length = 0;
while (length < buffer.length) {
const { bytesRead } = await handle.read(buffer, length, buffer.length - length, length);
if (bytesRead === 0) break;
length += bytesRead;
}
if (length > COMMIT_MESSAGE_LIMIT) throw new Error("unsupported_commit_message");
return buffer.subarray(0, length).toString("utf8");
} finally { await handle.close(); }
}
async function assertCommitMessageSaved(cwd: string, gitDir: string, signal?: AbortSignal) {
const message = await readCommitMessage(gitDir);
if (message === null) return;
const saved = [(await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim(),
...(await gitCacheCommand(cwd, ["log", "-g", "--format=%B%x00", "HEAD"], signal).catch(() => "")).split("\0").map(entry => entry.trim())];
if (!saved.includes(message.trim())) throw new Error("uncommitted_git_message");
}
export async function assertReproducibleManagedCheckout(checkout: string, input: {
repoUrl: string; resolveGitAuth: GitRemoteAuthProvider; signal?: AbortSignal;
}) {
repoUrl: string; resolveGitAuth: GitRemoteAuthProvider; signal?: AbortSignal; restorePoint?: boolean;
}): Promise<{ headOnRemote: boolean }> {
const { signal } = input;
const gitDir = await fs.lstat(path.join(checkout, ".git")).catch(() => null);
if (!gitDir?.isDirectory()) throw new Error("unproven_managed_checkout");
@@ -213,10 +259,37 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
const worktrees = (await gitCacheCommand(checkout, ["worktree", "list", "--porcelain"], signal))
.split("\n").filter(line => line.startsWith("worktree "));
if (worktrees.length !== 1) throw new Error("managed_checkout_has_worktrees");
await assertReproducibleGitFiles(checkout, signal, { preserveSettings: true });
await assertReproducibleBaseGitStorage(checkout, signal);
await assertReproducibleGitFiles(checkout, signal, { preserveSettings: true, restorePoint: input.restorePoint });
await assertReproducibleBaseGitStorage(checkout, signal, { savedCommitMessage: true });
await assertCommitMessageSaved(checkout, path.join(checkout, ".git"), signal);
const refs = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(refname)"], signal)).trim().split("\n").filter(Boolean);
if (refs.some(ref => !/^refs\/(heads|tags|remotes\/origin)\//.test(ref))) throw new Error("unsupported_managed_checkout_refs");
if (input.restorePoint) {
const head = (await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).trim();
const shas = (output: string) => output.split("\n").map(line => line.trim()).filter(sha => shaPattern.test(sha));
const covered = new Set([head,
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--format=%H", "HEAD"], signal).catch(() => "")),
...shas(await gitCacheCommand(checkout, ["rev-parse", "--verify", "-q", "ORIG_HEAD^{commit}"], signal).catch(() => ""))]);
const recorded = new Set([
...shas(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)", "refs/heads", "refs/tags"], signal)),
...shas(await gitCacheCommand(checkout, ["reflog", "show", "--all", "--format=%H"], signal).catch(() => "")),
]);
const current = (await gitCacheCommand(checkout, ["symbolic-ref", "--quiet", "HEAD"], signal).catch(() => "")).trim();
const otherTips = (await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname) %(refname)", "refs/heads", "refs/tags"], signal))
.split("\n").map(line => line.trim().split(" ")).filter(([sha, ref]) => shaPattern.test(sha ?? "") && ref !== current).map(([sha]) => sha!);
if (otherTips.length && await hasCommitsMissingUpstream(checkout, otherTips, input.repoUrl, input.resolveGitAuth, signal, [head])) {
throw new Error("unpushed_managed_checkout_branches");
}
const uncovered = [...recorded].filter(sha => !covered.has(sha));
if (uncovered.length + covered.size > MANAGED_CHECKOUT_LOCAL_TIP_LIMIT) throw new Error("managed_checkout_local_tip_budget");
if (uncovered.length && await hasCommitsMissingUpstream(checkout, uncovered, input.repoUrl, input.resolveGitAuth, signal, [...covered])) {
throw new Error("unpushed_managed_checkout_branches");
}
const headOnRemote = !await hasCommitsMissingUpstream(checkout, [head], input.repoUrl, input.resolveGitAuth, signal);
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
return { headOnRemote };
}
const localTips = new Set([
...(await gitCacheCommand(checkout, ["rev-parse", "--verify", "HEAD^{commit}"], signal)).split("\n"),
...(await gitCacheCommand(checkout, ["for-each-ref", "--format=%(objectname)"], signal)).split("\n"),
@@ -228,12 +301,13 @@ export async function assertReproducibleManagedCheckout(checkout: string, input:
}
const unreachable = await gitCacheCommand(checkout, ["fsck", "--connectivity-only", "--unreachable", "--no-progress"], signal);
if (unreachable.trim()) throw new Error("unreachable_managed_checkout_objects");
return { headOnRemote: true };
}
const MANAGED_CHECKOUT_LOCAL_TIP_LIMIT = 5_000;
async function hasCommitsMissingUpstream(checkout: string, localTips: string[], repoUrl: string,
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal) {
resolveGitAuth: GitRemoteAuthProvider, signal?: AbortSignal, covered: string[] = []) {
const store = await fs.mkdtemp(path.join(os.tmpdir(), "paperclip-git-checkout-proof-"));
try {
const options = { timeout: 120_000, signal, maxBuffer: 16 * 1024 * 1024, cwd: os.tmpdir(),
@@ -249,7 +323,7 @@ async function hasCommitsMissingUpstream(checkout: string, localTips: string[],
if (signal?.aborted) throw error;
throw new Error("managed_checkout_upstream_unavailable");
}
const missing = await execFile("git", ["-C", store, "rev-list", "--max-count=1", ...localTips, "--not", "--glob=refs/upstream/*"], options);
const missing = await execFile("git", ["-C", store, "rev-list", "--max-count=1", ...localTips, "--not", "--glob=refs/upstream/*", ...covered], options);
return missing.stdout.trim() !== "";
} finally {
await fs.rm(store, { recursive: true, force: true });
@@ -356,9 +430,7 @@ export async function assertReproducibleGitFiles(cwd: string, signal?: AbortSign
}
}
await assertEmptyRefs(path.join(gitDir, "refs"));
const message = await fs.readFile(path.join(gitDir, "COMMIT_EDITMSG"), "utf8").catch((error: NodeJS.ErrnoException) => {
if (error.code === "ENOENT") return null; throw error;
});
const message = await readCommitMessage(gitDir);
if (message !== null && message.trim() !== (await gitCacheCommand(cwd, ["log", "-1", "--format=%B"], signal)).trim()) {
const saved = options.restorePoint
? (await gitCacheCommand(cwd, ["log", "-g", "--format=%B%x00", "HEAD"], signal).catch(() => "")).split("\0").map(entry => entry.trim())
@@ -563,11 +635,8 @@ export function savedGitWorkspaceRestoration(input: { metadata?: Record<string,
|| (value.restoredAt !== undefined && !Number.isFinite(Date.parse(value.restoredAt)))
|| (value.contentsRestoredAt != null && !Number.isFinite(Date.parse(value.contentsRestoredAt)))) throw new GitCacheRestorationFailure();
validateGitWorkspaceContents(value.contents);
let restorePoint: GitRestorePointRef | null = null;
if (value.restorePoint) {
try { restorePoint = validateGitRestorePointRef(value.restorePoint, { companyId: input.companyId, id: input.workspaceId ?? "" }); }
catch { throw new GitCacheRestorationFailure(); }
}
validateRestorePoints(value, { companyId: input.companyId, id: input.workspaceId ?? "" });
const restorePoint: GitRestorePointRef | null = value.restorePoint ?? null;
return { commit: value.commit, restored: value.restoredAt !== undefined, contents: value.contents, restorePoint,
contentsRestored: value.contentsRestoredAt === undefined ? value.restoredAt !== undefined : value.contentsRestoredAt !== null };
}