Commit Graph
4 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 c38a471042 perf(server): download stale agent files before the agent row lock (#21694)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Durable agent files live in object storage. Each server keeps a
local copy (the projection), marked with the head generation it matches
> - When the projection is stale, the next locked read or write restores
it: it downloads every file from object storage. This happens for the
first run of each agent on a new server (every deploy on hosted
ephemeral servers) and after another server wrote
> - The download ran inside the transaction that holds the `agents` row
lock (`FOR UPDATE`), on the main pool for the run-start read
(`readCommittedForRuntime`) and on the 2-connection file-storage pool
for the store. A download can take up to 30 s
> - After a deploy at 1,000 agents, this is one long row lock and one
pinned connection per agent, all at once, and every writer of that agent
waits behind it
> - This pull request downloads the files outside the transaction and
takes the lock again only to swap them in
> - The benefit is that the row lock is held for milliseconds instead of
the download time (0 ms observed instead of 321 ms with a 300 ms
download in the test), and writers do not wait for a download

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The restore of a stale local copy of an agent's durable files (the first
run of each agent on a new server, or after another server wrote).

**Subsystem affected**
Server: durable agent files (`agent-file-durability.ts`,
`agent-file-store.ts`, `agent-instruction-revisions.ts`).

**Current behavior**
The restore downloads every file from object storage inside the
transaction that holds the `agents` row lock (`FOR UPDATE`) and a pooled
connection. A download can take up to 30 s, and every writer of that
agent waits behind it.

**Proposed behavior**
Under the lock, detect that a download is needed and roll back. Download
the head's files, verified by digest, into a private staging directory
with no lock. Run the transaction again, and swap the staged files in
only when the head generation is unchanged. After two changed heads,
restore under the lock as before.

**Reason and benefit**
After a deploy at 1,000 agents, every agent's first run restores at
once. The row lock and connection are now held for milliseconds instead
of the download time, and writers do not wait for a download.

**Additional context**
Refs the 1,000-concurrent-agents plan
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`). The save path
had the same problem for uploads and was fixed earlier with uploads
before the lock (`uploadAgentFilesBeforeCommit`); this is the restore
side. Reading the head before every transaction was considered; it adds
a query to every call, while the deferral costs nothing when the
projection is current.

## What Changed

- `agent-file-durability.ts`: the download loop moves into
`downloadAgentFiles`, unchanged.
`withAgentFilesRestoredBeforeLock(work)` runs `work` in a scope. Inside
the scope, the first restore of an attempt that needs a download throws
a private deferral error, which rolls back the transaction. The wrapper
stages the download in `.agent-files-<uuid>` next to the root and runs
`work` again. `restoreDurableAgentFiles` uses a staged directory only
when its generation equals the head it reads under the lock. Staged
directories are removed when the scope ends.
- Only the first restore of each attempt can defer, so nested restores
of other agents keep their old behavior. Outside the scope, the behavior
is unchanged.
- `agent-file-store.ts` (`locked`, so every store read, write, apply and
working-copy preparation) and `agent-instruction-revisions.ts`
(`readCommittedForRuntime`, `readCurrent`, `materializeCurrent`) run
their transactions in the scope.

## Verification

- New test `server/src/__tests__/agent-file-restore-locking.test.ts` on
embedded PostgreSQL, with an object-storage fixture:
- Hold time, with a 300 ms download, through `readCommittedForRuntime`
(the run-start read). Before: the download ran with the agent row
locked, and the row was locked 321 ms of 322 ms. After: the download ran
with the row unlocked, and a sampler saw the row locked for 0 ms.
- Concurrency: a board write of the same file runs while the reader's
download is blocked. Before: the writer waited for the reader's download
(5 s timeout). After: the writer commits a new head during the download.
The reader then discards its stale staged copy and returns the writer's
content. The projection is marked at the writer's generation, and no
staging directory is left.
- Failure: when object storage fails, the read fails once with the
storage error, after one attempt made without the row lock, and leaves
no staging directory.
- Progress: when the head generation changes during each of two
downloads, the third attempt restores under the lock as before (3
downloads), and the projection matches the final head.
- `npx vitest run` for agent-file-checkpoint-locking,
agent-file-checkpoints, agent-file-durability,
agent-file-restore-locking, agent-instruction-revisions,
agent-instruction-working-copies, agent-instructions-service,
agent-directory-working-copies and agent-instructions-routes: 287
passed, 1 skipped.
- `npx tsc --noEmit -p server/tsconfig.json`: passes.

## Risks

- A deferred attempt runs the transaction body again. The deferral
happens at the first restore, before the body writes files, and the
transaction rollback undoes database work, so the second attempt starts
from the same state.
- A staged download uses disk next to the agent root until the scope
ends, the same as the temporary directory of the old download.
- If the head changes during every download, the third attempt downloads
under the lock as before.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:14:47 -06:00
DottaandClaude Opus 5.5 a48eaaef57 perf(server): skip the agent row lock for unchanged agent-file checkpoints and upload before the lock (#21457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - An agent keeps its own files (AGENT_HOME). A warm native session
saves these files at the end of each turn (an agent-file checkpoint)
> - Each checkpoint took the agent row lock (`agents ... FOR UPDATE`) on
the small file-storage pool, also when the run changed no file
> - A checkpoint that changed files also uploaded the new bytes to
object storage while it held that lock. Instruction edits and other runs
of the same agent waited for the upload
> - With many agents, these long transactions use the 2-connection
file-storage pool and slow every agent
> - This pull request removes the lock from an unchanged checkpoint and
moves the uploads of a changed checkpoint before the lock
> - The benefit is that an unchanged checkpoint takes no row lock, and
the row lock is no longer held across object-storage I/O

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm agent-file checkpoint: `checkpoint()` in
`server/src/services/agent-directory-working-copies.ts`,
`agentFileStore.locked` and `apply` in `agent-file-store.ts`, and
`persistDurableAgentFiles` in `agent-file-durability.ts`.

**Current behavior**
- When the run changed no file, the checkpoint still ran
`store.locked(..., write = true)`. That transaction locks the agent row
(`FOR UPDATE` and `FOR NO KEY UPDATE`), restores and invalidates the
local projection, walks the canonical tree, and compares the durable
head. Its only result is the authorization check.
- When the run changed files, `persistDurableAgentFiles` uploaded each
new file to object storage inside the transaction that holds the agent
row lock.

**Proposed behavior**
- An unchanged checkpoint (`saveUnchanged`) first reads, without a lock,
whether the locked file-tree step would change anything
(`agentFileProjectionCurrent`): the durable head exists and the local
projection is marked at its generation, or durable files are off, no
head exists and the local adoption is recorded. If so, it runs only the
write authorization (`authorizeSave`): the agent interaction policy and
`authorizeInstructionCommit`, in a short transaction without row locks
that always rolls back, so it writes nothing. If not, it runs the locked
save as on `origin/master`, which adopts, restores or heals the durable
head. A de-authorized agent still gets the same error (the existing test
"rechecks authorization even for an unchanged warm checkpoint" still
passes).
- A changed checkpoint runs the same read-only authorization (it
consumes no accepted change confirmation), then uploads the run's
changed files before it takes the lock (`uploadAgentFilesBeforeCommit`).
A run that lost its save permission uploads nothing. The locked commit
runs the authorization again, and only that commit consumes a
confirmation. The locked commit uses an uploaded object only for a file
with the same SHA-256 and size. It uploads any other file itself, as
before.

**Reason and benefit**
The target is 1,000 agents in one controller process. Most warm turns
change no agent file. These turns now take no row lock and send fewer
statements. A changed turn holds the row lock for less time, because
object-storage latency is outside the lock.

## What Changed

- `agent-file-store.ts`: add `authorizeSave(companyId, agentId, actor)`
and `saveUnchanged(companyId, agentId, actor)`. `apply` calls it for a
checkpoint, then uploads the run's changed files before `locked`, and
passes the uploaded objects to `persistDurableAgentFiles`. `locked`
takes an optional map of uploaded objects.
- `agent-file-durability.ts`: move the verified upload stream into
`uploadAgentFile`. Add `uploadAgentFilesBeforeCommit`. It reads the
durable head without a lock, skips files the head already has at the
same path and digest, and uploads the others. If an upload fails, it
stops and the locked commit uploads the rest, so errors are the same as
before. `persistDurableAgentFiles` uses an uploaded object when the
canonical file has the same digest and size.
- `agent-directory-working-copies.ts`: an unchanged checkpoint calls
`saveUnchanged` instead of `store.locked(..., true)`.
- `agent-file-durability.ts`: add `agentFileProjectionCurrent` (one head
read and two file checks, no lock).
- The commit is still inside the agent row lock. The head cannot change
between the read and the write of the commit, so it needs no separate
compare-and-set. The upload before the lock does not need one either: an
object is immutable and verified by digest, and the commit records it
only for a file with that digest.

## Verification

- New `server/src/__tests__/agent-file-checkpoint-locking.test.ts`
(embedded PostgreSQL, object-storage fixture). All tests fail on the
code before the change and pass with it:
- An unchanged checkpoint completes while another transaction holds the
agent row lock. It sends no `FOR UPDATE`, `FOR NO KEY UPDATE` or `FOR
SHARE` statement and only one unlocked read of `agent_file_roots`.
- Durable files on, the agent has local files and no head: an unchanged
checkpoint creates the head, as on `origin/master` (added after review;
failed without the fallback).
- The local projection is not marked at the head: an unchanged
checkpoint takes the locked path and marks it again (added after review;
failed without the fallback).
- A changed checkpoint uploads its file while the agent row is not
locked (the fixture checks the lock with `FOR UPDATE NOWAIT` from
another connection during each upload). The head then records that
object.
- A run loses its save permission before a changed checkpoint. The
checkpoint is refused and uploads nothing (added after review; I saw it
fail without the authorization before the upload).
- An agent with `agents:suggest-changes` and one accepted change
confirmation saves a changed checkpoint. An unchanged checkpoint first
leaves the confirmation unconsumed. The changed save succeeds once and
consumes the confirmation once; the next change is refused before it
uploads (added after review; I saw it fail when the preview committed).
- Two warm runs of the same agent save at the same time. Both files are
in the canonical tree and in the head, and every head object has the
correct digest.
- An instruction edit commits while a checkpoint waits in its upload,
without a wait for the checkpoint. Then the checkpoint commits. Both
changes are in the head.
- A save stops between upload and commit (the instruction entry changes
while the bytes upload, so the commit refuses). The head does not change
and stays restorable. The next save commits the same file. The unused
object is never authoritative; the existing object collector removes it
after its grace period.
- Existing `agent-directory-working-copies.test.ts`,
`agent-file-durability.test.ts` and `agent-file-checkpoints.test.ts`
pass.
- Measurement (opt-in, `PAPERCLIP_AGENT_FILE_CHECKPOINT_BENCHMARK=1`,
test "measures checkpoint statements and agent row lock time"): one warm
copy with 51 files; 10 unchanged checkpoints; then 5 checkpoints that
each change one file, with 250 ms object-store latency. Wall times are
noisy (load average about 40). Statement counts are exact.

| | Before (`origin/master`) | After (final head) |
| --- | --- | --- |
| Unchanged checkpoint: statements | 26 | 25 (one unlocked head read,
and the read-only authorization) |
| Unchanged checkpoint: row-lock statements | 2 | 0 |
| Unchanged checkpoint: median time | 214 ms | 26–126 ms |
| Changed checkpoint: statements | 29 | 47 (the read-only authorization
before the upload and one head read) |
| Changed checkpoint: median time the agent row is locked | 924 ms |
51–227 ms |
| Changed checkpoint: uploads while the agent row is locked | 1 | 0 |

The "after" ranges come from two runs at different machine load (load
average between about 18 and 40).

## Risks

- Master side effects of the locked unchanged save, and what this change
does with each:
- Kept: the write authorization (`assertAgentInteraction`,
`authorizeInstructionCommit`), so a run that lost its permission still
gets the same error.
- Kept through the fallback: one-time adoption (no head yet), restore of
a projection that is not marked at the head, re-mark after a failed
persist (a write path removes the marker before it changes files), and
heal of a head that lags the root after such a failure.
`agentFileProjectionCurrent` returns false in each of these cases, so
the locked save runs.
- Dropped when the projection is current: the re-mark at the same
generation (no change), and the compare of the local tree with the head.
With the marker at the head generation, every writer that changed the
tree went through the locked save and moved the marker, so the compare
finds no difference. An edit made outside Paperclip directly in the
canonical directory is not picked up by an unchanged checkpoint any
more; the next save that goes through the locked path still records it.
- Dropped when the projection is current: the storage-limit check of the
canonical tree in `persistDurableAgentFiles`. An over-quota agent whose
run changed nothing no longer gets an error from that check; any save
that changes files still enforces the limits.
- Dropped: the `FOR NO KEY UPDATE` locks that `assertAgentInteraction`
takes on the actor and target agents (`lock: "update"`). Those locks
hold permission changes off a write until it commits. The read-only
authorization writes nothing, so there is nothing to hold them off. The
locked commit of a changed save still takes them. An unchanged save has
no mention texts, so master locked no other agents for it.
- Effects of `authorizeSave` outside the rolled-back transaction: none.
I read `assertAgentInteraction` (`agentInteractionDenial`: reads only),
`authorizeInstructionCommit`, `resolveInstructionActor` (reads only),
`authorizationService.decide` (reads only; its memo maps live on the
bound actor and on the service instance, both created for this call and
dropped after it), and `changeConsentGateService.assertConsented` (one
`UPDATE` of the confirmation row, rolled back). None of them publishes
live events, queues notifications or wakes, writes activity through
another connection, makes external calls, or changes an in-process cache
that outlives the call.
- Behavior change: on `origin/master`, an unchanged checkpoint of a
consent-gated agent (`agents:suggest-changes`) consumed an accepted
change confirmation, because the locked transaction committed. Now the
fast path does not; only the commit of a changed save consumes it. When
the fast path cannot be used (adoption, restore or heal), the locked
save runs as on master and can still consume a confirmation.

- Not in this change: the manifest is still stored twice in the receipt
(`baseline` and `checkpointManifest`). Servers of the previous version
read `receipt.baseline` during a deploy overlap, and the next capture
needs the stamps in `checkpointManifest`, so neither can be removed
safely now.
- Not in this change: on the volume-free server, `serial()` still opens
a dedicated connection with an advisory-lock transaction for each
checkpoint. That lock serializes controllers and is not a row lock;
changing it is a separate design decision.
- A changed checkpoint can now leave an unused object in storage when
its commit is refused after the upload (for example, the instruction
entry changes during the upload) or the process stops before the commit.
This was already possible for a failed commit. Such an object is never
in the durable head, so no reader can reach it, and the existing
`collectAgentFileObjects` maintenance removes it after
`AGENT_FILE_OBJECT_GRACE_MS`. A permission that was removed before the
save stops the save before any upload.
- Old and new servers can run at the same time: the head format and the
receipt format do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 14:41:46 -06:00
DottaandPaperclip 1d348bbef0 fix(server): restore durable agent folders with bounded concurrency (#18531)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A replacement web server restores agent files from shared storage.
> - The old restore reads each object in sequence.
> - Modest network latency makes a 1,024-file folder exceed its
deadline.
> - This pull request uses at most eight concurrent restore streams.
> - Every file must pass its size and hash checks before the folder is
published.

## Linked Issues or Issue Description

Refs #18518 and #18526. Local deployment qualification found that a 64
MiB folder could not be restored after the old server filesystem was
removed.

## What Changed

- Restore at most eight files concurrently within the existing file
authority service.
- Cancel sibling transfers on failure. Await all workers before removing
the temporary folder.
- Keep the existing deadline, file-transfer database pool, authorization
and publication rules.
- Add regression coverage for bounded concurrency, corruption, settled
cancellation, retained previous bytes and retry.
- Document the behavior and update the deployment plan with local
qualification evidence.

## Verification

- Real PostgreSQL and S3-compatible storage: 1,024 files of 64 KiB each,
plus instructions, with 25 ms added to each object GET. Delete the
entire predecessor home and disable local adoption before reading
through the successor.
- Before: the read failed after 30.065 seconds. After: the same case
passed in 4.180 seconds and fetched 1,025 objects. All owned test
resources were cleaned.
- The durable-file suite passed eight tests against PostgreSQL. Server
TypeScript compilation passed. All 56 latest-head CI checks are complete
and green or intentionally skipped, including full typecheck, build,
unit and browser suites. Greptile is 5/5 with no review findings or open
threads.
- The plan separately records a released-container overlap proof: 130
reads, 26 saves, zero failures, two deleted predecessor containers and a
retained authenticated session. That image predates this restore change.

## Risks

- A cold folder can use eight simultaneous streams instead of one. Two
file-transfer database connections bound the simultaneous restore
callers. Streams verify bytes without loading entire objects into
memory.
- The 30-second deadline is unchanged. The measured size and latency do
not qualify every folder at the maximum storage limits.
- No schema, API or UI changes. Paid-model and hosted deployment
acceptance remain separate work.

## Model Used

OpenAI Codex, GPT-6. Tool use, code execution and repository review. The
runtime does not expose a more specific model ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 04:09:47 -06:00
DottaandPaperclip 006d5691a6 feat(server): support volume-free web server handoff (#18518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web server also owns controller admission and user commands.
> - A rolling deployment starts a replacement before the old server
exits.
> - Shared database state alone does not preserve local agent files,
browser chunks, or in-process commands.
> - This pull request adds an opt-in ephemeral server contract and a
local overlap test.
> - The benefit is a web server that can relinquish traffic and local
files without losing acknowledged work.

## Linked Issues or Issue Description

Refs #18403, #18327, #18335, #18341.

**Current behavior**

The control-plane web server still has local agent-file authority and
local command targets. A new container can miss saved bytes or send a
command to the wrong process. An existing browser tab also needs chunks
from its original build.

**Proposed behavior**

Eligible servers share only PostgreSQL, object storage and stable
secrets. A replacement waits for readiness. The old server drains
admitted requests, keeps owner commands available through handoff, and
releases controller leases. Explicit maintenance adopts existing files
before the volume is removed.

**Reason and benefit**

This is the application prerequisite for rolling app-only deployments.
Schema migrations and permanent multi-replica operation are outside this
change.

## What Changed

- Store agent-file manifests in PostgreSQL and immutable bytes in S3.
Treat local roots as disposable projections. Preserve existing
permission checks and edit conflicts.
- Add an explicit migration, eligibility and object-retention command to
the production runtime bundle.
- Add `/api/ready`, admitted-request drain, browser restart signaling
and retained immutable UI chunks.
- Carry transaction-aware work hints across controllers during handoff
and active execution. Keep listeners closed when idle.
- Deliver durable Stop and runtime responses to the execution owner.
Recheck request lifecycle and access before response dispatch.
- Reject local-only configuration in ephemeral mode. Keep the feature
disabled for existing installations by default.
- Add an isolated PostgreSQL/S3/proxy/browser deployment harness with
independent server homes and retained failure evidence.

## Verification

- All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`:
workspace typecheck, build, Runner checks, server/general tests and
browser shards. Greptile reviewed the runtime changes at 5/5 with all
findings resolved; the final commit updates maps and the plan.
- Local targeted checks passed: coordinator 27; Stop/recovery 33;
PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap
Stop 1; restart recovery 75. PostgreSQL command tests simulate the
provider edge.
- Local browser proof passed 91 reads and 19 saves; reconnect took
1,013ms. The tab opened Audit from S3 after deletion of A's whole
home/build. This uses compiled workspace code and local-trusted auth,
not released containers or live agents.
- Local workspace typecheck/build passed; current server and runtime
bundle passed. The earlier broad local suite was stopped after three
hours and is not a passing receipt. Its retained failures and isolated
reruns are documented in the plan.
- Stacked #18526 adds stricter graceful-exit proof, an unready
replacement control, live Runner qualification and a Stop handoff
correction. Hosted enablement remains gated on those results and Cloud
#1096. No hosted stack has been enabled.

## Risks

- Apply migration 0422 before enabling the feature. Existing data must
pass each owner's migration and eligibility checks. Do not discard a
volume based on a single migration page.
- Durable manifests remain authoritative after the opt-in is disabled.
Rolling back to a binary that ignores them is unsupported.
- Cold directory restoration and complete shutdown timing still need
qualification before enabling a hosted stack.
- Retained UI assets must cover the supported browser and rollback
window. Cleanup is explicit; no new idle maintenance polling is
introduced.
- This change does not claim production zero downtime or authorize a
fleet-wide enablement.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window limit are not exposed in this session. Used reasoning,
code editing, shell execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 01:58:54 -06:00