Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 a7ea2a799a fix(ephemeral): save retained runtime skill bundles at startup and never strand a volume-free server on lost ones (#22190)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A volume-free server keeps no local volume. It must find every file
it needs in durable storage.
> - Before a volume-free server starts, a data gate refuses to start if
records still point at files on a local disk.
> - Old native runs can keep runtime skill bundles that a release before
#19729 wrote only to its own disk. The gate reports these as
`runtime_skill_assets_require_migration`.
> - On a volume-free server those files are not on the disk, and no
volume can be attached again. So the refusal repeats on every restart,
and nothing can clear it. A staging stack crash-looped like this and
stayed down.
> - This pull request makes the server copy every retained bundle that
is still on its disk into durable storage, and start over the bundles
that are gone, the same way #20990 handles harness backups.
> - The benefit is that a volume-free stack can always start again. No
one has to repair it by hand.

## Linked Issues or Issue Description

Refs #19675, #19729, #20990

**What happened?**

A volume-free staging stack ran for twelve days on a release that kept
runtime skill bundles only on its container disk. When that container
stopped, every newer release crash-looped at boot with `Volume-free
server data is not eligible: runtime_skill_assets_require_migration`.
The old container was gone, so the bundles were gone. The deploy could
not roll back, and a retry could not help.

**Expected behavior**

A volume-free server starts. It saves each retained bundle that is still
on its disk to durable storage. It starts over a bundle that is not on
its disk when the volume-free owner marker proves that no volume is
mounted, and restart recovery owns those runs.

**Steps to reproduce**

1. Retain a native run with an admitted runtime skill bundle that has no
version ID and no durable preservation receipt.
2. Remove the bundle from the instance root.
3. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true` and the
volume-free owner marker present.
4. Startup fails with `runtime_skill_assets_require_migration`, on every
restart.

## What Changed

- `runtime-skill-preservation.ts`:
- The obligation scan is one generator.
`hasLocalRuntimeSkillObligations` uses it.
- New `preserveLocalRuntimeSkillObligations` copies each obligated run's
bundles with `preserveRuntimeSkillAssets` and returns `{ bundles,
unsaved }`.
- A missing source is decided at its reader:
`readLocalManifest`/`readSource` throw `RuntimeSkillSourceAbsent` (other
instance root, or a missing directory, manifest or listed file). Only
that error marks a run unsaved. Any other failure, including an `ENOENT`
from storage or cache work, propagates.
- `preserveRuntimeSkillAssets` no longer aborts on an invalid or
non-durable existing receipt. It copies the local bytes again and
replaces the receipt.
- One durability helper (`preservedAssetRowDurable` for metadata,
`preservedAssetDurable` with a metadata or object read) replaces the
copies in the scan, staging, locking, preservation and rehydration.
- `runtime-skill-preservation-reference.ts`: the scan matches only null
pins, which is exactly what preservation copies. A non-null pin is a
version pin and cannot become an obligation that nothing can settle.
- `file-manifest-cache.ts`: new `inspectFileManifest` returns
`verified`, `missing` or `changed` from the same walk.
`verifyFileManifest` wraps it.
- `ephemeral-server-contract.ts`: the gate saves the bundles still on
the disk. If none stay unsaved, the blocker clears. Unsaved runs are
waived only when the volume-free owner marker is present, with a
warning. A save failure still refuses. The decision uses the copy pass
result, not another scan. One exported blocker constant.
- `ephemeral-migration-data.ts`: offline enrollment accepts the blocker
and copies the bundles from the attached volume, in a phase after the
package phases. Any unsaved run still fails the enrollment.
- `index.ts`: comment update only.
- Tests: `ephemeral-server-runtime-skill-startup.test.ts` (real scan,
real copy, embedded Postgres) and an S3-gated case in
`ephemeral-migration.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/ephemeral-server-runtime-skill-startup.test.ts`
- Red on `origin/master`: 2 of 3 fail with the production error
`Volume-free server data is not eligible:
runtime_skill_assets_require_migration`.
- Green with this change: all cases pass (lost bundle, saved bundle, no
marker, mixed present and lost, one missing listed file, non-version
pin, mixed pins, non-durable receipt recapture, storage failure, storage
ENOENT, unsaved set).
- `cd server && npx vitest run
src/__tests__/ephemeral-server-startup-run-state.test.ts
src/__tests__/runtime-skill-preservation.test.ts` pass (existing gate
and preservation behavior unchanged).
- `cd server && npx tsc --noEmit -p .` passes.
- `ephemeral-migration.test.ts` needs `PAPERCLIP_TEST_S3_ENDPOINT`. It
typechecks, but it was not run here.

## Risks

- A run whose bundle is lost now starts under restart recovery instead
of blocking the whole server. Recovery of that one run can fail. Before,
the whole server stayed down, and the bundle was lost anyway.
- Startup does more work only when unsaved legacy bundles exist. A
current release saves bundles before admission (#19729), so new runs
never add this work.
- No schema change. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:20 -06:00
DottaandPaperclip a3eecf1fc9 Preserve admitted runtime skill bundles during server replacement (#19675)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs keep the exact skill files admitted for execution and
recovery.
> - Older inputs can name immutable local bundles without a skill
version ID.
> - Installed skill heads alone do not preserve those admitted bundles.
> - Server replacement needs a durable source for each retained bundle.
> - This pull request copies authenticated bundles through the existing
package owner and restores their exact projections.
> - The benefit is preservation during online migration without stopping
active work or settling recovery warnings.

## Linked Issues or Issue Description

Refs #19656. Related: #19660.

**What happened?**

Older native execution inputs can contain skill bundles with a null
version ID. Recovery skips those bundles. Startup installation checks
can pass even when those retained inputs still require the old
filesystem.

**Expected behavior**

Preserve each retained admitted bundle in object storage. Restore its
original bytes, modes and digests on an empty replacement host. Keep its
null pin and all run, controller and recovery state. Block startup and
enrollment while a concrete local asset obligation remains.

**Steps to reproduce**

Retain a native run with an admitted local skill bundle and a null
version ID. Convert its installed skill head to durable storage. Replace
the server home and attempt recovery. The old bundle path remains in the
recovered context.

## What Changed

- Add a bounded `runtime-assets` phase to the existing skill package
maintenance CLI. It copies exact admitted files under the runtime asset
publication lock, verifies the durable package, and appends a run-bound
receipt after source revalidation.
- Keep original files, null pins, task intent, warnings and controller
leases. Support an active run through copy-only preservation. Do not
start a turn or claim provider ownership.
- Verify and reuse a healthy canonical local bundle before storage
reads. Restore missing or damaged bundles through the existing package
and native asset materializers. Use the authenticated original manifest
to preserve exact read and execute modes. Accept a later contract
revision only when the run binding and exact skill selection remain
unchanged.
- Check retained recovery obligations and current task sessions during
both startup and migration eligibility. Ignore ordinary admission that
has no completed execution input.
- Protect receipt-backed packages from orphan collection. Serialize
receipt publication and deletion on the package row, then check
references in a fresh statement after the lock.
- Add real PostgreSQL and filesystem regressions. Replace a
delete-method proxy with an actual upload-renewal row-lock race. Publish
mutation activity only after commit.

## Verification

- Baseline production startup gate failed the new concrete-asset
regression as expected. The fixed gate passed. The actual before-fix
preservation owner also failed recovery from a healthy local bundle
during a simulated storage outage. A verified local-first path fixes
that regression.
- Eight preservation tests passed with real PostgreSQL, streaming
storage, archive extraction and native materialization. They cover cold
recovery, exact modes and binary bytes, active ownership, changed
sources and bindings, contract revisions, missing or corrupt receipts,
and publication/collection races.
- Six affected package collection/deletion controls and three existing
startup controls passed. The existing versioned recovery consumer passed
a sequential retry.
- One fixture attempt failed on its cleanup foreign-key order; the
fixture was corrected. One existing suite hit its unchanged 30-second
database startup hook before test execution. The unchanged single
consumer passed on retry. These failures remain recorded.
- Source, changed-test and CLI TypeScript checks passed. Rebase
preserved both the local instruction-copy and runtime-asset guards.
Current-head changed-test checks passed. Final PostgreSQL rerun passed
all eleven preservation/startup cases. Current CLI check passed.
Required CI remains pending at the trusted-runner gate.
- Current canonical bundled creation uses package preparation plus
atomic content/head publication. Native preparation requests immutable
version snapshots before delivery. Converted installations and new
canonical bundled creation therefore do not recreate the historical
null-version asset dependency.
- Fresh Greptile review on exact head
`54c7dd048c9016c1af9fc68213d5ae053ac16499` passed at 5/5 with zero
unresolved threads.
- Module boundaries, whitespace, and credential/internal-reference scans
passed. No live source writes, provider actions or migration apply were
performed.

## Risks

Missing, changed or unsafe local files remain blockers. This command
does not repair a previously published package whose objects later
disappear or become corrupt. It also refuses malformed preservation
receipts. It retains the original local files and prior evidence and
keeps removal blocked. Automatic repair requires a separate canonical
package-owner recovery operation; direct database edits are not an
accepted recovery path. Failed publication keeps the original files and
any unreferenced package for normal collection. The readiness check is
explicit startup or enrollment work, not a background scan. Its cost
grows with retained admitted asset obligations. Older readers must be
upgraded before the old volume is removed. This change preserves asset
bytes; it does not settle business recovery or qualify unrelated volume
dependencies.

## Model Used

OpenAI GPT-6 (Codex). The runtime did not expose a more specific backend
model ID or context-window size. Used reasoning, code editing, and tool
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 18:23:53 -06:00