Commit Graph
6 Commits
Author SHA1 Message Date
DottaandPaperclip 374a918d67 fix(runner): retain qualified OpenCode artifacts for cold recovery (#19622)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native recovery must keep admitted provider state and useful
workspace files.
> - OpenCode binds its saved session to exact executable paths and
hashes.
> - A replacement image can remove those files even when session data is
saved.
> - Retaining a manifest alone cannot restore the missing launch files.
> - This pull request saves the qualified files before provider work and
restores the same owned paths.
> - The benefit is cold continuity without relaxing the Runner guard or
replaying a prior turn.

## Linked Issues or Issue Description

Refs #19558. That fix retains the selected pack for ordinary
continuation. This change adds durable file retention for a cold
allocation. Related #19487 adjusts the admission budget; it does not
retain launch files. A search of open and closed OpenCode cold/durable
artifact PRs found no duplicate.

**What happened?**

A saved OpenCode session could outlive the image that supplied its Node
executable, proxy, and native executable. A new image supplied different
bytes. The Runner then correctly refused the changed launch profile. The
saved harness did not contain the old files.

**Expected behavior**

Core must save qualified launch files before the first turn. It must
restore those exact files under the same owned account and absolute
paths. Unknown ownership, missing files, and changed hashes must still
stop recovery.

**Steps to reproduce**

Start an admitted remote OpenCode session. Save its workspace, encrypted
checkpoint, and harness data. Destroy the original allocation and
controller files. Restore into a new allocation with an image whose
qualified proxy has different bytes. Resume the same session. The new
opt-in local qualification command automates this path with actual
OpenCode and a local synthetic model.

## What Changed

- Store company-bound, encrypted launch objects and canonical run/backup
references. Generate migration 0438 through the normal database
workflow.
- Stage qualified Node, proxy, and native OpenCode files under a private
content-addressed OS-account home. Record the account UID and exact
launch paths. Preserve the Rust profile guard.
- Gate first provider work on durable capture under the current native
claim. Bound Stop/deadline waits while the actual late transfer keeps
its worker and scratch ownership until settlement. Restore cold caches
with verified staging and atomic rename. Keep live reattachment and
historical image-rooted sessions on their existing paths.
- Reuse bounded canonical expiration owners. Atomically fence old
checkpoint authority with a strict company/run-scoped retirement marker
before external deletion. Keep its indexed cleanup anchor across failed
deletes and restart. Preserve earlier resume handles and reject later
header or source-consumer writes. Add no idle scan or network call
inside a transaction.
- Add storage, race, executor, filesystem, and actual Rust resume
regressions. Add the opt-in local Linux qualification command and
operating limits in `doc/opencode-cold-recovery.md`.

## Verification

- RED before repair: real PostgreSQL/AES cold restore lacked the launch
files; normal executor preparation chose the new image pack. Separate
regressions exposed lost upload receipts, missing cache repair,
last-owner retirement, an account-UID mismatch, and a resume-handle race
during external deletion. Each expected failure was captured before its
repair.
- Local qualification before the final master integration: 1,169
executor tests, 233 storage/archive/admission/expiration tests, all
seven server typecheck layers, migration safety, and the actual Rust
cold-resume test passed.
- The maintained Linux command passed with two cached immutable
Paperclip images whose proxy hashes differ. It removed the original
account/controller trees, restored the production published cwd under a
new lease, resumed the same actual OpenCode session, kept a useful file
and prior model context, and made one new synthetic model request. No
external model call or network was available in either container.
- The object-store fixture uses real encrypted StorageService streams
with an independent owned local directory. It is not hosted S3
acceptance or a complete task HTTP/actor workflow. Historical tasks
still require normal effects reconciliation and recovery after
deployment.
- Review regressions reproduced partial-cache publication,
resume-versus-deletion, fresh-writer and source-consumer resurrection,
and unbounded Stop/deadline waits before each repair. They now pass
through real filesystem, encrypted storage, PostgreSQL, and SDK
boundaries. File validation is split from database ownership (384/831
source lines).
- Clean runtime head `aadab7acc69ff959678030db46ffe2dcfbefe28f`: the
maintained actual OpenCode cold qualification passed with `dirty:false`.
It preserved the provider session and recorded launch paths, restored
useful workspace content and prior model context, and made exactly one
resumed synthetic model request. The 314,560,000-byte launch capture
took 23.118 seconds; the later harness was 833,536 bytes. Subsequent
repairs change backup-reference retirement and governed continuity-cache
classification; capture, restore, launch, and published-workspace inputs
are unchanged, so that runtime receipt remains scoped to those paths.
The continuity-break consumer is verified separately.
- Retention repair `c14d386cecf7123257918ba6a8d47307df2da397`: 68/68
affected real PostgreSQL/AES storage tests and 15/15 focused executor
authority/admission cases pass. All seven server typecheck layers pass
(source layers and unchanged test shards, with the final 590-file test
layer repeated after fixture edits). All review threads have
evidence-backed resolutions. Its subsequent CI caught the separate
lifecycle-classification gap described below. The refreshed Paperclip
review reported paused work, not a completed new review.
- Final head `4d1cd3e3309e328b11abc85adc6d78c6d0c7f55d`: all 19 focused
root-entry/executor cases and all three source declaration layers pass.
[Greptile reviewed this exact head at
5/5](https://github.com/paperclipai/paperclip/pull/19622#issuecomment-6695984874).
Final-head [CI
verification](https://github.com/paperclipai/paperclip/actions/runs/42308908167/job/128214100339)
passes: 60 successful checks, five conditional skips, and the explicitly
neutral paused Paperclip review. No check is pending or failing.
- CI also caught the launch cache missing from the session-root
lifecycle contract. Both the writer scan and a real governed-replacement
consumer failed locally before repair: old launch bytes remained in the
active root. The cache now archives with the prior session state; the
consumer verifies exact bytes are preserved in the archive and absent
from the active root.
- The latest review found that backup retirement could delete an
independent run pin while a selected session retained the checkpoint. A
real PostgreSQL/AES regression failed before the repair. Backup-only
ownership now uses a NULL-safe exact backup predicate; the run pin
belongs to fenced checkpoint expiration. The regression verifies
checkpoint decryption and launch-file restoration after pending-backup
cleanup, followed by deletion only after removing the final resume
handle.
- An executor first-session case checks new-session durable capture
through the real capture helper and receipt propagation into the actual
harness manifest at the backup boundary. Real storage cases verify the
run-profile receipt and canonical backup/run pins.
- Sequential closure reuse and simultaneous first admission are
different scopes. A deterministic two-uploader test measures both
independent uploads, one verified ready winner, loser-only deletion, and
usable restoration for both runs. Each capture is bounded to 512 MiB and
five minutes; up to three pending receipts per run are retained. There
is no claim of global single-flight capture or a company-wide egress
bound.
- Earlier optional warm-ownership verification encountered missing local
binaries. After exact-source setup, all nine ownership assertions
passed; two other suites encountered local PostgreSQL setup/teardown
contention, so those interrupted attempts are not passes. Required
current-head CI supplies the broad final coverage. No deployment or
hosted historical recovery is claimed.

## Risks

- This adds two tables and a forward migration. The file object is
bounded to 512 MiB. Capture adds one initial transfer; later harness
archives exclude those large files. Company-scoped ready objects are
shared and retained until the last canonical owner retires. Concurrent
initial admissions can duplicate bounded transfers; each keeps
independent ownership and cancellation rather than waiting on another
controller. A continuity break retains the controller cache with its
prior state until normal session-root cleanup, adding up to one bounded
launch closure per retired state.
- Cold continuity requires the same qualified Linux platform, supported
runtime dependencies, owned account UID/home, and published absolute
cwd. A mismatch fails closed. No alias or Rust guard change is included.
- Existing image-rooted profiles have no retained file object. This
change cannot backfill missing historical binaries. A fresh governed
session must reconcile prior effects first.
- Live reattachment does not read artifact objects. Storage or
provenance failures before a new turn hold admission. They do not grant
new execution, model, tool, or replay authority.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
execution, and local browser tools. The exact served model ID and
context window are not exposed in this session. No claim is made for
unavailable model metadata.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 17:30:00 -06:00
DottaandPaperclip 7c0922a7b6 fix(native): preserve provider backups during slow workspace saves (#19359)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Remote native sessions save provider state so another web server can
continue the same conversation.
> - A provider can suspend while its accepted result is still saving a
large workspace.
> - The run has released its live lease at that point. A new
backup-owner lookup rejects this uncommitted state.
> - This pull request captures the real owner during backend creation
and retains it with the current session authority.
> - The existing database checks still reject a changed boot,
generation, result, or successor before inspection and during
publication.
> - The benefit is that a slow workspace save no longer prevents the
provider backup.

## Linked Issues or Issue Description

**What happened?**

A remote session can fail to save its provider backup with
`native_harness_storage_owner_missing`. The provider has suspended, but
its accepted result is still in `workspace_finalizing`. The close
callback tries to acquire a new owner after the run released its lease.
A later allocation cannot use a backup that was never published.

**Expected behavior**

Save the suspended provider with the authority acquired during
execution. Recheck that authority before provider inspection and before
and after upload. Keep result finalization and provider-work permissions
unchanged.

**Steps to reproduce**

1. Start a remote native session with durable provider storage.
2. Accept its result and release the live lease while its workspace save
remains pending.
3. Close the provider transport. The old owner lookup rejects the save
before provider inspection.

**Paperclip version or commit**

Reproduced on master `bf10bc695abaef6b20e85b287f456b5ce2edb266`.

**Deployment mode**

A web server with object storage and remote managed Runner sessions.

Related: #19225 retains failed-turn ownership through close. #18436
restores an external controller checkpoint before a changed warm process
is replaced. This change fixes late backup-owner acquisition while an
accepted result saves its workspace.

## What Changed

- Capture the provider backup owner when the remote backend still owns
its execution lease.
- Bind that owner to the current session authority. An older close
callback selects the current warm attachment's owner. Revalidate the
captured fence before inspecting or skipping provider state.
- Add a regression for the close callback during workspace finalization.
- Restore encrypted provider files with a captured owner while workspace
finalization remains pending. Reject boot, generation, result, and
successor changes during upload.
- Document the ownership contract.

## Verification

- Observed the callback regression fail on the old code with
`native_harness_storage_owner_missing`. The same test passes with this
fix.
- `cd server && node ../node_modules/vitest/vitest.mjs run --config
vitest.config.ts
src/services/native-runtime/native-harness-backup-storage.integration.test.ts`:
59 passed, using real PostgreSQL and encrypted object write/readback.
- `node scripts/check-module-boundaries.mjs`: passed.
- Full workspace typecheck: passed with the complete production fix.
- Native execution: 1,151 passed. Managed warm lifecycle: 9 passed.
These tests require a lost-owner error during shutdown, including before
an incomplete provider checkpoint is skipped. The final change to the
lifecycle test also requires rejection after lease renewal is lost.
- Full local `pnpm build`: passed on the final head.
- The full local `pnpm test:run` stopped in chat shard 6 on five
existing deadline assertions. Earlier chat shards passed; the PR CI chat
shards passed. This is not a full local-suite pass.
- The first CI server shard 4 attempt failed the unchanged legacy
environment-inheritance fixture at its five-second completion wait. The
same failure reproduced locally on the repair head and on an earlier
checkout without this repair. One unchanged CI retry passed. No fixture
assertion or timeout change is shipped.
- Latest-head CI has passed. Greptile reviewed the final head at 5/5
with no open threads. The separate Paperclip Review is paused. This PR
does not claim staging acceptance.

## Risks

The backup owner now lives with its session authority. A wrong authority
selection must fail rather than publish provider state. Database
publication still checks the original company, task, run, boot,
generation, and accepted result before and after upload. Tests change
the owner during upload and verify that the selected backup stays
intact. No schema, data migration, or new provider-work permission
changes.

## Model Used

OpenAI Codex, GPT-6 model family, with reasoning and code execution
tools. The host does not expose the exact runtime model ID or
context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 18:04:47 -06:00
DottaandPaperclip f7a4c5678f feat(runtime): pre-copy completed harness history before enrollment (#19137)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server needs provider history in shared object
storage.
> - Offline enrollment must preserve old backups before it can detach a
volume.
> - A large copied dataset still exceeded the maintenance deadline with
parallel transfers.
> - Most of its bytes belong to successful turns with released runtime
authority.
> - This pull request pre-copies that history while unrelated executions
continue.
> - The final offline migration still verifies every source and grants
enrollment authority.

## Linked Issues or Issue Description

Refs #19108, #19038, #18943.

**What happened?**

A copied instance has 1,953 local harness backups and 54.8 GB of
declared source bytes. A resumed parallel migration still reached its
25-minute deadline. It cannot safely enroll by moving all history during
maintenance.

**Expected behavior**

Copy eligible completed history while the server remains online. Defer
active, failed or changed sources. Keep all existing data and final
enrollment checks.

**Steps to reproduce**

1. Retain many local native harness backups on a volume-backed instance.
2. Attempt a full offline conversion within the maintenance deadline.
3. Observe that most time moves historical bytes rather than current
execution state.

## What Changed

- Add finite `native-harness-prestage.mjs` preparation from a normal
runtime build. Require explicit opt-in and shared object storage.
Process one source at a time with a 90-minute deadline.
- Use the recorded released runtime owner of each successful, finished
turn. Do not assert that all writers stopped.
- Recheck checkout, related queued/running work, successful result
ownership, session, Runner, exact lease stamp and unchanged backup at
database publication fences.
- Defer source changes, malformed source JSON and unsafe backup roots.
Log each such source and its reason. Keep database failures and
cancellation fatal. Keep completed objects valid for retry. Never return
an enrollment-ready receipt.
- Reuse the bounded migration source walker. Keep offline admission and
final verification unchanged.
- Keep private lifecycle state, task ownership and provider policies
unchanged. Document the operator path and its limits.

## Verification

- A zero-source scan now checks the existing database obligation query.
A known mismatch blocks with a source-missing receipt; empty
fresh/already-durable scans and nonempty deferred scans remain valid
advisory results. The prior command fails the mismatch regression. All
16 command and runtime-bundle tests pass for this follow-up.
- At head `7ba3dbc7aa`, 87 focused command, database-timeout, storage,
archive, publication and runtime-bundle tests passed with real
PostgreSQL and filesystem restoration.
- The current follow-up passes all 68 backup and archive tests. A source
renamed after preparation fails the new regression on the previous head,
then defers and permits the next backup with this fix. An archive output
ENOENT remains fatal. Existing database-error and cancellation tests
pass.
- New race tests fail against the previous storage owner: a changed
backup, queued turn or revoked lease otherwise publishes.
- Tests prove unrelated executions stay running, same-task work defers,
failed history stays deferred, retry verifies existing objects, and
private bytes restore after deleting the old home.
- Command review fixes require the configured existing vault key, retain
it for the process, set database lock/statement/idle bounds, close
clients on cancellation, and cap shutdown at ten seconds. Abort
immediately sets failure status and prints one blocked receipt. A real
child with real shortened timers reproduces the old missing-receipt exit
and verifies the fix. Tests also use a real PostgreSQL lock.
- Malformed manifest or runner JSON and symlink replacement fail the new
preparation and publication-fence regressions on the previous code. The
fixed command defers each damaged source and copies the next eligible
backup. Separate tests keep database SyntaxError and cancellation fatal.
Offline migration still rejects damaged sources.
- The bundle test runs the emitted preparation entry with plain Node
without starting the app.
- A prior copied-data experiment completed with 1,738 new publications,
nine verified current objects and 206 deferred sources. Later checks
found different database and filesystem capture times: 43 source runs
were absent from the older database snapshot, and one required
historical manifest was absent from the retained filesystem archive.
This experiment does not prove a consistent restore or enrollment
readiness.
- Full current-head CI at ce63517d0c
passed: 54 successful checks, two skipped and one neutral, with no
failed or pending statuses. Typecheck, tests and build pass. CI:
https://github.com/paperclipai/paperclip/actions/runs/41970370960. The
earlier superseded run lost a spot runner after a shutdown signal; that
infrastructure cancellation is retained and is not counted as a passing
run.
- Greptile and Ponytail both rate this exact head 5/5; all actionable
review threads are resolved. Comment-only cleanup removes newly added
prose comments under CONTRIBUTING.md. TypeScript output with
removeComments:true is byte-identical for all seven changed TypeScript
files; 20 existing command, runtime and bundle tests pass after cleanup.
- Serving enrollment, full normal-image migration, empty-home
restoration and active-agent release handoffs remain separate acceptance
gates.

## Risks

- Preparation still consumes CPU, disk and storage requests. It uses one
transfer and at most two database connections. Check memory headroom for
a separate operator process.
- A task can gain work during upload. The final fence rejects that
publication and retains the previous current object.
- Deferred sources remain migration obligations. Preparation never
permits detaching the volume.
- Ordinary backup callers do not opt into the new preparation constraint
and keep their existing behavior.

## Model Used

OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning,
code editing, shell tools and test execution. The runtime does not
expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 23:15:46 -06:00
DottaandPaperclip 7d46532c6d fix(runtime): bound parallel offline harness backup transfers (#19108)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An ephemeral web server must retain provider history outside its
local disk.
> - Offline enrollment moves verified local harness backups to object
storage.
> - A cold rehearsal reached 699 current backup rows while processing
1,953 sources before its 25-minute deadline.
> - The serial transfer loop prevents this existing dataset from
enrolling.
> - This pull request overlaps at most four transfers within the
existing total scratch limit.
> - The benefit is faster preservation with the same ownership and
publication checks.

## Linked Issues or Issue Description

Refs #19038, #18943, #18932.

**What happened?**

Offline enrollment processed 1,953 local provider backups one at a time.
The normal Cloud image reached 699 current backup rows (including nine
pre-existing rows) before maintenance expired. The serving instance
remained unchanged during the isolated test.

**Expected behavior**

Enrollment must preserve every source within bounded time and disk use.
It must stop all in-flight transfers before it reports failure.

**Steps to reproduce**

1. Prepare a settled instance with multiple verified native harness
backups.
2. Run offline enrollment against shared object storage.
3. Observe that the serial owner waits for each archive, upload,
readback and publication before starting the next source.

## What Changed

- Process groups of up to four verified offline harness backups.
- Reserve space for the archive and encrypted copy before each group
starts. Keep the total scratch limit equal to one maximum-size serial
transfer.
- Derive a smaller archive limit from a closed manifest directory set.
Preserve unlisted files with an exclusive maximum-size reservation.
- Enforce the reservation while streaming. Keep the existing maximum for
ordinary runtime backups.
- Cancel and await peer transfers after failure. Keep completed
publications valid for retry.
- Preserve the original archive-stream failure when the tar writer also
fails after destruction.
- Document the migration bound and add integration regressions for
overlap, restoration, cancellation, retry, extra files and archive
limits.

## Verification

- Serial baseline: the new overlap regression fails with one active
transfer when four are required.
- Candidate: 59 targeted tests pass with real PostgreSQL, S3, filesystem
archives and restored provider bytes. The bundled-entry test is skipped
locally because this source worktree has no runtime bundle. The full
normal image rehearsal remains required.
- Follow-up: 48 archive/storage checks pass after correcting the
PAX-header allowance. The real encoder produces a 972,801,024-byte
full-entry projection, above the old 805,306,368-byte allowance and
below the new 1,024,001,024-byte bound. The existing global object limit
remains in force. The old formula fails the new regression. Exact-size
restoration and one-byte-over-limit rejection pass.
- The previous revision passed full CI after one setup-route timeout
retry. That failure remains recorded; the corrected current revision
f7c2be1289 passed fresh full CI (41937700826) and Greptile 5/5. Its
review thread is resolved.
- Local server typecheck found four test-only Promise library errors;
those are fixed. The remaining 26 diagnostics come from stale linked
dependency declarations outside the changed files. Clean full CI is
required before merge.
- The resumed isolated staging-copy experiment also exceeded its
25-minute deadline. It reached 1,389 current backups. This improves
transfer throughput but does not by itself qualify enrollment; online
preparation is tracked in #19137.
- Full fresh normal-image migration, empty-home restoration and hosted
release handoffs remain deployment acceptance gates.

## Risks

- Parallel work increases concurrent object-storage and database
requests to at most four transfers. The scratch reservation still bounds
archive and ciphertext files together.
- A malformed or changed source can fail its streaming limit. The
migration retains its source and stops before export.
- Partial completed publications survive failure. Retry verifies them
under the existing hash and ownership rules.
- No serving configuration, deployment pin, database contract or
provider lifecycle policy changes in this pull request.

## Model Used

OpenAI GPT-6.1 Codex, exact model ID `gpt-6.1-sol`. Used for reasoning,
code editing, shell tools and test execution. The runtime does not
expose its context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 18:16:36 -06:00
DottaandPaperclip aaea05e158 fix(native): preserve backups when completed warm sessions retire (#18986)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native Runner keeps provider history when a warm session ends.
> - Object storage must preserve that history before the sandbox is
released.
> - A completed turn releases its live controller lease.
> - The backup owner lookup rejects that released lease and stops the
next task preparation.
> - This pull request uses the existing completed-run checkpoint fence
for backup storage.
> - The next turn can prepare after the previous provider state is
saved.

## Linked Issues or Issue Description

Refs: #18932, #18011, #18984.

**What happened?**

A warm Codex turn completed and kept its provider session alive. The
next turn stopped during preparation with
`native_harness_storage_owner_missing`. The backup lookup required a
live lease after normal completion had released it.

**Expected behavior**

Save the completed session under its retained controller generation and
committed result. Keep the existing owner and successor checks. Do not
start another provider turn during the save.

**Steps to reproduce**

1. Run a warm native session with object-backed provider backups.
2. Commit the first turn and release its controller lease.
3. Start the next task preparation and retire the previous session.
4. Observe the backup owner error before provider work starts.

**Paperclip version or commit**

The regression is present at `86814c6550f52e99ede150a883bdc773cbb540e3`.

**Deployment mode**

A hosted server with a remote native Runner and S3 storage.

## What Changed

- Accept a released backup owner only when its exact controller record
has a committed result.
- Reuse the checkpoint owner policy for same-controller preparation and
native successors.
- Recheck the controller generation, committed result, and released
state after upload.
- Preserve live-owner rejection and the rule that a late upload cannot
replace a newer conversation backup.
- Add real PostgreSQL storage and restore regressions. Document the
completed-run backup rule.

## Verification

- On 2026-11-04, two new regressions failed on the unchanged source with
`native_harness_storage_owner_missing`. These covered an idle completed
session and a same-controller preparing successor.
- After the fix, 54 selected tests passed: 30 durable backup tests and
24 existing early-retirement checkpoint tests. After the fixture-only
upstream sync, all 58 selected storage, checkpoint, process, and
affected fixture cases passed again.
- The storage cases use real PostgreSQL, encrypted archives, object
transfer, and restoration. They cover invalid completion evidence,
foreign or expired successors, active predecessors, changed generations,
changes during upload, and newer backup publication.
- Command: `pnpm exec vitest run
server/src/services/native-runtime/native-harness-backup-storage.integration.test.ts
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t 'released|backup|early retirement|durably retires an accepted warm
Runner'`.
- An independent source review found no outstanding code issue.
- Full repository typecheck passed with `node
scripts/typecheck-low-memory.mjs` before the upstream fixture-only sync.
Production source is unchanged across that sync.
- An assembled process probe passed with the Runner binary built from
this head and a fake Codex protocol peer. It started real processes,
committed a result, released the source lease, claimed the next
controller, and completed the source stop, drain, and suspend commands.
It recorded one provider turn start and kept the task bound to its
successor.
- Command: `pnpm exec vitest run
server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
-t 'durably retires an accepted warm Runner'`.
- CI exposed the old allocation-readiness test setup in three cases. The
branch now includes the merged #18984 fixture correction. The same three
cases passed locally after that sync.
- An earlier CI test shard was cancelled after its runner received a
shutdown signal. Its log was retained before one rerun. Another old-head
shard reported failure with no retrievable job log or annotations; its
cause remains unknown. Current-head CI must pass independently.
- Local `pnpm build` passed at this head. All 1,904 chat cases passed
across the supported local shards; the earlier shards used
byte-identical source before the fixture-only sync. The interrupted
default command is retained and is not reported as a pass.
- The local bulk server run had separate enrollment acknowledgment,
environment-run status, monitor-settle, idle-history teardown, and
repository-rehome failures. In isolation, idle-history passed all 75
tests and the unchanged repository cases passed with their original
limits (4.9s large files, 6.4s refs). The remaining four local failures
reproduced on unchanged master `04a24f4e06` under the same environment
and limits. Their causes remain unproven. Logs and the baseline
comparison were retained; the local full suite is not claimed as passed.
- Current-head CI first hit the unchanged repository-rehome large-file
case's 15s timeout. The same isolated case passed in 4.9s. After
source/call-path comparison, one bounded failed-shard retry passed all
115 executed suites; no assertion or timeout was changed. The original
timeout remains recorded.
- Local verification uses macOS 26.5.2 arm64, Node 26.4.0, pnpm 9.15.4,
PostgreSQL 18.1, and Cargo 1.97.1. The current tested head is
`622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`.
- The full unfiltered [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/41868587974)
passed at `622f69f76ff68c71eacdcbcbc6ab4d4c6eb83bf9`: all 12 general
server, 3 chat, 9 serialized, workspace, native Runner, build/typecheck,
E2E, and final verify lanes. The file selector was skipped, so this was
the complete fallback matrix.
- Greptile reports 5/5 on this exact head with no open review threads.
Disposable local test processes were cleaned up and the source tree is
clean.
- The real hosted Codex and S3 workflow has not been rerun because the
local verification environment has no remote provider credentials. The
original task remains unchanged.

## Risks

The main risk is a stale controller publishing provider history after a
new owner starts. Both storage transactions keep the company, run, boot,
and generation fence. A released owner must keep its committed result.
Existing successor checks and the newer-backup publication guard remain
in force. The live checkpoint writer still requires a non-null lease. No
migration or API change is required.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository tools, and code
execution. The exact serving model ID and context window are not exposed
in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks; separate
baseline broad-suite failures are disclosed above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 11:15:58 -06:00
DottaandPaperclip 612d9931a6 Support per-turn Runner sessions on ephemeral web servers (#18932)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The web server controls native Runner sessions in remote sandboxes.
> - An ephemeral web server must recover without its previous local
filesystem.
> - The existing admission rule accepts warm sessions. Managed Claude,
Grok, and OpenCode sessions resolve to per-turn execution.
> - Per-turn execution also needs durable provider-session bytes after
the sandbox is retired.
> - This pull request adds encrypted backup publication and removes the
warm-only admission rule.
> - Both lifecycle modes can use the existing native ownership and
checkpoint contracts during a web deployment.

## Linked Issues or Issue Description

Refs: #18518, #18529, #18826, #18835, #18841, #18864, #18887.

**What existing behavior does this improve?**

Native Runner execution on web servers with
`PAPERCLIP_EPHEMERAL_SERVER=true`.

**Current behavior**

The admission rule requires a warm Runner lifecycle. Managed provider
credentials can resolve an agent to per-turn execution, so enabling the
ephemeral deployment profile blocks those turns. A database controller
checkpoint does not preserve provider files after the old web home and
per-turn sandbox are removed.

**Proposed behavior**

Admit native execution in either resolved lifecycle. Keep provider and
credential choices unchanged. Publish encrypted provider state to shared
storage before sandbox cleanup. Restore it from a fresh web home for the
next turn. Use the existing controller lease and generation checks for
active takeover.

**Reason and benefit**

Operators can remove the web server volume without excluding per-turn
providers or forcing managed accounts into warm sessions. New native
Daytona profiles automatically enter both rolling-deployment
qualification lanes.

**Breaking changes**

The additive migration creates a backup publication ledger. Existing
volume-backed local-disk installations retain their local backup path.
S3 installations publish encrypted backups and require a stable secrets
master key. Existing-volume enrollment must import surviving session
data before detaching the volume.

## What Changed

- Admit native Runner execution independently of its warm or per-turn
lifecycle. Keep host-owned legacy execution excluded from ephemeral
deployments.
- Store private provider state as encrypted objects. Journal uploads
before I/O, read them back, and fence publication against the current
native owner. Keep the previous published backup when storage or
ownership checks fail.
- Restore exact provider bytes into a private cache. Require an exact
durable backup before sandbox cleanup. Extend the existing bounded
retention pass to retire stored objects without adding idle polling.
- Import selected legacy controller and provider state in the existing
offline maintenance process. Preserve queued work, task pointers,
pending recovery, and original files. Prefer canonical database
checkpoints.
- Preserve already-failed legacy controllers with missing original
artifact proof as encrypted archives. These archives do not become
execution authority. Successful retained sessions still require proof.
Report archival and executable imports separately.
- Verify unmanaged Grok API keys through the existing xAI account
validator. The live test exposed a readiness gap that kept remote Grok
agents paused after platform checks. Preserve failed credentials and the
separate runtime warning.
- Ingest Grok whole-prompt usage at the owning ACP boundary. Bind the
receipt to the exact outbound request/session; reject replay, conflicts,
and incomplete authority. Separate turn cost from cumulative session
cost in the sidecar schema and Rust normalization. Preserve managed
billing identity and identify direct xAI API credentials.
- Preserve published workspace directory names across cold allocations,
including destroyed-allocation cleanup. Keep unpublished epoch/lease
directories isolated and refuse to overwrite an occupied preparation.
This fixes the cold-session identity regression exposed by #18922.
- Increase bounded checkpoint-header inspection to 1 MiB. Retain the
aggregate inspection limit.
- Register all nine native Daytona profiles in per-turn and warm
rolling-deployment suites. Delete the entire departing server home,
exercise ordinary reads and writes, require active ownership transfer,
then check a later turn on another fresh server. Permit only exact
notification replay after reconnect.

Ownership stays with the existing modules: Runner validates its protocol
state; Core validates run scope and controller ownership; storage owns
object I/O; the maintenance process owns offline conversion; the E2E
wrapper owns process replacement and probes. No provider protocol is
added to the shared admission rule. Grok receipt parsing stays in its
provider module. The shared sidecar gains an explicit cost basis because
its existing price field means session-cumulative spend, while Grok
reports prompt totals. Existing consumers retain cumulative behavior
when the new field is absent.

## Verification

- Focused native admission, archive validation, publication fencing,
cleanup, readiness, legacy import, and E2E observer tests pass. Real
PostgreSQL cases verify that queued intent and historical recovery rows
stay unchanged.
- Full typecheck and build pass on the rebased branch. The review
follow-up passes server typecheck, 29 focused
PostgreSQL/archive/readiness tests, and module-boundary checks. Revision
fb8d39a7a1 reached 55 successful CI checks and Greptile 5/5 with no
unresolved threads. The Grok follow-up passes Runner/server typechecks,
238 ACP runtime/projection/usage tests, 16 sidecar-schema tests, the
Rust cost regression, and Core billing regression. The e9aae9cf revision
passed all 57 successful checks, with 5 skips and 2 neutral checks, and
exact-head Greptile 5/5. The latest 9cbec30a4f follow-up changes only
E2E catalog/admission controls; its 291 affected tests pass, all 57
check contexts succeeded (5 skips, 2 neutral), and exact-head Greptile
is 5/5 with no open review threads.
- Migration numbering and safety checks pass.
- A local real PostgreSQL/S3 roundtrip preserved 2,870,101,591 payload
bytes across 89,100 archive entries after deleting the source directory.
Restored SHA-256 matched; upload/readback took 56.5 seconds, restore
74.8 seconds, process peak RSS was 1.40 GB, and all fixture resources
were removed.
- Archives preserve contained relative links, reject link
chains/traversal, and enforce independent byte/entry/object limits.
Offline verification streams hashes. Orphaned deleted-run backups stay
on the original volume. Obsolete object retention preserves the selected
backup, live lease references, and pending uploads with owners.
- Isolated API-key Codex deployment: original active runner/session
transferred to the replacement and committed. All 567 reads, 57 writes,
and 57 readbacks succeeded. A second replacement completed a
same-session continuation with all 287 reads and 29 writes succeeding.
The strict cell failed two agent-authored shell commands; this result is
retained as failed.
- Isolated API-key Claude deployment: original active runner/session
transferred and committed. All 584 reads, 59 writes, and 59 readbacks
succeeded. A second replacement completed a same-provider-session
continuation, with 211 reads and 22 writes succeeding. The strict cell
failed an initial completion call that omitted a registered attachment
citation; the agent corrected it. This result is retained as failed.
- Isolated API-key OpenCode deployment: the same active runner/session
transferred and committed. All 481 reads, 49 writes, and 49 readbacks
succeeded. A second replacement retained the provider session and
completed useful work, with 156 reads and 16 writes succeeding. The
strict cell failed agent tool calls: a denied external-directory write,
malformed artifact hashes, and a second completion after the first
result was committed. There was one submitted/completed turn and one
committed result; reconnect notification replay was identical. This
result is retained as failed.
- Grok's real first turn now completes with settled usage and provider
cost ($0.41397); the next turn is admitted normally. That attempt then
exposed a shared workspace-path identity regression before deployment.
Three cold-restore cases reproduced it before the fix; 106
workspace/transport tests and four focused cold-restore cases pass
afterward, including destroyed allocation and occupied-directory
preservation. Server typecheck and build pass. At b5b0dbb03f both
controller switches passed the read/write probe without failure, and the
active runner retained its process while the controller generation
advanced. The third turn exposed an invalid reasoning-token subset
check; ad07e681ba fixes it with the actual receipt regression. Two later
live attempts failed with an upstream Grok rate limit before deployment.
A tiny preflight prompt succeeded before the second attempt, but the
full workload was still rate-limited. The final live gate remains
incomplete. No accounting or identity guard was bypassed. Earlier failed
attempts and their exact resource-cleanup receipts remain recorded.
- Earlier Codex/Claude/OpenCode receipts predate #18922; they establish
their tested revisions. Current-head live qualification remains pending;
#18943 separately preserves pre-enrollment logical/provider workspace
addresses and adds actual legacy migration proof. Its first Codex
migration attempt stopped safely at skill-version verification.
- Earlier attempts remain recorded: missing local provider-pack setup
blocked one Claude launch, and the observer rejected identical reconnect
replay in another. Setup and the observer were corrected before the
latest run.
- Named staging enrollment has not started. Its serving revision and
post-deployment journeys remain separate delivery gates.

- Broader E2E harness validation exposed deployment-catalog and
installed-lane coverage drift; the affected 291 controls now pass.
Twelve other failures reproduce on unchanged base 1a39dea9aa. They
concern cleanup call expectations, dated operating-overhead fixtures, a
Pi profile-version expectation and workflow-security expectations. No
baseline failure is reported as passing.

## Risks

- Provider archives can contain credentials. Objects use authenticated
encryption with scoped associated data and a wrapped data key. The
original stable master key remains required.
- Storage failure prevents publication and sandbox cleanup. Operators
must keep shared storage available and preserve the previous volume
during enrollment.
- Legacy state without verified identity cannot become executable
recovery authority. Archival preserves evidence and existing failures;
it does not recover already-lost provider files or clear blocked tasks.
- Local overlap probes establish the tested application behavior. They
do not establish Railway routing behavior or live managed-subscription
qualification.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, code execution,
and live integration tests. The exact runtime variant and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-04 02:12:51 -06:00