Commit Graph
5 Commits
Author SHA1 Message Date
Nicky LeachandPaperclip 08f3e7ebec Retain passive Codex output while Core reconnects (#19029)
## Thinking Path

> - Paperclip manages work that AI agents execute through a Runner.
> - Core controls the work while the Runner owns the provider process.
> - Core can restart while an admitted provider turn is active.
> - Reconnect waits used to leave provider output in the process pipe.
> - Ordinary provider polling can also wait on a provider RPC before
Core reconnects.
> - This pull request drains passive local Codex output into bounded
durable storage between reconnect attempts.
> - Core receives the original events after recovery and keeps authority
over tools and approvals.

## Linked Issues or Issue Description

Refs #17594. Related work: merged #19353 supplies the Codex
receipt-limit fence and deadline-aware reconnect scheduler.

This PR now contains only shared passive intake and the already-live
native Codex adapter. A controller outage must preserve admitted work
and event identity without restoring a provider, sending a provider RPC,
or executing a tool during reconnect.

## What Changed

- Add a default no-op disconnected-poll hook. Managed providers and ACPX
retain that default.
- Drain passive frames from the existing local Codex process between
reconnect attempts. Requests, terminal reconciliation and lineage checks
wait for the connected path.
- Persist each accepted event before acknowledging the provider queue.
Soft-limit pressure retains the rejected suffix until Core ACKs free
space. Hard failures stop the owned provider and preserve the durable
prefix.
- Preserve credential observations and the existing lease, runtime,
grace and provider deadline bounds. Document the intake contract and
test replay through the real PRP controller.

## Verification

The slice was replayed onto master `bf10bc695a`. `git range-diff
f453509e6f..20626230b9 bf10bc695a..HEAD` reports an unchanged patch. The
diff has 11 files and contains no ACPX adapter or sidecar changes.

Targeted local checks on this head:

- `cargo test --manifest-path
packages/paperclip-runner/runner/Cargo.toml --locked -p
paperclip-runner-core --lib disconnected -- --test-threads=1`: 12
passed. These cover passive deferral, no restoration, the real managed
HTTP worker, durable backpressure, oversized output and existing
receipt-limit cleanup.
- The 10 `credential_observer::tests::` checks passed. The Runner daemon
and recovery example build, and workspace Rust formatting passes.
- `vitest run src/control-plane/runner-recovery-fixture.test.ts
--maxWorkers=1`: four passed. Both controller replacement modes retain
the same process and replay event identities. The deadline and
silent-handshake cases also pass.
- `vitest run src/live/runnerd-codex-transport.test.ts -t
late-remote-release-error --maxWorkers=1`: one passed. Whitespace and
added-line secret checks pass.

Earlier review found that disconnected ordinary polling could perform
synchronous managed history reads. The real-worker regression retained
here proves no history or stream-reopen request occurs until connected
polling. Earlier slice preparation hit a sandbox loopback denial and a
missing local daemon; the permitted loopback test and the test after
building the daemon both passed. The current-base checks above run with
those prerequisites in place.

Historical CI on the old combined head failed an unordered
AI-connections migration comparison. The faithful full-file local check
passed 158 tests, and the failure remains recorded
[here](https://github.com/paperclipai/paperclip/actions/runs/42112385671/job/127543554397).
Earlier receipt-limit delivery failures are addressed by merged #19353.
Previous review scores belong to the old combined diff. On current head
`b92d4b604ee6992eeefbc02c8d41ec4f2a608a4d`, [CI run
42125840239](https://github.com/paperclipai/paperclip/actions/runs/42125840239)
passed. All 59 checks completed with 55 successes and four intentional
skips. Standard review 13 and Apex review 14 each scored 5/5 on this
exact head, with zero comments and no unresolved threads. The single
Apex request is [recorded
here](https://github.com/paperclipai/paperclip/pull/19029#issuecomment-6671993072).

## Risks

During an outage, native Codex intake uses 25 ms ticks between reconnect
attempts. A provider request or terminal waits for Core. Saturation can
pause intake or produce a failure. Managed remote streams retain their
existing limits. Tests use controlled local providers; live provider
qualification and deployment are separate.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning effort, through
Codex. Used source inspection, local native and PRP tests, and code
review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 15:29:38 -08:00
Nicky LeachandPaperclip b61a815d3d Enforce Codex receipt-limit deadlines while Core is offline (#19353)
## Thinking Path

> - Paperclip uses a Runner to execute agent work.
> - The Runner keeps a record of each tool call, called a receipt.
> - When that storage reaches its limit, the Runner asks Codex to stop.
> - Core can stop responding while the stop request is pending.
> - Codex may already have written its completion behind earlier output.
> - The Runner must keep its stop deadline and preserve that real
completion.

## Linked Issues or Issue Description

Refs #19029. This is its first isolated slice; #19029 remains unmerged.
Related #17594 covers controller replacement. Related #17516 retains
receipts during explicit stop; this change handles the autonomous
receipt-limit deadline.

## What Changed

- Keep the existing 2s/30s stop deadlines while Core is offline,
including when a connection waits for a reply. Other lease and runtime
deadlines still apply. The Runner sleeps between connection attempts and
deadline checks.
- Stop the owned local Codex process when its deadline expires. Read its
remaining output within the existing size and message limits, then
process it in order. Only after output has ended and all earlier
messages have been processed can the Runner report a missing completion.
- After the local process has stopped as requested, skip replies that
only reject or cancel old requests. Commands, user responses and actual
tool results still require successful delivery.
- Keep the stop behavior, regression tests and transport documentation
in this slice.

## Verification

Standalone checks on this branch:

- Seven existing Codex tool-record limit integration cases pass.
- Twenty focused checks cover deadline cleanup and retained output,
including 273 distinct messages followed by one real completion, missing
completion, and a pipe that stays open.
- Separate checks cover strict tool-result delivery, three process-stop
cases, and lease/runtime/reconnect time limits.
- Four controller process tests pass. They include Core accepting a
connection but never replying, and a reconnect delay longer than the
allowed recovery period.
- Existing warm attachment still retains the completed turn's remaining
output and usage.
- Rust formatting, diff checks and the local secret scan pass.

The extraction first omitted an existing test helper and the connected
retry preamble. Standalone compilation and receipt tests caught both
omissions. Restoring them produced the results above. Full regression
checks run in GitHub CI.

Hosted CI passed both native platforms and typecheck, but [server11
failed](https://github.com/paperclipai/paperclip/actions/runs/42118725143/job/127565828671)
one unchanged status-card test. Its fake clock advances while real
database work runs; the final clock assertion was 50 ms past its limit.
The identical test, service and scheduler passed all 36 cases in [the
earlier baseline
shard](https://github.com/paperclipai/paperclip/actions/runs/42111889388/job/127541685625).
The local comparison first needed a built SDK, then stopped at embedded
Postgres startup before any test ran. It did not reproduce the
assertion. The single failed-jobs-only retry passed. The original
failure and the blocked local comparison remain recorded here; the retry
does not establish a precise timing cause.

At exact head `f453509e6f`, all 59 hosted checks completed: 55 succeeded
and four were skipped. Snyk also passed. Standard review round 1 and
selected review round 2 each scored 5/5 on this head, with no unresolved
threads.

## Risks

The Runner now stops its local Codex process at the original deadline
even while Core is unavailable. If another program keeps the output pipe
open, or queued output exceeds the existing limits, the Runner cannot
confirm the outcome. It reports that uncertainty and keeps the records
already written to disk. Tests use real child processes with controlled
provider fixtures. Live provider and deployment qualification remain
separate.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`), high reasoning effort, through
Codex. Used source inspection, native subprocess tests and targeted
TypeScript tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 14:47:17 -08:00
DottaandPaperclip 3910352c92 feat(runner): add experimental Cursor Cloud provider (#18068)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Runner gives each provider durable execution, tool, and completion
contracts.
> - Cursor Cloud currently has a direct adapter outside that native
contract.
> - Cloud work needs exact ownership across HTTP, SSE, Stop, and
recovery.
> - This PR adds a qualified Cloud API driver and verified proxy to
Runner.
> - Product setup and hosted placement remain separate dependent
changes.

## Linked Issues or Issue Description

Refs #5664, #11803, #17299, and #17471. Product integration is #18069.
Managed-only API placement is #18310. The related direct-adapter
model/transcript work is not replaced. No duplicate native Cursor Cloud
integration was found.

## What Changed

- Add native `cursor_cloud` / `cursor_cloud_api` with explicit
model/repository validation, bounded HTTP/SSE, durable per-turn
checkpoints and a qualified Node proxy launched through the Rust facade.
- Commit terminal custody before publication/release; coalesce
consumers, preserve ambiguous create/Stop/recovery fences, reject late
tools, and distinguish definitely-unsent oversized requests from
ambiguous responses.
- Support scoped cold attachment, current authority, reconnect and
verified artifact staging. Keep unsupported warm/external and Windows
paths closed.
- Preserve early source cursors and verify immutable legacy prefixes
before suffix replay, including supported compaction/masking. Optional
proof publication failure falls back to complete cold verification.
- Preserve canonical records at the durable native producer with
JSON-safe privacy and the existing 8MiB bound; drain admitted successor
flushes. Reconcile upstream semantic/pack/schema/lifecycle changes and
retain S3 and legacy CLI checks.

## Verification

Core head `37c442ec8dd4`: all 66 check contexts are terminal
successful/neutral/skipped, including required `ci / verify` and `ci /
e2e`; fresh exact-head Greptile is 5/5 with zero unresolved threads.
Dependent Product `81e828d20a77` and Top `dd2959689627` include the
later client handoff correction and are running their own fresh gates.

Core: 47/47 Rust, 4/4 compiled-native retirement, 19/19 settlement and
normal Runner build passed. Runtime Top `9af1a5de42a2`: build and
recursive typecheck passed. Earlier source-scoped evidence includes 398
focused tests, five relay tests and token/type gates. The 251-assertion,
7/8-suite run failed physical PostgreSQL teardown; the unchanged stable
WSP suite passed 128 through the documented wrapper.

The downstream fixture corrections passed 7/7 browser and 121/121
startup tests without timeout/oracle relaxation; a broken-unsubscribe
control fails. Hosted runtime `9af1a5de42a2` has a passing normal build
and recursive typecheck. Latest dependent heads additionally contain a
pure client fix for stale live/history overlap, with 48 focused tests,
UI typecheck and token gates passing. Its new client is not deployed in
the frozen hosted image; server/native/API/provider source is unchanged
by these four downstream files.

Fresh hosted Linux admission on `9af1a5de42a2` retired its owned
runner/proxy/socket with zero prompts. The saved configuration survived
reopening, and Test authenticated the bound key/catalog/selected model
without a prompt. Artifact with `claude-sonnet-4-5` completed,
downloaded the exact 27 bytes, matched 121 canonical identities against
complete logs and independently reached FINISHED. The active Stop retry
was clicked within its immutable boundary, produced one canonical
interrupted turn and acknowledged run-only cancellation, retained a
valid 35.076s quiet window (maximum gap 4.584s), and independently
remained CANCELLED/artifact-absent beyond the original 180-second
command deadline. All provider-observer credential callbacks, DB clients
and loaders closed; accounting is settled and explicitly unpriced. The
first fresh Stop retains its failed 10.373s sampling gap and every
sample.

On the upgraded9af server, the historical F226 artifact was reopened and
downloaded with its exact original hash after its provider agent had
been deleted/GET404 earlier today. Historical F226 full Artifact/Stop
cleanup, Question, corrected Plan and reconnect retain their original
identities. Deleting the three newly created disposable provider agents
and the new-case post-delete grade remain blocked on explicit user
approval after automatic review rejected expanding the two original
approved IDs. No new-case cleanup or complete fresh matrix pass is
claimed; this is separate from the directly observed task/Stop behavior
and retained exact cleanup proof.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier startup-title, title-save,
custody/Stop and deletion failures with unknown causes and bounded
recoveries. `dea023` Artifact passed 17 checks but Stop failed its false
warning and 50 duplicate log identities. Pre-repair ACK failure and
original Core CI failure remain recorded; retry alone establishes no
cause or full local-suite pass. Provider dollar cost stays UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. The caller’s
explicit values are submitted and behavior checked live. Unpriced
provider dollar cost stays UNKNOWN.

Only qualified per-turn controllers are admitted. Uncertain work and
incomplete cleanup stay fenced. Existing damaged/duplicate archives
remain failed evidence. Prefix proof adds bounded object-store I/O; the
8MiB producer bound does not guarantee future providers’ output fits.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 08:38:57 -06:00
DottaandPaperclip 9f524ec1a7 fix: persist native Runner usage before transport acknowledgement (#18354)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agent execution must preserve usage so Paperclip can price work and
enforce budgets.
> - Remote Runner execution can outlive its web receiver.
> - The receiver previously acknowledged Runner events before its later
usage recorder saved the accounting receipt.
> - This pull request saves required native usage receipts before that
transport acknowledgement.
> - A replacement receiver can accept the same retained report and
create one ledger charge.

## Linked Issues or Issue Description

**What happened?**

Native remote Runner usage could leave its durable outbox before the web
receiver saved its accounting receipt. The later recorder required a
local receipt directory. A replacement web receiver could lose the only
remaining report when the database write failed.

**Expected behavior**

The Runner retains unaccepted structured usage. Postgres owns accepted
receipts. Repeated delivery preserves one receipt for each source report
and one charge for the run. Missing usage keeps accounting pending.

**Steps to reproduce**

1. Start a supported built-in native remote Runner.
2. Make the receipt database insert fail after the provider reports
usage.
3. Replace the web receiver and its local directory.
4. Reconnect the same Runner and inspect the saved receipt and ledger.

**Paperclip version or commit**

The initial investigation used
`ebd05e9f4c88c66af7473db86bbaacfd1544aa0a`.

**Deployment mode**

A web server with a remote native Runner and shared Postgres.

Related work: #18327 owns output logs and provider traces. This change
has no dependency on its S3 export path. #18031 covers interrupted Codex
receipt recovery. #18005 covers legacy spool reads. #17516 covers Codex
stop-boundary retention.

## What Changed

- Add native producer receipt delivery through the existing Runner
callback before transport acknowledgement.
- Reuse `run_usage_receipts`, stable source sequences, and deterministic
report identities. Reject conflicting replay and retry a lost commit
response with the same receipt.
- Restore per-turn usage from structured receipt facts. Preserve
provider turns that share one PRP envelope turn.
- Bind retained warm transport callbacks to the current run authority
epoch, allowing old attachment control acknowledgements while preserving
accounting ownership.
- Persist Codex session baselines and cumulative totals in the existing
provider checkpoint. Mark qualified whole-run deltas, preserve them
across resume/restart/attachment, and leave old unmarked reports
incomplete and unpriced.
- Normalize the bounded final usage tail before the terminal event.
Retain later increases for explicit reconciliation, block further
attachment, and fail an affected successor without changing its
baseline.
- Complete runs from the accepted immutable receipt and its exact saved
rate-card amount; preserve existing live-budget enforcement.
- Skip the local receipt recorder for supported built-in native remote
Runner execution. Keep local, legacy, and external Dot/Muse behavior
unchanged.
- Keep pricing, reservations, ledger settlement, source fencing, and
budget holds with their existing owners.
- Put the actual Runner integration test in the existing Rust-cached
native test lane. Build before opening Postgres and give cold
compilation its own bounded budget.

## Verification

- Fresh Greptile review is 5/5 on
`e7376a50c8162770da1d581ecf25c8cffac64520`, with no unresolved findings.
Parent and independent peer review approved the final changes.
- Real PostgreSQL 18.6 and a real Rust Runner exercise failed receipt
insertion, receiver-directory loss, replay from the same producer, and
exactly one 25-cent ledger charge. The actual accounting spool path
stays absent. Optional logs/traces are disabled. The provider is
controlled; no paid request is made and the fixture never pre-saves the
expected receipt.
- Producer checks passed 32 usage/provider tests,37 terminal controls,
corrected completion, all 3 previously failing exit cases, and both full
transport goal-continuation cases. Rejected active-goal completion
retains its 120-token tail before failure. Baseline 100→successor150→33
duplicate old snapshots→old120 retains explicit incomplete
reconciliation evidence and positive budget exposure.
- Heartbeat executor checks passed 982/982. PostgreSQL tests preserve
exact 1.002 USD pricing, the 100.2-cent ledger amount, budget holds,
source takeover, partial/final receipts, cost-only corrections, equal
rate-card replay, and stale-completion interleaving. Newer incomplete
usage remains pending rather than settling an older snapshot.
- Runtime replay and strict usage-schema checks passed 2/2; generated
contracts and Rust formatting pass. Test-partition contracts passed
32/32.
- [Final combined-head
evidence](https://github.com/paperclipai/paperclip/pull/18354#issuecomment-6531526158)
records full workspace typecheck/build and 41/41 accounting tests
against all five cloud PR heads, including this final accounting commit.
It includes commands and evidence limits.
- Exact-head CI is green: 53successful checks, 6intentional skips, and
the legacy Snyk status passed. Typecheck, build, Runner, server,
browser, and aggregate gates pass. Runner Vitest1/2 initially failed an
unchanged mocked Antigravity login fixture; its focused case passed
locally, then one unchanged job retry passed 131 files/1995 tests,
including all 11 Antigravity tests. The initial failed attempt remains
recorded in CI.

Initial local attempts exposed the host SysV semaphore limit and
corrected harness path, IPC readiness, and queue-counter defects. A
temporary test loader selected an existing POSIX-semaphore PostgreSQL
18.6 build without changing repository dependencies. An optional
expanded debug provider run was interrupted at its unrelated 8192-turn
rollover stress case; no full-file local pass is claimed. Required
failure cases were run directly, and the full Runner CI remains
required.

## Risks

- The guarantee covers built-in native remote Runner execution. Local
native execution, legacy adapters, external Dot/Muse execution, and
server-originated model calls retain their existing contracts.
- This does not provide traffic switching or general web-server
failover. The existing controller identity, checkpoint, and generation
gates still apply.
- The receipt JSON gains optional structured producer facts. No schema
migration or accounting S3 spool is added. Text output and raw trace
payloads are excluded.
- Codex accounting authority requires a Runner artifact built from this
provider baseline implementation and qualified by digest. Update
receivers and strict usage-schema readers before releasing that Runner;
PRP does not negotiate this payload field. Existing unmarked Codex
reports remain pending/unpriced. No deployed Runner artifact is claimed
fixed.
- Usage arriving beyond the bounded terminal tail may require manual
reconciliation or a fresh session. Its evidence remains in the existing
producer checkpoint/outbox; no adjustment ledger or recurring scan is
added.

## Model Used

OpenAI `gpt-6.1-sol` assisted with the implementation, tests, and
documentation through code execution and repository tools. The context
window size is unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 15:20:42 -05:00
DottaandPaperclip 0a08d37aa0 feat(runner): recover controller authority without interrupting execution (#17097)
## Thinking Path

> - Paperclip manages agents and owns their business authority.
> - Runner owns execution transport, acknowledgements and replay.
> - A replacement controller must recover that transport without
starting another provider turn.
> - An external store can retain the required control records without
copying streamed text.
> - A duplicate local journal adds synchronous disk work without
providing the replacement server's authority.
> - This PR adds the optional recovery contract and its standalone
proof. Server adoption and browser delivery have separate owners.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

A surviving runner can reconnect to a replacement controller with its
admitted process, session and active turn.

**Current behavior**

Local controller files do not provide recovery on empty replacement
storage. Rewriting those files after an external checkpoint is installed
also blocks normal controller work.

**Proposed behavior**

An optional callback stores opaque checkpoints. The replacement restores
that checkpoint before it constructs the controller. Installing the
callback makes the external store authoritative. Local storage holds a
marker that requires restoration. Consumers without the callback retain
their file journal.

PR 1 of four: #17097 → #17439 → #17455 → #17456. Related work: #17390
and #17361. The pre-split reference remains
`codex/hot-upgrades-reference-2026-10-24`.

## What Changed

- Add opaque checkpoint encoding, restoration, incremental persistence
and authenticated sparse recovery cursors.
- Keep authentication, commands, semantic inputs and lifecycle events
behind their commit barriers. Share semantic-call ownership between live
dispatch and recovery.
- Keep streamed text out of recovery checkpoints. Preserve bounded
retention, original deadlines and admitted process identity.
- Stop normal local journal writes after external persistence is
installed. Fail closed on a local marker until external restoration
occurs. Preserve current master's staged file writes for standalone
consumers.
- Keep provider-native admission and authorization behind adapters.
Preserve historical Copilot attestations and bind the merged launch and
transport sources in v68; preserve published v66 and prior development
identities.
- Add the one-process scripted executor fixture and deterministic
outage, replay, failure and backpressure tests. Add no production test
endpoint or auxiliary process.

## Verification

The completion-recovery correction passes three restart regressions that
fail before its respective transport/driver fixes. All 230
controller/attestation cases, all 302 Codex transport cases in the full
repeat, and all three one-process fixture cases pass. The broader
Codex/Dot/Muse run passes 326 cases and times out one existing mid-goal
case; its isolated retry and complete Codex repeat pass on unchanged
source and binaries. The initial timeout remains recorded. Conformance
and replay parity pass.

Historical v65/v66 attestation bytes remain frozen; v67 binds the
changed transport. The local runner-only API-authority lane passes 2,027
cases but its 47 HTTP cases cannot initialize embedded Postgres on this
Mac. The integrated owned-database HTTP path is verified in PR 2. At
preceding head `0d8d1b8250`, the Actions workflow passes after one retry
of runner-service shutdowns. Greptile is 5/5 with no unresolved threads.
The interrupted attempt is retained. All checks on that preceding head
are terminal and green/neutral/skipped.

Run `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/control-plane/durable-prp-control-plane.test.ts
src/live/runnerd-codex-transport.test.ts --maxWorkers=1` and `node
packages/paperclip-runner/scripts/copilot-profile-identity.mjs --check`.

The [qualification
record](https://github.com/paperclipai/paperclip/blob/codex/hot-upgrades-qualification/doc/plans/2026-10-22-hot-upgrade-qualification.md)
retains exact revisions, failed attempts and qualification limits. The
local cost campaign observes zero hot-path local journal fsync calls. It
does not qualify a native harness or hosted deployment.

Master reconciliation `d5fc443626` preserves the published v66 and prior
development identity bytes and binds the combined transport in v68. All
290 focused runner/profile cases and runner typecheck pass. The first
sandboxed invocation cannot open loopback listeners; the permitted
repeat passes. Fresh CI and Greptile are pending for this updated head.

Current merge candidate `4fb3d1835b` also fixes public-reader
publication before external commit. New held/rejected-write cases fail
before correction and pass afterward. The full controller repeat passes
168 cases. The combined lane passes 390 other
recovery/transport/profile/fixture cases; its first two overlap fixture
failures are retained and corrected to observe callback entry instead of
provisional state. Runner TypeScript build and typecheck pass.
Current-head GitHub Actions is green and Greptile is 5/5 with all
threads resolved. The superseded master-reconciliation CI shutdown
remains recorded.

## Risks

The persistence callback is an acknowledgement barrier. Failure must
stop delivery. A consumer that installs external persistence must
restore its committed checkpoint before restarting; the marker is not a
recovery snapshot. Sparse display recovery does not reconstruct
acknowledged text or repeat execution. This slice has no Postgres
schema, application authorization or browser behavior.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 17:10:32 -05:00