PaperClipAI

Jannik/PaperClipAI

0 Fork 0

mirror of https://github.com/paperclipai/paperclip.git synced 2026-10-30 13:20:54 +01:00

Commit Graph

Author	SHA1	Message	Date
DottaandPaperclip	567c043b17	feat(server): expose bounded hot-upgrade readiness (#18037 ) ## Thinking Path > - Paperclip coordinates agent work and keeps its authority in the server. > - Remote native runners can preserve execution through a server replacement. > - Deployment operators need evidence that current runs support that recovery. > - Process-local drain counters do not establish this fact. > - This PR adds an optional, operator-only readiness observation from existing records. > - A deployment consumer can use that report while the existing runtime retains recovery authority. ## Linked Issues or Issue Description Refs #17439. What happened? A deployment cannot distinguish recoverable remote native executions from runs that require a task drain. The existing status endpoint reports only process-local counts. Expected behavior An instance operator can request a bounded readiness report without stopping admission or exposing checkpoint material. Missing ownership or persistence evidence must prevent readiness. Ordinary status reads must keep their current cost. Steps to reproduce Run a remote native task. Read the task-drain status. It reports active work but supplies no recovery eligibility. Deployment code therefore has no affirmative signal to select an active-run handover. ## What Changed - Add `GET /api/instance/task-drain?upgradeReadiness=1` for instance administrators and existing Cloud control callers. - Return aggregate eligibility from current run identities, controller leases and bound opaque checkpoint headers. Unknown work, pending admission and incomplete reads prevent a positive report. - Bound the explicit read to 1,001 rows, encrypted headers of at most 256 KiB each and 8 MiB total, and a two-second SQL statement timeout in a read-only transaction. Keep ordinary status reads in memory. - Add PostgreSQL, count-transition and route authorization tests. Update the API contract, operator docs and access maps. ## Verification - All seven readiness cases pass against an isolated PostgreSQL 18 fixture. They cover missing checkpoints, legacy adapters, missing process/session identities, foreign owners, expired leases, missing lease tokens, failure state and in-flight work. The positive SQL fixture uses the real Runner checkpoint writer and encryption. Regular sandbox workspace identity and plugin environment-lease identity are tested with independent IDs. Stopped/released/undispatched plugin tasks and every mismatched identity field are rejected. Local/legacy/malformed/oversized or mismatched headers are rejected. A held PostgreSQL table lock proves the two-second cancellation budget, rollback/connection release and unrelated query responsiveness. - All 123 focused readiness, instance-settings and Cloud control assertion tests pass, including operator-only access and no readiness read on ordinary status requests. Runner’s 168 checkpoint tests pass. Runner and server typechecks pass after review fixes. - Workspace typecheck, full build, module-boundary and map checks passed locally before the master rebase. Current-head GitHub CI is now green, including typecheck, build, all grouped/serialized server and Runner tests, browser shards, policy and review-dispatch checks. Greptile completed cleanly at 5/5 for `f15fb0c7e9dfb975090cd5d791a1c8e6701275f8`; all actionable threads are resolved. - The broad local test run was stopped after the macOS embedded PostgreSQL fixture failed to bootstrap. The SQL test was then run on the isolated PostgreSQL fixture. No assertion or timeout was weakened. One unrelated existing route test exceeded its 15-second limit under concurrent local builds; it passed on one isolated retry and the subsequent whole route suite passed. - Hosted active-run upgrade acceptance has not run. This PR supplies an observation; it does not change rollout policy or establish live continuity by itself. ## Risks This is an instance-wide observation, so company members and agent keys must not receive it. A readiness snapshot can age as work changes; shutdown and startup retain their existing execution authority and fencing. The report does not validate a replacement artifact or authorize a schema migration. It decrypts only bounded headers through the checkpoint owner and delegates format/identity inspection to Runner; full journal restoration and physical sandbox authentication remain with recovery. No migration, recurring query, new checkpoint write or wire protocol change is included. The additive Runner header inspection contract reuses its existing parser. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, source inspection, tool use and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass for the affected contracts; the broad-run limitation is recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 11:28:15 -05:00
DottaandPaperclip	25f1070fbd	fix(runner): preserve checkpoint proof across warm preparation (#18011 ) ## Thinking Path > - Paperclip coordinates agent work and preserves its execution state. > - Native Runner checkpoints preserve authority across controller changes. > - Warm sessions also need checkpoint writes while their successor prepares. > - External persistence replaces the local state file with a marker. > - Two existing checks treated those valid states as lost ownership or prior activity. > - This PR uses the existing preparation lease and canonical controller state to keep those checks accurate. > - The result is safe warm-session retirement and fresh-session startup on reused sandboxes. ## Linked Issues or Issue Description Related: #17439 and #17320. The open #17401 changes the recovery welcome window and does not cover these paths. What happened? A configuration change could fail before the next provider turn with `runner did not durably suspend before checkpoint`. The task already pointed to its preparing successor, but that successor did not yet have a native coordinator. The old completed runner's maintenance checkpoint was rejected. A separate fresh session on a reused sandbox could fail with `resumed_unavailable`. Checkpoint registration had replaced the full local state file with an external-persistence marker. The bootstrap check still required the old full-file format. Process supervision then reported unconfirmed termination before launch. Expected behavior Retire the completed warm runner under the same process's live preparation lease. Admit a fresh remote session only when the registered controller proves exact identity, no prior activity and pending bootstrap commands, and the remote session directory can be claimed atomically. Steps to reproduce 1. Complete a native warm turn with external checkpoint persistence. 2. Change configuration so instruction preparation must retire that session, then submit a follow-up. 3. Separately, admit a fresh native session on a reused sandbox lease after external checkpoint registration. Paperclip version or commit Reproduced in the checkpoint implementation introduced by #17439. Fixed against current master. Deployment mode Source checkout tests and a hosted Docker/Daytona incident. No deployment or original-task retry is part of this PR. ## What Changed - Accept completed-run maintenance under the exact successor's live, same-process preparation lease only before a native coordinator exists. - Preserve original controller generation, accepted-result, no-new-work, company and task checks. - Pass the registered controller through the existing connection callback and read its current canonical state for untouched-session proof. - Keep marker-only, identity, activity, backup and atomic remote-directory guards. - Add regressions and update execution semantics and session verification recipes. ## Verification - Warm retirement red: the real Postgres writer rejected the preparing successor's `runner.suspend` checkpoint with `native_runner_checkpoint_owner_lost`. - Warm retirement green: 14 Postgres ownership cases pass, including stale/foreign/expired owners, conflicting native authority and new provider work. - Bootstrap red: real controller registration and external persistence caused the resumed and replacement positives to fail. Six external negative cases passed. - Bootstrap green: all 34 bootstrap cases, the bootstrap-state predicate and 13 connection-driver cases pass (48 total). - `node scripts/check-maps.mjs` and `git diff --check` pass. - Full `pnpm -r typecheck` and `pnpm build` pass. - Full local `pnpm test:run` remains partial/interrupted. The default attempt could not initialize embedded Postgres on this Mac. A dedicated Postgres 18 attempt ran for 17 minutes without a completed file result before interruption; the generic helper separately confirmed unsupported embedded Postgres (`initdb` failed after five attempts) and ignores the external database URL. The generic probe skipped one test. Setup and interrupted-run logs are retained; neither is claimed as a behavioral red or a full local pass. - Real source-hosted Codex/Daytona acceptance was attempted at `6ce7fcacc4` with `gpt-6.1-sol`, `xhigh`. The actual task stopped before provider work at `runner_ingress_auth_failed`, with zero authenticated connections and zero source events. A separate environment-qualification handshake reached challenge/response; that does not prove task authentication. The three-turn warm reuse, configuration retirement, useful output and reload workflow remains unverified, so this PR stays draft. - The live attempt used the cached Linux amd64 Runner binary `7437a4e32a099180889be534fcfb2a086046514490c1448aba46c55848983ed8` from the immutable Daytona image. The current staging binary could not be copied because SSH timed out, and the configured current EXE image returned registry 403. No compatible current local or CI-downloadable artifact was available. Earlier setup failures and cleanup receipts are retained; no provider success is claimed. Disposable test resources were cleaned up. - Exact-head Greptile completed at 5/5 with no open feedback: https://github.com/paperclipai/paperclip/pull/18011#issuecomment-6469515952. All current-head CI checks pass: 53 successful checks and two intended Storybook skips. ## Risks The preparation fallback affects maintenance checkpoints only after an accepted result. An existing native coordinator prevents that fallback. Stale controller generations, expired or foreign preparation leases, wrong tasks and new provider work remain denied. The bootstrap check retains the live controller object and reads current state each time. It does not trust a marker or restore local state beneath a live controller. Pending turns, prior events/connections, backups and existing remote directories retain recovery gates. No schema, provider binary, credential, permission or lifecycle-policy change is required. ## Model Used OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and collaborating agents. This session does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-28 09:09:09 -05:00
DottaandPaperclip	09ad9adce2	feat(server): adopt running remote agents through durable controller checkpoints (#17439 ) ## Thinking Path > - Paperclip must keep admitted remote work alive while its main server is replaced. > - Runner owns recovery mechanics and exposes opaque checkpoints through #17097. > - The server owns company scope, encryption, Stop, tool authority and execution ownership. > - A successor must prove current ownership before it adopts the original execution. > - Checkpoint bursts must also leave capacity for normal requests and lease renewal. > - This PR persists and adopts that authority through the existing runtime boundaries. ## Linked Issues or Issue Description What existing behavior does this improve? Remote agents retain the original execution during a bounded main-server outage. Current behavior A server restart can lose the controller's local state and connection. An unsafe replacement could repeat a turn or an uncertain mutation. Checkpoint writes in the request pool can also delay ordinary requests during an aligned burst. Proposed behavior Restore encrypted control records from Postgres and adopt the original runner under a fenced lease. Recover known tool receipts through current tool authority. Give control checkpoints a shared pool capped at two connections per database owner. Depends on #17097. Successors are #17455 and #17456. Qualification and presentation remain separate review slices. ## What Changed - Add company/run-bound encrypted checkpoint records and migration 0394. Keep the SQL idempotent after renumbering against current master. - Recover startup and expired ownership through native runtime and heartbeat. Preserve the sandbox lease, admitted runner artifact, harness configuration, native session and active turn. - Keep Stop, finalization, preparation ownership and stale-writer fencing transactional. Preserve graceful detach and natural crash takeover. - Recover committed tool receipts through the existing authority. Reauthorize reads and hold uncertain mutations. - Bound startup retention with the existing coordinator and preserve zero idle scans. - Share at most two checkpoint connections per database client. Keep application and liveness capacity separate. Close the checkpoint pool during shutdown; late writes cannot reopen it. - Count SQL across both application and checkpoint clients. Prove request/liveness progress, command recovery and failure after pool shutdown with real Postgres. ## Verification Project creation recovery now uses the existing project activity receipt owner. It retains authenticated input normalization, fingerprint checks and current run authority after the receipt lock. An absent receipt returns no result before mutation. The new lost-HTTP-response and no-effect regression uses real API routes and Postgres. The seven transport cases pass locally. This Mac skips all 20 database cases. On current source `5de803c2cd`, Linux [job 122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890) executes all 20 project-tool cases successfully (9.480 seconds), including the committed-project lost-response regression. Its shard passes 1,627 cases with three unrelated skips. Server typecheck passes. The transport regression initially has two failures; a subsequent incorrect expectation about schema defaults was corrected before the final pass. Ordinary execution adds no checkpoint writes or recurring work. Current candidate `5de803c2cd` includes the project-creation receipt recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior idempotent checkpoint DDL. Every master SQL file and journal entry is preserved. Snapshots 0390–0394, migration checks and server typecheck pass. The ordinary warm-turn regression fails on its second turn before `a219ab51ad`. After correction, three completed warm turns select the correct prior checkpoint and preserve the original session. All 288 provider-wiring cases pass. The control plane, external persistence callback, identity rotation and filesystem are real; the backend and database-load seam are scripted. The 13 warm admission/configuration variants also pass after final master integration. This does not qualify an underlying harness. At current source `5de803c2cd`, Linux [job 122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907) passes all 940 native-session-executor cases (61.431 seconds), including the warm external-checkpoint behavior; the shard passes 2,358 cases with one skip. Current [Runner job 122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475) passes the native recovery lane: 115 passes and five container-only skips, including the encrypted restore guard, fresh preparation (63.189 seconds), blocked claim lock (1.226 seconds) and recovery beyond 101 blocked owners (3.897 seconds). The first lane passes 2,215 cases with one skip. The SQL cost test reports zero idle and streaming SQL for both output volumes; this is component evidence, not a production capacity claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes all 938 native-executor cases, including the three-turn external-checkpoint regression. Job 122549306469 passes the native recovery lane with 115 passes and five container-only skips, including the encrypted-restore session guard, fresh preparation (63.664 seconds), claim-lock contention (3.961 seconds), and 101-owner recovery (3.824 seconds). This Mac skips the real-Postgres suite. The earlier full local executor attempt has 921 passes and 16 failures during embedded-Postgres initialization before assertions. The original fixture collisions and their corrections remain recorded. At earlier candidate `8fbe62b41d`, server shard job 122535673841 has 3,876 passes, one failure and one skip in an unchanged expiry-worker retry/drain timing test. The same server/database tree on qualification source `b64cac6951` passes all 20 expiry cases and all 3,877 shard cases. The single diagnostic retry, job 122541891915, passes all 20 expiry cases and all 3,877 shard cases (one skip). No assertion was weakened. The original failure remains recorded and its exact timing race is not proven. All Actions checks and legacy status passed on `8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5. Its Paperclip Review remained queued before the new master conflict required integration. The new candidate needs fresh exact-head CI and reviews; old checks do not qualify it. Migration generation is checkpoint-only, all master SQL and journal entries remain unchanged, and migration checks, server typecheck and all 13 warm-owner variants pass after integration. The first migration check was blocked by TSX sandbox IPC; its permitted repeat passes. The first warm-test command used an incorrect path and ran no tests; the corrected command supplies the 13-case receipt. All 16 database pool-lifetime/options/teardown cases pass after granting loopback permission; the first batch had two EPERM failures and 14 passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch, zero-streaming-write and idle-SQL receipts remain source-attributed in the [qualification record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md). That record retains earlier assertion, setup and cleanup failures and their corrections. No failed gate is waived. The historical 300-session component campaign passes four after phases: steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s; aligned-burst read p99 is 103–105 ms after the bounded pool, versus 584–689 ms before. This comparison isolates the persistence corrections. It does not measure the final stack's hosted active-agent capacity. The hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but both arms fail absolute latency gates and candidate peak cgroup memory fails its gate. The active-agent/viewer matrix, different-artifact upgrade and rollback remain open. Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`. Previous head `f2db8a6568` had one failed workspace setup gate: [job 122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441) passes all 4,169 UI tests and 285 CLI cases (one skip), but the company import/export suite reaches its 30-second health deadline during doctor startup. Its output does not reach the server-start step. The CLI, server, database, shared, adapter and plugin trees are identical on qualification source `a866619d7b`, whose [job 122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663) passes that fixture and all 286 CLI cases. The one diagnostic retry, [job 122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225), passes the formerly failing fixture and all 286 CLI cases. All Actions checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile completed at 5/5 with no open feedback. The current integrated head requires fresh checks and a completed Paperclip Review before merge. The original failure remains recorded; its exact timing cause is unproven. A repeated failure requires investigation and repair rather than another retry. All current-head CI, legacy status, exact-head completed Greptile 5/5, resolved feedback and ordinary GitHub merge eligibility remain required before merge. The latest integration preserves master’s published Copilot v70 profiles and every published migration. Checkpoint migration `0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and retains the prior idempotent DDL. The Runner vendor rebuild and replay parity checks pass. Server typecheck and all 13 warm-owner variants pass after that rebuild. The imported report-lock patch is byte-identical to master; its intentional patch-context whitespace is retained, and the feature delta passes whitespace checks. No hosted experiment was run. ## Risks The pool adds up to two active connections per instance, shared across agents. Checkpoint delivery can wait behind that bound; it still requires its commit. Steady component-test Postgres CPU rises by about 1.7 percentage points of a two-CPU allocation. These corrections do not remove the stack's control and renewal writes. The migration and key must survive replacement. Fresh-database tests and local load do not establish different-artifact hosted upgrade or rollback continuity. ## Model Used OpenAI GPT-6 through Codex, with reasoning, repository tools and code execution. The runtime does not expose a more specific model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [ ] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>	2026-10-27 21:39:23 -05:00

Author

SHA1

Message

Date

DottaandPaperclip

567c043b17

feat(server): expose bounded hot-upgrade readiness (#18037 )

## Thinking Path

> - Paperclip coordinates agent work and keeps its authority in the
server.
> - Remote native runners can preserve execution through a server
replacement.
> - Deployment operators need evidence that current runs support that
recovery.
> - Process-local drain counters do not establish this fact.
> - This PR adds an optional, operator-only readiness observation from
existing records.
> - A deployment consumer can use that report while the existing runtime
retains recovery authority.

## Linked Issues or Issue Description

Refs #17439.

**What happened?**

A deployment cannot distinguish recoverable remote native executions
from runs that require a task drain. The existing status endpoint
reports only process-local counts.

**Expected behavior**

An instance operator can request a bounded readiness report without
stopping admission or exposing checkpoint material. Missing ownership or
persistence evidence must prevent readiness. Ordinary status reads must
keep their current cost.

**Steps to reproduce**

Run a remote native task. Read the task-drain status. It reports active
work but supplies no recovery eligibility. Deployment code therefore has
no affirmative signal to select an active-run handover.

## What Changed

- Add `GET /api/instance/task-drain?upgradeReadiness=1` for instance
administrators and existing Cloud control callers.
- Return aggregate eligibility from current run identities, controller
leases and bound opaque checkpoint headers. Unknown work, pending
admission and incomplete reads prevent a positive report.
- Bound the explicit read to 1,001 rows, encrypted headers of at most
256 KiB each and 8 MiB total, and a two-second SQL statement timeout in
a read-only transaction. Keep ordinary status reads in memory.
- Add PostgreSQL, count-transition and route authorization tests. Update
the API contract, operator docs and access maps.

## Verification

- All seven readiness cases pass against an isolated PostgreSQL 18
fixture. They cover missing checkpoints, legacy adapters, missing
process/session identities, foreign owners, expired leases, missing
lease tokens, failure state and in-flight work. The positive SQL fixture
uses the real Runner checkpoint writer and encryption. Regular sandbox
workspace identity and plugin environment-lease identity are tested with
independent IDs. Stopped/released/undispatched plugin tasks and every
mismatched identity field are rejected. Local/legacy/malformed/oversized
or mismatched headers are rejected. A held PostgreSQL table lock proves
the two-second cancellation budget, rollback/connection release and
unrelated query responsiveness.
- All 123 focused readiness, instance-settings and Cloud control
assertion tests pass, including operator-only access and no readiness
read on ordinary status requests. Runner’s 168 checkpoint tests pass.
Runner and server typechecks pass after review fixes.
- Workspace typecheck, full build, module-boundary and map checks passed
locally before the master rebase. Current-head GitHub CI is now green,
including typecheck, build, all grouped/serialized server and Runner
tests, browser shards, policy and review-dispatch checks. Greptile
completed cleanly at 5/5 for `f15fb0c7e9dfb975090cd5d791a1c8e6701275f8`;
all actionable threads are resolved.
- The broad local test run was stopped after the macOS embedded
PostgreSQL fixture failed to bootstrap. The SQL test was then run on the
isolated PostgreSQL fixture. No assertion or timeout was weakened. One
unrelated existing route test exceeded its 15-second limit under
concurrent local builds; it passed on one isolated retry and the
subsequent whole route suite passed.
- Hosted active-run upgrade acceptance has not run. This PR supplies an
observation; it does not change rollout policy or establish live
continuity by itself.

## Risks

This is an instance-wide observation, so company members and agent keys
must not receive it. A readiness snapshot can age as work changes;
shutdown and startup retain their existing execution authority and
fencing. The report does not validate a replacement artifact or
authorize a schema migration. It decrypts only bounded headers through
the checkpoint owner and delegates format/identity inspection to Runner;
full journal restoration and physical sandbox authentication remain with
recovery. No migration, recurring query, new checkpoint write or wire
protocol change is included. The additive Runner header inspection
contract reuses its existing parser.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, source inspection, tool
use and code execution. The host does not expose the exact serving model
ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass for the affected contracts;
the broad-run limitation is recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-28 11:28:15 -05:00

DottaandPaperclip

25f1070fbd

fix(runner): preserve checkpoint proof across warm preparation (#18011 )

## Thinking Path

> - Paperclip coordinates agent work and preserves its execution state.
> - Native Runner checkpoints preserve authority across controller
changes.
> - Warm sessions also need checkpoint writes while their successor
prepares.
> - External persistence replaces the local state file with a marker.
> - Two existing checks treated those valid states as lost ownership or
prior activity.
> - This PR uses the existing preparation lease and canonical controller
state to keep those checks accurate.
> - The result is safe warm-session retirement and fresh-session startup
on reused sandboxes.

## Linked Issues or Issue Description

Related: #17439 and #17320. The open #17401 changes the recovery welcome
window and does not cover these paths.

**What happened?**

A configuration change could fail before the next provider turn with
`runner did not durably suspend before checkpoint`. The task already
pointed to its preparing successor, but that successor did not yet have
a native coordinator. The old completed runner's maintenance checkpoint
was rejected.

A separate fresh session on a reused sandbox could fail with
`resumed_unavailable`. Checkpoint registration had replaced the full
local state file with an external-persistence marker. The bootstrap
check still required the old full-file format. Process supervision then
reported unconfirmed termination before launch.

**Expected behavior**

Retire the completed warm runner under the same process's live
preparation lease. Admit a fresh remote session only when the registered
controller proves exact identity, no prior activity and pending
bootstrap commands, and the remote session directory can be claimed
atomically.

**Steps to reproduce**

1. Complete a native warm turn with external checkpoint persistence.
2. Change configuration so instruction preparation must retire that
session, then submit a follow-up.
3. Separately, admit a fresh native session on a reused sandbox lease
after external checkpoint registration.

**Paperclip version or commit**

Reproduced in the checkpoint implementation introduced by #17439. Fixed
against current master.

**Deployment mode**

Source checkout tests and a hosted Docker/Daytona incident. No
deployment or original-task retry is part of this PR.

## What Changed

- Accept completed-run maintenance under the exact successor's live,
same-process preparation lease only before a native coordinator exists.
- Preserve original controller generation, accepted-result, no-new-work,
company and task checks.
- Pass the registered controller through the existing connection
callback and read its current canonical state for untouched-session
proof.
- Keep marker-only, identity, activity, backup and atomic
remote-directory guards.
- Add regressions and update execution semantics and session
verification recipes.

## Verification

- Warm retirement red: the real Postgres writer rejected the preparing
successor's `runner.suspend` checkpoint with
`native_runner_checkpoint_owner_lost`.
- Warm retirement green: 14 Postgres ownership cases pass, including
stale/foreign/expired owners, conflicting native authority and new
provider work.
- Bootstrap red: real controller registration and external persistence
caused the resumed and replacement positives to fail. Six external
negative cases passed.
- Bootstrap green: all 34 bootstrap cases, the bootstrap-state predicate
and 13 connection-driver cases pass (48 total).
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Full `pnpm -r typecheck` and `pnpm build` pass.
- Full local `pnpm test:run` remains partial/interrupted. The default
attempt could not initialize embedded Postgres on this Mac. A dedicated
Postgres 18 attempt ran for 17 minutes without a completed file result
before interruption; the generic helper separately confirmed unsupported
embedded Postgres (`initdb` failed after five attempts) and ignores the
external database URL. The generic probe skipped one test. Setup and
interrupted-run logs are retained; neither is claimed as a behavioral
red or a full local pass.
- Real source-hosted Codex/Daytona acceptance was attempted at
`6ce7fcacc4` with `gpt-6.1-sol`, `xhigh`. The actual task stopped before
provider work at `runner_ingress_auth_failed`, with zero authenticated
connections and zero source events. A separate environment-qualification
handshake reached challenge/response; that does not prove task
authentication. The three-turn warm reuse, configuration retirement,
useful output and reload workflow remains unverified, so this PR stays
draft.
- The live attempt used the cached Linux amd64 Runner binary
`7437a4e32a099180889be534fcfb2a086046514490c1448aba46c55848983ed8` from
the immutable Daytona image. The current staging binary could not be
copied because SSH timed out, and the configured current EXE image
returned registry 403. No compatible current local or CI-downloadable
artifact was available. Earlier setup failures and cleanup receipts are
retained; no provider success is claimed. Disposable test resources were
cleaned up.
- Exact-head Greptile completed at 5/5 with no open feedback:
https://github.com/paperclipai/paperclip/pull/18011#issuecomment-6469515952.
All current-head CI checks pass: 53 successful checks and two intended
Storybook skips.

## Risks

The preparation fallback affects maintenance checkpoints only after an
accepted result. An existing native coordinator prevents that fallback.
Stale controller generations, expired or foreign preparation leases,
wrong tasks and new provider work remain denied.

The bootstrap check retains the live controller object and reads current
state each time. It does not trust a marker or restore local state
beneath a live controller. Pending turns, prior events/connections,
backups and existing remote directories retain recovery gates. No
schema, provider binary, credential, permission or lifecycle-policy
change is required.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use, code execution and
collaborating agents. This session does not expose a more specific model
ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-28 09:09:09 -05:00

DottaandPaperclip

09ad9adce2

feat(server): adopt running remote agents through durable controller checkpoints (#17439 )

## Thinking Path

> - Paperclip must keep admitted remote work alive while its main server
is replaced.
> - Runner owns recovery mechanics and exposes opaque checkpoints
through #17097.
> - The server owns company scope, encryption, Stop, tool authority and
execution ownership.
> - A successor must prove current ownership before it adopts the
original execution.
> - Checkpoint bursts must also leave capacity for normal requests and
lease renewal.
> - This PR persists and adopts that authority through the existing
runtime boundaries.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Remote agents retain the original execution during a bounded main-server
outage.

**Current behavior**

A server restart can lose the controller's local state and connection.
An unsafe replacement could repeat a turn or an uncertain mutation.
Checkpoint writes in the request pool can also delay ordinary requests
during an aligned burst.

**Proposed behavior**

Restore encrypted control records from Postgres and adopt the original
runner under a fenced lease. Recover known tool receipts through current
tool authority. Give control checkpoints a shared pool capped at two
connections per database owner.

Depends on #17097. Successors are #17455 and #17456. Qualification and
presentation remain separate review slices.

## What Changed

- Add company/run-bound encrypted checkpoint records and migration 0394.
Keep the SQL idempotent after renumbering against current master.
- Recover startup and expired ownership through native runtime and
heartbeat. Preserve the sandbox lease, admitted runner artifact, harness
configuration, native session and active turn.
- Keep Stop, finalization, preparation ownership and stale-writer
fencing transactional. Preserve graceful detach and natural crash
takeover.
- Recover committed tool receipts through the existing authority.
Reauthorize reads and hold uncertain mutations.
- Bound startup retention with the existing coordinator and preserve
zero idle scans.
- Share at most two checkpoint connections per database client. Keep
application and liveness capacity separate. Close the checkpoint pool
during shutdown; late writes cannot reopen it.
- Count SQL across both application and checkpoint clients. Prove
request/liveness progress, command recovery and failure after pool
shutdown with real Postgres.

## Verification

Project creation recovery now uses the existing project activity receipt
owner. It retains authenticated input normalization, fingerprint checks
and current run authority after the receipt lock. An absent receipt
returns no result before mutation. The new lost-HTTP-response and
no-effect regression uses real API routes and Postgres. The seven
transport cases pass locally. This Mac skips all 20 database cases. On
current source `5de803c2cd`, Linux [job
122593415890](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415890)
executes all 20 project-tool cases successfully (9.480 seconds),
including the committed-project lost-response regression. Its shard
passes 1,627 cases with three unrelated skips. Server typecheck passes.
The transport regression initially has two failures; a subsequent
incorrect expectation about schema defaults was corrected before the
final pass. Ordinary execution adds no checkpoint writes or recurring
work.

Current candidate `5de803c2cd` includes the project-creation receipt
recovery fix and master `2cd747d1b6`. Migration 0394 retains the prior
idempotent checkpoint DDL. Every master SQL file and journal entry is
preserved. Snapshots 0390–0394, migration checks and server typecheck
pass.

The ordinary warm-turn regression fails on its second turn before
`a219ab51ad`. After correction, three completed warm turns select the
correct prior checkpoint and preserve the original session. All 288
provider-wiring cases pass. The control plane, external persistence
callback, identity rotation and filesystem are real; the backend and
database-load seam are scripted. The 13 warm admission/configuration
variants also pass after final master integration. This does not qualify
an underlying harness.

At current source `5de803c2cd`, Linux [job
122593415907](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415907)
passes all 940 native-session-executor cases (61.431 seconds), including
the warm external-checkpoint behavior; the shard passes 2,358 cases with
one skip. Current [Runner job
122593415475](https://github.com/paperclipai/paperclip/actions/runs/40620576702/job/122593415475)
passes the native recovery lane: 115 passes and five container-only
skips, including the encrypted restore guard, fresh preparation (63.189
seconds), blocked claim lock (1.226 seconds) and recovery beyond 101
blocked owners (3.897 seconds). The first lane passes 2,215 cases with
one skip. The SQL cost test reports zero idle and streaming SQL for both
output volumes; this is component evidence, not a production capacity
claim. At prior candidate `1ff7bb44f4`, Linux job 122549307215 passes
all 938 native-executor cases, including the three-turn
external-checkpoint regression. Job 122549306469 passes the native
recovery lane with 115 passes and five container-only skips, including
the encrypted-restore session guard, fresh preparation (63.664 seconds),
claim-lock contention (3.961 seconds), and 101-owner recovery (3.824
seconds). This Mac skips the real-Postgres suite. The earlier full local
executor attempt has 921 passes and 16 failures during embedded-Postgres
initialization before assertions. The original fixture collisions and
their corrections remain recorded.

At earlier candidate `8fbe62b41d`, server shard job 122535673841 has
3,876 passes, one failure and one skip in an unchanged expiry-worker
retry/drain timing test. The same server/database tree on qualification
source `b64cac6951` passes all 20 expiry cases and all 3,877 shard
cases. The single diagnostic retry, job 122541891915, passes all 20
expiry cases and all 3,877 shard cases (one skip). No assertion was
weakened. The original failure remains recorded and its exact timing
race is not proven. All Actions checks and legacy status passed on
`8fbe62b41d`, with completed Greptile 5/5 and no open feedback. The next
candidate `1ff7bb44f4` also passed all Actions checks and Greptile 5/5.
Its Paperclip Review remained queued before the new master conflict
required integration. The new candidate needs fresh exact-head CI and
reviews; old checks do not qualify it. Migration generation is
checkpoint-only, all master SQL and journal entries remain unchanged,
and migration checks, server typecheck and all 13 warm-owner variants
pass after integration. The first migration check was blocked by TSX
sandbox IPC; its permitted repeat passes. The first warm-test command
used an incorrect path and ran no tests; the corrected command supplies
the 13-case receipt.

All 16 database pool-lifetime/options/teardown cases pass after granting
loopback permission; the first batch had two EPERM failures and 14
passes. Earlier semantic-tool/HTTP recovery, adoption, encryption-batch,
zero-streaming-write and idle-SQL receipts remain source-attributed in
the [qualification
record](https://github.com/paperclipai/paperclip/blob/2ff66f57da0da35bac1c19a2366ea4b84d87a986/doc/plans/2026-10-22-hot-upgrade-qualification.md).
That record retains earlier assertion, setup and cleanup failures and
their corrections. No failed gate is waived.

The historical 300-session component campaign passes four after phases:
steady read p99 is 107–108 ms at 500 reads/s and 100 writes/s;
aligned-burst read p99 is 103–105 ms after the bounded pool, versus
584–689 ms before. This comparison isolates the persistence corrections.
It does not measure the final stack's hosted active-agent capacity. The
hosted zero-agent API pair leaves SQL and CPU essentially unchanged, but
both arms fail absolute latency gates and candidate peak cgroup memory
fails its gate. The active-agent/viewer matrix, different-artifact
upgrade and rollback remain open.

Run `PAPERCLIP_TEST_DATABASE_URL=<owned-disposable-postgres> pnpm
--filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`.

Previous head `f2db8a6568` had one failed workspace setup gate: [job
122555672441](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122555672441)
passes all 4,169 UI tests and 285 CLI cases (one skip), but the company
import/export suite reaches its 30-second health deadline during doctor
startup. Its output does not reach the server-start step. The CLI,
server, database, shared, adapter and plugin trees are identical on
qualification source `a866619d7b`, whose [job
122555886663](https://github.com/paperclipai/paperclip/actions/runs/40609057937/job/122555886663)
passes that fixture and all 286 CLI cases. The one diagnostic retry,
[job
122561057225](https://github.com/paperclipai/paperclip/actions/runs/40608984238/job/122561057225),
passes the formerly failing fixture and all 286 CLI cases. All Actions
checks and legacy status on `f2db8a6568` pass. Its exact-head Greptile
completed at 5/5 with no open feedback. The current integrated head
requires fresh checks and a completed Paperclip Review before merge. The
original failure remains recorded; its exact timing cause is unproven. A
repeated failure requires investigation and repair rather than another
retry.

All current-head CI, legacy status, exact-head completed Greptile 5/5,
resolved feedback and ordinary GitHub merge eligibility remain required
before merge.

The latest integration preserves master’s published Copilot v70 profiles
and every published migration. Checkpoint migration
`0394_conscious_whizzer.sql` was generated with `pnpm db:generate` and
retains the prior idempotent DDL. The Runner vendor rebuild and replay
parity checks pass. Server typecheck and all 13 warm-owner variants pass
after that rebuild. The imported report-lock patch is byte-identical to
master; its intentional patch-context whitespace is retained, and the
feature delta passes whitespace checks. No hosted experiment was run.

## Risks

The pool adds up to two active connections per instance, shared across
agents. Checkpoint delivery can wait behind that bound; it still
requires its commit. Steady component-test Postgres CPU rises by about
1.7 percentage points of a two-CPU allocation. These corrections do not
remove the stack's control and renewal writes. The migration and key
must survive replacement. Fresh-database tests and local load do not
establish different-artifact hosted upgrade or rollback continuity.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, repository tools and code
execution. The runtime does not expose a more specific model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>

2026-10-27 21:39:23 -05:00