Files
PaperClipAI/server
DottaandPaperclip 567c043b17 feat(server): expose bounded hot-upgrade readiness (#18037)
## Thinking Path

> - Paperclip coordinates agent work and keeps its authority in the
server.
> - Remote native runners can preserve execution through a server
replacement.
> - Deployment operators need evidence that current runs support that
recovery.
> - Process-local drain counters do not establish this fact.
> - This PR adds an optional, operator-only readiness observation from
existing records.
> - A deployment consumer can use that report while the existing runtime
retains recovery authority.

## Linked Issues or Issue Description

Refs #17439.

**What happened?**

A deployment cannot distinguish recoverable remote native executions
from runs that require a task drain. The existing status endpoint
reports only process-local counts.

**Expected behavior**

An instance operator can request a bounded readiness report without
stopping admission or exposing checkpoint material. Missing ownership or
persistence evidence must prevent readiness. Ordinary status reads must
keep their current cost.

**Steps to reproduce**

Run a remote native task. Read the task-drain status. It reports active
work but supplies no recovery eligibility. Deployment code therefore has
no affirmative signal to select an active-run handover.

## What Changed

- Add `GET /api/instance/task-drain?upgradeReadiness=1` for instance
administrators and existing Cloud control callers.
- Return aggregate eligibility from current run identities, controller
leases and bound opaque checkpoint headers. Unknown work, pending
admission and incomplete reads prevent a positive report.
- Bound the explicit read to 1,001 rows, encrypted headers of at most
256 KiB each and 8 MiB total, and a two-second SQL statement timeout in
a read-only transaction. Keep ordinary status reads in memory.
- Add PostgreSQL, count-transition and route authorization tests. Update
the API contract, operator docs and access maps.

## Verification

- All seven readiness cases pass against an isolated PostgreSQL 18
fixture. They cover missing checkpoints, legacy adapters, missing
process/session identities, foreign owners, expired leases, missing
lease tokens, failure state and in-flight work. The positive SQL fixture
uses the real Runner checkpoint writer and encryption. Regular sandbox
workspace identity and plugin environment-lease identity are tested with
independent IDs. Stopped/released/undispatched plugin tasks and every
mismatched identity field are rejected. Local/legacy/malformed/oversized
or mismatched headers are rejected. A held PostgreSQL table lock proves
the two-second cancellation budget, rollback/connection release and
unrelated query responsiveness.
- All 123 focused readiness, instance-settings and Cloud control
assertion tests pass, including operator-only access and no readiness
read on ordinary status requests. Runner’s 168 checkpoint tests pass.
Runner and server typechecks pass after review fixes.
- Workspace typecheck, full build, module-boundary and map checks passed
locally before the master rebase. Current-head GitHub CI is now green,
including typecheck, build, all grouped/serialized server and Runner
tests, browser shards, policy and review-dispatch checks. Greptile
completed cleanly at 5/5 for `f15fb0c7e9dfb975090cd5d791a1c8e6701275f8`;
all actionable threads are resolved.
- The broad local test run was stopped after the macOS embedded
PostgreSQL fixture failed to bootstrap. The SQL test was then run on the
isolated PostgreSQL fixture. No assertion or timeout was weakened. One
unrelated existing route test exceeded its 15-second limit under
concurrent local builds; it passed on one isolated retry and the
subsequent whole route suite passed.
- Hosted active-run upgrade acceptance has not run. This PR supplies an
observation; it does not change rollout policy or establish live
continuity by itself.

## Risks

This is an instance-wide observation, so company members and agent keys
must not receive it. A readiness snapshot can age as work changes;
shutdown and startup retain their existing execution authority and
fencing. The report does not validate a replacement artifact or
authorize a schema migration. It decrypts only bounded headers through
the checkpoint owner and delegates format/identity inspection to Runner;
full journal restoration and physical sandbox authentication remain with
recovery. No migration, recurring query, new checkpoint write or wire
protocol change is included. The additive Runner header inspection
contract reuses its existing parser.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, source inspection, tool
use and code execution. The host does not expose the exact serving model
ID or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass for the affected contracts;
the broad-run limitation is recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:28:15 -05:00
..