DottaandPaperclip
567c043b17
## Thinking Path > - Paperclip coordinates agent work and keeps its authority in the server. > - Remote native runners can preserve execution through a server replacement. > - Deployment operators need evidence that current runs support that recovery. > - Process-local drain counters do not establish this fact. > - This PR adds an optional, operator-only readiness observation from existing records. > - A deployment consumer can use that report while the existing runtime retains recovery authority. ## Linked Issues or Issue Description Refs #17439. **What happened?** A deployment cannot distinguish recoverable remote native executions from runs that require a task drain. The existing status endpoint reports only process-local counts. **Expected behavior** An instance operator can request a bounded readiness report without stopping admission or exposing checkpoint material. Missing ownership or persistence evidence must prevent readiness. Ordinary status reads must keep their current cost. **Steps to reproduce** Run a remote native task. Read the task-drain status. It reports active work but supplies no recovery eligibility. Deployment code therefore has no affirmative signal to select an active-run handover. ## What Changed - Add `GET /api/instance/task-drain?upgradeReadiness=1` for instance administrators and existing Cloud control callers. - Return aggregate eligibility from current run identities, controller leases and bound opaque checkpoint headers. Unknown work, pending admission and incomplete reads prevent a positive report. - Bound the explicit read to 1,001 rows, encrypted headers of at most 256 KiB each and 8 MiB total, and a two-second SQL statement timeout in a read-only transaction. Keep ordinary status reads in memory. - Add PostgreSQL, count-transition and route authorization tests. Update the API contract, operator docs and access maps. ## Verification - All seven readiness cases pass against an isolated PostgreSQL 18 fixture. They cover missing checkpoints, legacy adapters, missing process/session identities, foreign owners, expired leases, missing lease tokens, failure state and in-flight work. The positive SQL fixture uses the real Runner checkpoint writer and encryption. Regular sandbox workspace identity and plugin environment-lease identity are tested with independent IDs. Stopped/released/undispatched plugin tasks and every mismatched identity field are rejected. Local/legacy/malformed/oversized or mismatched headers are rejected. A held PostgreSQL table lock proves the two-second cancellation budget, rollback/connection release and unrelated query responsiveness. - All 123 focused readiness, instance-settings and Cloud control assertion tests pass, including operator-only access and no readiness read on ordinary status requests. Runner’s 168 checkpoint tests pass. Runner and server typechecks pass after review fixes. - Workspace typecheck, full build, module-boundary and map checks passed locally before the master rebase. Current-head GitHub CI is now green, including typecheck, build, all grouped/serialized server and Runner tests, browser shards, policy and review-dispatch checks. Greptile completed cleanly at 5/5 for `f15fb0c7e9dfb975090cd5d791a1c8e6701275f8`; all actionable threads are resolved. - The broad local test run was stopped after the macOS embedded PostgreSQL fixture failed to bootstrap. The SQL test was then run on the isolated PostgreSQL fixture. No assertion or timeout was weakened. One unrelated existing route test exceeded its 15-second limit under concurrent local builds; it passed on one isolated retry and the subsequent whole route suite passed. - Hosted active-run upgrade acceptance has not run. This PR supplies an observation; it does not change rollout policy or establish live continuity by itself. ## Risks This is an instance-wide observation, so company members and agent keys must not receive it. A readiness snapshot can age as work changes; shutdown and startup retain their existing execution authority and fencing. The report does not validate a replacement artifact or authorize a schema migration. It decrypts only bounded headers through the checkpoint owner and delegates format/identity inspection to Runner; full journal restoration and physical sandbox authentication remain with recovery. No migration, recurring query, new checkpoint write or wire protocol change is included. The additive Runner header inspection contract reuses its existing parser. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, source inspection, tool use and code execution. The host does not expose the exact serving model ID or context-window size. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass for the affected contracts; the broad-run limitation is recorded above - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
11 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Instance operations
An operator can install and run Paperclip, diagnose the instance, configure supported deployment settings, manage its service, and produce a recoverable database backup.
Feature ID: instance-operations.
Implementation: CLI entry point, instance settings, backup command.
Agent access: companion.
Sub-features
install-update:instance-operations/install-update— inspect install channel/version and apply supported update or rollback operations.server-service:instance-operations/server-service— start/stop the intended instance and inspect its service health.configuration:instance-operations/configuration— configure database, server binding, storage, secrets, and available experimental settings.diagnostics:instance-operations/diagnostics— use doctor/health output to distinguish readiness from a listening port.backup:instance-operations/backup— create and inspect backups and verify restore only into a separate disposable target.runtime-setup:instance-operations/runtime-setup— install verified Pi/Cursor runtime assets or import a release-matched Linux Pi companion into an owned published installation.
How to get to it (user POV)
operator-cli
Use paperclipai install, run, service, doctor, configure, channels, or update --help for the intended operation.
settings-backup
Use currently visible instance settings for experiments/access and paperclipai db:backup --help for one-off backup. Some legacy general-settings URLs redirect.
runtime-provisioning
On an owned published server installation, use paperclipai runtime setup pi, paperclipai runtime setup cursor, or paperclipai runtime import-remote <companion-directory> --sha256 <trusted-companion-json-digest>. Run as the OS user operating that installation; this is separate from source-checkout dependency installation.
Driving it with Playwright and the Paperclip CLI
Preconditions: follow the baseline. Use a disposable data directory/service identity. Do not update, restart, or restore over an unrelated live instance as a documentation test.
operator-cli
Automated: doctor, update command, and service manager cover command contracts.
Manual: Inspect the selected installation/channel and configuration, start the isolated instance, confirm health and browser access, then stop that instance. For update changes, use the dry-run/check path first and verify installed version and rollback state on an owned disposable installation.
Routine HTTP monitoring may use GET /api/liveness (or HEAD). It returns only
process startup status and the public build commit without SQL or authentication.
It remains reachable during an idle drain. Use /api/health for explicit database
readiness, signed bootstrap, and authorized diagnostics; liveness does not
establish database health or claim a workspace.
Automated: static UI idle completion covers real aborted file streams, disconnects before file lookup, nested routers, and unfinished parent/fallback writes. Mounted Express applications retain their conservative hold. It does not qualify provider cleanup or a hosted sleep/wake cycle.
Manual: On an otherwise idle disposable instance, close a static UI download served by the admitted application before it finishes. Repeat with a disconnect before headers arrive. Confirm that neither leaves a permanent request hold. An unfinished application write must still prevent sleep after its client leaves.
settings-backup
Automated: experimental settings and database backup routes cover their UI/API contracts.
Manual: Change a harmless experimental setting, reload, and confirm the affected route honors it. Back up a disposable instance, inspect the reported file and metadata, and restore using the documented procedure into a second isolated target. Verify a known record after restoration.
Cloud staging: On a managed *.staging.paperclip.app instance, verify experiments
are visible, Cloud-enabled controls are locked on, and other experiments are
editable by a Cloud stack Admin or instance admin with changes surviving reload.
With an old stored false value and a Cloud true value, verify the effective
setting is true and an API attempt to disable it returns 403 without saving.
Repeat the enabled-feature check on production and managed stacks without a hostname.
As a stack Admin, verify a harmless experiment saves while general instance
settings and task-drain operations remain denied. Demote that user to Member
and verify the next experimental write is denied.
Production keeps its hidden controls and managed feature values. Staging retains
other Cloud restrictions. Settings policy tests,
route tests, and
health tests exercise simulated Cloud
identities and persistence; they do not establish a deployed staging walkthrough.
runtime-provisioning
Automated: Runtime commands, setup, remote Pi companion, and Pi package provisioning cover parsing, fixtures, package identity, and integrity. They do not qualify every published installation or live provider.
Manual: Use a disposable published installation and its exact runtime release. Run setup for the intended host runtime, inspect verified version/path, and repeat to confirm the existing installation is reused. Test a corrupt, linked, or foreign installation and verify rejection preserves unrelated files. Import a matching Linux x64 Pi companion with an independently trusted manifest digest; repeat for verified_existing, then alter manifest, inventory, source/profile identity, or target files and verify failure without replacing unrelated state. Inspect cleanup/cancellation. Finally use the selected adapter/account to run a bounded task separately: installed bytes do not prove model execution.
Sources: Runtime CLI, instance-config bypass, Pi provisioning, Cursor provisioning, and companion importer.
Gotchas
-
Runtime setup operates on a published installation and verifies package assets. It is not a substitute for an account connection, environment qualification, or a useful task run.
-
Source-checkout update/check commands report manual update guidance; they do not qualify a managed installation. Rollback restores retained CLI payloads, not database migrations. Service status applies to its selected service identity, not automatically to an owned foreground dev server.
-
A successful backup write does not prove restoration works.
-
Settings visibility and mutability differ for cloud-managed and self-hosted instances.
-
Keep the three data paths distinct: first-party Telemetry, operator-configured OpenTelemetry, and local run logs.
Storybook verification
backup
Capability ID: instance-operations/backup.
Scope: ui Storybook: Mount shipped InstanceGeneralSettings in CompanySettings with daily/weekly/monthly retention presets, selected values, saving/error/loading and cloud-hidden controls. Backup creation and restore remain separate CLI/operator checks. Production: PATCH /api/instance/settings/general must enforce instance settings authority and managed hidden fields, persist retention and survive reload. Separately inspect a real backup and restore a known record into a second disposable target; a successful write does not prove restoration. Sources: Source: InstanceGeneralSettings.tsx, CompanySettings.tsx, instance-settings.ts, db-backup.ts. Tests: InstanceGeneralSettings.test.tsx, instance-settings-routes.test.ts, instance-settings-managed-overlay.test.ts, instance-database-backups-routes.test.ts
install-update
Capability ID: instance-operations/install-update.
Scope: cli Storybook: No authored story required for install, channel selection, update or rollback command execution. Production: Use an owned managed installation: check/dry-run, installed version, retained payload rollback and failure. Source-checkout guidance is a separate case; rollback does not undo database migrations. Sources: Source: index.ts, install.ts, update.ts. Tests: install-command.test.ts, update-command.test.ts
diagnostics
Capability ID: instance-operations/diagnostics.
Scope: cli
Storybook: No authored story required for doctor/health command results.
Production: Run doctor against an owned data directory; check readiness and nonzero failures. Instance operators can inspect GET /api/instance/task-drain?upgradeReadiness=1 before a compatible app replacement. Verify that unknown or unsupported executions prevent readiness and that the read does not hold admission; separately prove same-run continuity through an actual upgrade. A listening port alone does not prove readiness; repair requires observing the repaired result.
Sources: Source: doctor.ts. Tests: doctor.test.ts