DottaandPaperclip
006d5691a6
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Its web server also owns controller admission and user commands. > - A rolling deployment starts a replacement before the old server exits. > - Shared database state alone does not preserve local agent files, browser chunks, or in-process commands. > - This pull request adds an opt-in ephemeral server contract and a local overlap test. > - The benefit is a web server that can relinquish traffic and local files without losing acknowledged work. ## Linked Issues or Issue Description Refs #18403, #18327, #18335, #18341. **Current behavior** The control-plane web server still has local agent-file authority and local command targets. A new container can miss saved bytes or send a command to the wrong process. An existing browser tab also needs chunks from its original build. **Proposed behavior** Eligible servers share only PostgreSQL, object storage and stable secrets. A replacement waits for readiness. The old server drains admitted requests, keeps owner commands available through handoff, and releases controller leases. Explicit maintenance adopts existing files before the volume is removed. **Reason and benefit** This is the application prerequisite for rolling app-only deployments. Schema migrations and permanent multi-replica operation are outside this change. ## What Changed - Store agent-file manifests in PostgreSQL and immutable bytes in S3. Treat local roots as disposable projections. Preserve existing permission checks and edit conflicts. - Add an explicit migration, eligibility and object-retention command to the production runtime bundle. - Add `/api/ready`, admitted-request drain, browser restart signaling and retained immutable UI chunks. - Carry transaction-aware work hints across controllers during handoff and active execution. Keep listeners closed when idle. - Deliver durable Stop and runtime responses to the execution owner. Recheck request lifecycle and access before response dispatch. - Reject local-only configuration in ephemeral mode. Keep the feature disabled for existing installations by default. - Add an isolated PostgreSQL/S3/proxy/browser deployment harness with independent server homes and retained failure evidence. ## Verification - All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`: workspace typecheck, build, Runner checks, server/general tests and browser shards. Greptile reviewed the runtime changes at 5/5 with all findings resolved; the final commit updates maps and the plan. - Local targeted checks passed: coordinator 27; Stop/recovery 33; PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap Stop 1; restart recovery 75. PostgreSQL command tests simulate the provider edge. - Local browser proof passed 91 reads and 19 saves; reconnect took 1,013ms. The tab opened Audit from S3 after deletion of A's whole home/build. This uses compiled workspace code and local-trusted auth, not released containers or live agents. - Local workspace typecheck/build passed; current server and runtime bundle passed. The earlier broad local suite was stopped after three hours and is not a passing receipt. Its retained failures and isolated reruns are documented in the plan. - Stacked #18526 adds stricter graceful-exit proof, an unready replacement control, live Runner qualification and a Stop handoff correction. Hosted enablement remains gated on those results and Cloud #1096. No hosted stack has been enabled. ## Risks - Apply migration 0422 before enabling the feature. Existing data must pass each owner's migration and eligibility checks. Do not discard a volume based on a single migration page. - Durable manifests remain authoritative after the opt-in is disabled. Rolling back to a binary that ignores them is unsupported. - Cold directory restoration and complete shutdown timing still need qualification before enabling a hosted stack. - Retained UI assets must cover the supported browser and rollback window. Cleanup is explicit; no new idle maintenance polling is introduced. - This change does not claim production zero downtime or authorize a fleet-wide enablement. ## Model Used OpenAI Codex, GPT-6 family. The exact serving model ID and context-window limit are not exposed in this session. Used reasoning, code editing, shell execution and browser automation. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
14 KiB
navigation
| navigation | ||||||
|---|---|---|---|---|---|---|
|
Instance operations
An operator can install and run Paperclip, diagnose the instance, configure supported deployment settings, manage its service, and produce a recoverable database backup.
Feature ID: instance-operations.
Implementation: CLI entry point, instance settings, backup command.
Agent access: companion.
Sub-features
install-update:instance-operations/install-update— inspect install channel/version and apply supported update or rollback operations.server-service:instance-operations/server-service— start/stop the intended instance and inspect its service health.configuration:instance-operations/configuration— configure database, server binding, storage, secrets, and available experimental settings.diagnostics:instance-operations/diagnostics— use doctor/health output to distinguish readiness from a listening port.backup:instance-operations/backup— create and inspect backups and verify restore only into a separate disposable target.runtime-setup:instance-operations/runtime-setup— install verified Pi/Cursor runtime assets, stage pending Antigravity assets on macOS ARM64, or import a release-matched Linux Pi companion into an owned published installation.
How to get to it (user POV)
operator-cli
Use paperclipai install, run, service, doctor, configure, channels, or update --help for the intended operation.
settings-backup
Use currently visible instance settings for experiments/access and paperclipai db:backup --help for one-off backup. Some legacy general-settings URLs redirect.
runtime-provisioning
On an owned published server installation, use paperclipai runtime setup pi, paperclipai runtime setup cursor, paperclipai runtime setup antigravity, or paperclipai runtime import-remote <companion-directory> --sha256 <trusted-companion-json-digest>. Run as the OS user operating that installation; this is separate from source-checkout dependency installation. Antigravity setup stages a pinned patched macOS ARM64 runtime; it remains unavailable for tasks until native acceptance passes.
Driving it with Playwright and the Paperclip CLI
Preconditions: follow the baseline. Use a disposable data directory/service identity. Do not update, restart, or restore over an unrelated live instance as a documentation test.
operator-cli
Automated: doctor, update command, and service manager cover command contracts.
Manual: Inspect the selected installation/channel and configuration, start the isolated instance, confirm health and browser access, then stop that instance. For update changes, use the dry-run/check path first and verify installed version and rollback state on an owned disposable installation.
Routine HTTP monitoring may use GET /api/liveness (or HEAD). It returns only
process startup status and the public build commit without SQL or authentication.
It remains reachable during an idle drain. Use /api/health for explicit database
readiness, signed bootstrap, and authorized diagnostics; liveness does not
establish database health or claim a workspace.
Deployment routing uses public GET /api/ready (or HEAD). It returns 200 only
after startup recovery finishes and 503 while starting, recovering, or draining.
The probe reads in-memory state without SQL or session resolution. Local
readiness tests establish that
boundary; hosted overlap and active-run continuity require separate qualification.
Automated: static UI idle completion covers real aborted file streams, disconnects before file lookup, nested routers, and unfinished parent/fallback writes. Mounted Express applications retain their conservative hold. It does not qualify provider cleanup or a hosted sleep/wake cycle.
Manual: On an otherwise idle disposable instance, close a static UI download served by the admitted application before it finishes. Repeat with a disconnect before headers arrive. Confirm that neither leaves a permanent request hold. An unfinished application write must still prevent sleep after its client leaves.
Automated: historical cleanup exercises the runtime inspection path for failed Cloud sandbox leases, including verification leases without a run. A signed cleanup-only absence observation can remove this blocker without changing failed outcomes or granting run continuation. An unresolved provider resource still prevents idle sleep. Manual: on an owned Cloud staging workspace, confirm the exact published build, reconcile failed test allocations, then verify actual provider suspension and normal hosted entry after wake. Local tests do not prove that hosted journey.
settings-backup
Cloud startup reads nonsecret application choices from a private, versioned
PostgreSQL singleton after a bounded one-time legacy config seed. A fresh
process and different home restore the saved choices without consulting a stale
.env file. Invalid saved data or an unavailable database prevents startup;
saved telemetry opt-out applies before client creation. The
database controls and
startup-order controls
cover these local boundaries. See field ownership.
These controls do not qualify a deployed Cloud rollout or durable file storage.
Automated: experimental settings and database backup routes cover their UI/API contracts.
Cloud migration: in one maintenance window, stop legacy file writers and run the configuration/package qualification command. Inspect its database receipt, update a saved preference, then repeat against a second owned application/image root with the first home unavailable. Use an explicit package evidence digest for same-package comparison. Confirm stale qualification and missing legacy files cannot clear blockers. This recipe covers configuration/packages and does not qualify whole-server zero downtime, secret bytes, or volume removal. Local PostgreSQL evidence path includes a real fixture worker; it does not establish a hosted replacement.
Managed adapters: change a visible adapter preference as an instance admin, then use a second application process against the same disposable database to confirm catalog and new-agent selection reflect it. Verify saved agents remain usable. Restart with an operator adapter declaration, confirm its effective list, and reject conflicting toggles. Remove the declaration and confirm saved preferences return. Database tests cover local persistence, concurrent writers, audit rollback, and failure responses; local compatibility tests cover the legacy file. No deployed walkthrough is claimed.
Manual: Change a harmless experimental setting, reload, and confirm the affected route honors it. Back up a disposable instance, inspect the reported file and metadata, and restore using the documented procedure into a second isolated target. Verify a known record after restoration.
Cloud staging: On a managed *.staging.paperclip.app instance, verify experiments
are visible, Cloud-enabled controls are locked on, and other experiments are
editable by a Cloud stack Admin or instance admin with changes surviving reload.
With an old stored false value and a Cloud true value, verify the effective
setting is true and an API attempt to disable it returns 403 without saving.
Repeat the enabled-feature check on production and managed stacks without a hostname.
As a stack Admin, verify a harmless experiment saves while general instance
settings and task-drain operations remain denied. Demote that user to Member
and verify the next experimental write is denied.
Production keeps its hidden controls and managed feature values. Staging retains
other Cloud restrictions. Settings policy tests,
route tests, and
health tests exercise simulated Cloud
identities and persistence; they do not establish a deployed staging walkthrough.
runtime-provisioning
Automated: Runtime commands, setup, remote Pi companion, and Pi package provisioning cover parsing, fixtures, package identity, and integrity. They do not qualify every published installation or live provider.
Manual: Use a disposable published installation and its exact runtime release. Run setup for the intended host runtime, inspect verified version/path, and repeat to confirm the existing installation is reused. Test a corrupt, linked, or foreign installation and verify rejection preserves unrelated files. Import a matching Linux x64 Pi companion with an independently trusted manifest digest; repeat for verified_existing, then alter manifest, inventory, source/profile identity, or target files and verify failure without replacing unrelated state. Inspect cleanup/cancellation. Finally use the selected adapter/account to run a bounded task separately: installed bytes do not prove model execution.
Sources: Runtime CLI, instance-config bypass, Pi provisioning, Cursor provisioning, and companion importer.
Antigravity: provisioning, closure/integrity regressions, and pending admission tests. Exercise public-layout setup and verify both native files and cleanup; separately verify that configuration/API/driver/sidecar reject the pending provider. These checks do not establish authenticated task execution.
Gotchas
-
Runtime setup operates on a published installation and verifies package assets. It is not a substitute for an account connection, environment qualification, or a useful task run.
-
Source-checkout update/check commands report manual update guidance; they do not qualify a managed installation. Rollback restores retained CLI payloads, not database migrations. Service status applies to its selected service identity, not automatically to an owned foreground dev server.
-
A successful backup write does not prove restoration works.
-
Settings visibility and mutability differ for cloud-managed and self-hosted instances.
-
Keep the three data paths distinct: first-party Telemetry, operator-configured OpenTelemetry, and local run logs.
Storybook verification
backup
Capability ID: instance-operations/backup.
Scope: ui Storybook: Mount shipped InstanceGeneralSettings in CompanySettings with daily/weekly/monthly retention presets, selected values, saving/error/loading and cloud-hidden controls. Backup creation and restore remain separate CLI/operator checks. Production: PATCH /api/instance/settings/general must enforce instance settings authority and managed hidden fields, persist retention and survive reload. Separately inspect a real backup and restore a known record into a second disposable target; a successful write does not prove restoration. Sources: Source: InstanceGeneralSettings.tsx, CompanySettings.tsx, instance-settings.ts, db-backup.ts. Tests: InstanceGeneralSettings.test.tsx, instance-settings-routes.test.ts, instance-settings-managed-overlay.test.ts, instance-database-backups-routes.test.ts
install-update
Capability ID: instance-operations/install-update.
Scope: cli Storybook: No authored story required for install, channel selection, update or rollback command execution. Production: Use an owned managed installation: check/dry-run, installed version, retained payload rollback and failure. Source-checkout guidance is a separate case; rollback does not undo database migrations. Sources: Source: index.ts, install.ts, update.ts. Tests: install-command.test.ts, update-command.test.ts
diagnostics
Capability ID: instance-operations/diagnostics.
Scope: cli
Storybook: No authored story required for doctor/health command results.
Production: Run doctor against an owned data directory; check readiness and nonzero failures. Instance operators can inspect GET /api/instance/task-drain?upgradeReadiness=1 before a compatible app replacement. Verify that unknown or unsupported executions prevent readiness and that the read does not hold admission; separately prove same-run continuity through an actual upgrade. A listening port alone does not prove readiness; repair requires observing the repaired result.
Sources: Source: doctor.ts. Tests: doctor.test.ts