Files
PaperClipAI/doc/hot-restart-deploys.md
T
DottaandClaude Fable 5.2 ac114d9479 refactor: cut DEVELOPING.md to a setup and workflow guide (#21670)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `AGENTS.md` tells every contributor and agent to read
`doc/DEVELOPING.md` for setup, commands, testing, and worktrees
> - That file grew to 5,495 lines. Most of it was notes on how one
subsystem behaves, added one pull request at a time
> - Readers could not find the commands. Agents also read behavior notes
as rules
> - This pull request cuts `doc/DEVELOPING.md` to a 660-line setup and
workflow guide. It moves the subsystem notes to focused docs
> - The benefit is a guide that a person or an agent can read in full,
and a clear place for each kind of note

## Linked Issues or Issue Description

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/DEVELOPING.md`

**What's wrong?**

The file was 5,495 lines. It opened with 112 lines about case handoffs
before it reached Prerequisites. Unrelated sections sat below the
"OpenClaw Docker UI One-Command Script" heading. Statements about
current behavior read like policy.

**Suggested fix**

Keep setup, commands, and workflow in `doc/DEVELOPING.md`. Move each
subsystem note to a doc for that subsystem. Add an index.

## What Changed

- `doc/DEVELOPING.md` is now 660 lines: prerequisites, start dev, health
checks, local run, test drive, Docker, instance layout, database,
storage, secrets, worktrees, test commands, Storybook, CLI, and an index
of the other docs.
- 15 new docs hold the moved text without wording changes:
`ci-workflows`, `test-infrastructure`, `worktrees`,
`hot-restart-deploys`, `heartbeat-internals`, `runner-internals`,
`workspace-repositories`, `workspace-storage`,
`skills-and-teams-catalog`, `ui-internals`, `case-handoffs`,
`chat-connector-development`, `instance-configuration`,
`integration-api-notes`, `database-backups`.
- "Publish a branch Storybook" moved to `doc/STORYBOOK-DEPLOYMENT.md`.
The privacy rollout notes moved to `doc/ISSUE-PRIVACY.md`.
- Deleted about 280 lines: the story-by-story Storybook notes, "Agent
Artifact Uploads" (a copy of `doc/AGENT-ARTIFACTS.md`), two pointer-only
sections, "Workspace feature checks (2026-10-09)", and "Chat and task
navigation caching". The test-drive build-lock paragraph is now four
lines.
- Repointed every inbound link and every script or workflow comment that
named a moved section.
- `AGENTS.md` now says where a behavior note goes, so the guide does not
grow again.

## Verification

- A script split the file by line range. It asserts that each of the
5,495 source lines is kept, moved, or dropped exactly one time.
- A link check on every changed Markdown file found 0 missing files and
0 missing anchors.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `node --check` passes on the three scripts with changed comments.
- `pnpm verify` was not run. This worktree has no installed
dependencies, and the change has no runtime code.

## Risks

- Low risk. The change is documentation and comments only.
- A link to a moved section from outside this repository will land on
the top of `doc/DEVELOPING.md`.
- The new docs are not in the audited prose list in
`.github/scripts/ci-inputs.mjs`. An edit to one of them runs full CI. An
edit to `doc/DEVELOPING.md` does not. This PR does not change that list.

## Model Used

- Claude Fable 5.2 (`claude-fable-5-2`), Claude Code desktop, with tool
use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 18:02:27 -06:00

13 KiB

Hot-restart deploys

How a server restart or deploy keeps runs and the UI alive, and how to recover a blocked deploy.

During a restart, the UI's connectivity store (ui/src/lib/connectivity.ts) confirms an outage with a /api/health probe, then polls health with backoff (1s, 2s, 5s, 10s, then every 15s; a longer Retry-After wins). While the server is unreachable, React Query pauses reads and meta.replay = "idempotent" mutations instead of failing them; on recovery they resume and live queries refresh once. Other mutations fail fast with readable copy and are not replayed. Transient errors (network drops, non-JSON responses, 408/425/429/502/503/504, and gateway codes such as tenant_app_unavailable in any status) are classified in ui/src/api/errors.ts.

A new page shows Reconnecting to Paperclip with a Try again action and waits for startup health to become ready. Valid startup metadata remains available to sign-in and invitation pages. An already open page stays mounted during temporary background failures so unsaved edits survive; after about two seconds of continuous trouble, one app-level Connection interrupted banner appears, followed briefly by Back online. Recovery does not reload the browser. Authorization failures still require sign-in or an explicit retry. Storybook App / Connection recovery shows the startup screens and banner.

Writes keep the user's text through an outage:

  • Comments, new tasks, and board sends store the request and its key (clientRequestId or idempotencyKey) before they send (ui/src/lib/pending-send.ts, ui/src/hooks/useDurableSubmit.ts). After a transient failure or a lost response, the composer shows Sending when reconnected… with Resend now and Cancel, and resends with the same key on recovery. The server dedupes by that key. Restored board publications wait for an explicit retry. Restored plain comments and keyed requests resend after their initial backoff. A combined comment that also reassigns, reopens, or changes run settings waits for a matching server receipt or Cancel: comment deduplication alone cannot safely replay those task changes. Cancel returns the text for a fresh, intentional send. A new task restored without its staged files is not sent; the dialog asks for the files again. Restored subtasks keep their saved parent and goal defaults even when reopened from the normal New Task action. Successful staged-file uploads are identified by content digest as well as metadata, so distinct files with identical names, sizes and timestamps are both uploaded. Without browser hashing support, uploads proceed without receipt-based deduplication.
  • Document and inline-editor autosave (ui/src/hooks/useDurableAutosave.ts) keeps unsaved edits in browser storage, shows Will save when reconnected, and saves the newest draft for each task or document on recovery. A 409 whose server copy equals the draft counts as saved. A temporary failure of that verification stays pending. Exhausted attempts show a manual retry action. Admitted edits can finish after route navigation unmounts the editor. That work waits for recovery events while offline, uses bounded backoff while online, and verifies the current account before each retry. The latest admitted save is shared by account and target across editor mounts, so a superseded detached retry cannot overwrite a newer edit after reopening the same task. Replayable task-field updates and interaction resolutions bind the original account, account generation and operation at dispatch. A sign-out or account change cancels that intent before it can send, including after waiting behind another write; callbacks cannot apply an old result to the next account. Stored document drafts and mounted editors belong to the signed-in user. Storage failure shows a warning to keep the tab open or copy the text.
  • Issue field updates go through useUpdateIssueMutation: optimistic, rolled back with readable copy on failure, and replayed on reconnect for absolute fields. Updates run in order, so an older update that retries cannot overwrite a newer one.
  • Uploads and other non-idempotent requests never resend on their own; they keep the input and offer Retry.

The banner counts every write that waits for the connection.

To reproduce a deploy blip in a dev or QA build, set the outage simulator in the browser console and remove it to end the outage:

localStorage.paperclipSimulateOutage = "503:tenant_app_unavailable"    // until removed
localStorage.paperclipSimulateOutage = "503:tenant_app_unavailable@20" // for 20 seconds
localStorage.paperclipSimulateOutage = "network"                       // dropped connection
localStorage.removeItem("paperclipSimulateOutage")

Reads render through useQueryView(query) or <QueryView> (ui/src/components/QueryView.tsx). The view state is one of loading, ready, stale (data plus a transient failure: render the data, no red text), reconnecting (no data yet while the app-wide connectivity state is not online: a quiet placeholder that fills in by itself when the probe loop recovers), or error (a real failure: describeError copy with a Retry button). A transient failure with no data while the server is reachable (a 429, one plugin worker restarting, a 504 on one slow route) is also error once the retry policy gives up, because no recovery loop would refetch it. "Not found" renders only when classifyError says not_found, never for an outage. <QueryErrorState> is the error presentation on its own, in page, panel, and inline sizes. Plugin chrome (usePluginSlots, usePluginLaunchers) is the one exception: it never reports a transient failure and stays collapsed instead.

node scripts/check-query-error-rendering.mjs --summary reports UI code that renders raw query errors ({error.message}, isError ?) or sets retry: false. It is report-only for the long tail; pass --enforce to fail on every finding. Files listed in ENFORCED_FILES in the script have already migrated, and a finding in one of them fails the check in every mode. Add a file to that list when you migrate it. For an intentional exception (a mutation result rendered inline, a prop that is already outage-aware), end the line with // query-error-ok: <reason>. Reads that must fail fast use retry: retryTransientOnly(0) instead of retry: false.

Primary-instance rebuilds that restart paperclip.service can request one-shot live-run adoption instead of using the normal graceful shutdown drain. Before restarting the service, write the marker from the newly staged app with the current service PID:

old_main_pid="$(systemctl show paperclip.service -p MainPID --value)"
pnpm --filter @paperclipai/server exec tsx ../scripts/request-hot-restart.ts --server-pid "$old_main_pid"
systemctl restart paperclip.service

The staged command records the target server's boot identity and operating system process start time with the PID. It reads process metadata through /proc on Linux, ps on macOS and BSD, and PowerShell on Windows. These identities let a later request reclaim an abandoned marker after the operating system recycles the numeric PID. Older markers stay compatible and use process start metadata when available. When OS metadata is unavailable, the current server's health-reported boot time can still prove that a legacy marker predates the process now using its PID. Paperclip refuses to create a new request without at least one identity source. Supported-platform process probes fail explicitly instead of silently treating a live PID as either the original owner or a recycled process when identity cannot be established.

Use --drain-required only when the deploy intentionally requires the old terminate-and-retry behavior. Without that flag, the old server verifies that the marker targets its own PID, stops new scheduler work, waits for any queue-claim callback already in flight, snapshots currently running heartbeat run IDs and child PIDs, and skips the shutdown drain so eligible detached local-agent processes can keep running. ACP-backed local runs use server-owned stdio and cannot survive their parent server, so the old server instead persists their complete snapshot, changes the marker to drainRequired with drainReason: "active_acp_run", and drains only those runs to bounded conversation retries. These retries resume the prior session when compatible, otherwise carry the full task conversation into a fresh session. They do not automatically replay tool calls or require receipts for every prior action. Detached CLI runs remain eligible for adoption during the same mixed restart. If an ACP process terminates but its terminal run update does not persist, startup classifies it as lost with reason selective_drain_not_finalized rather than treating the drain as successful. On startup the new server writes $PAPERCLIP_HOME/instances/${PAPERCLIP_INSTANCE_ID:-default}/hot-restart-report.json with previousServerPid, newServerPid, previousServerVersion, newServerVersion, drainReason, adoptedRunIds, finalizedWhileDownRunIds, lostRunIds, and per-run classifications before the normal orphan reaper runs.

When Paperclip manages embedded PostgreSQL, it suppresses that dependency's eager SIGINT/SIGTERM cleanup hooks. Paperclip owns signal ordering so the heartbeat snapshot and any required drain complete while the database is still available; the coordinated shutdown path stops embedded PostgreSQL afterward.

The request command records the preflight set of running heartbeat IDs and writes an instance-scoped marker plus a PID-targeted legacy home-root handoff marker. This lets a previous server version capture its snapshot at the old path while the new server correlates that snapshot back to the authoritative instance request. If any preflight run ID is absent from the shutdown snapshot, the startup report includes it in lostRunIds; a missing snapshot therefore cannot look like a zero-loss restart.

A healthy guarded deploy must compare the report against /api/health (version or serverVersion) and treat any lostRunIds entry as a continuity failure that needs recovery before marking deployment complete.

Recovering a deploy blocked by missing process metadata

If the currently installed version already has a running local-agent heartbeat whose processPid and processGroupId are both null, that pre-fix run cannot be made adoptable retroactively. Cross that version boundary once with the normal drain-and-retry path:

old_main_pid="$(systemctl show paperclip.service -p MainPID --value)"
pnpm --filter @paperclipai/server exec tsx ../scripts/request-hot-restart.ts \
  --server-pid "$old_main_pid" --drain-required
systemctl restart paperclip.service

After the fixed server starts, wait for the replacement codex_local heartbeat to spawn, then confirm its run record has an identity (use an authenticated API request in authenticated mode):

PAPERCLIP_API_BASE="${PAPERCLIP_API_URL:-http://127.0.0.1:3100}"
PAPERCLIP_API_BASE="${PAPERCLIP_API_BASE%/api}"
curl -fsS "$PAPERCLIP_API_BASE/api/heartbeat-runs/$RUN_ID" \
  | jq -e '.status == "running" and (.processPid != null or .processGroupId != null)'

The next continuity check should use the ordinary marker without --drain-required. After restart, require both an empty loss list and an explicit outcome for the run that was live before restart:

jq -e --arg run "$RUN_ID" \
  '(.lostRunIds | length) == 0 and ((.adoptedRunIds + .finalizedWhileDownRunIds) | index($run) != null)' \
  "$PAPERCLIP_HOME/hot-restart-report.json"

An alive child appears in adoptedRunIds; a child that completed during the restart window appears in finalizedWhileDownRunIds. Either is continuous. A lostRunIds entry remains a failed deploy and must not be waived.

For a recovery from a version that can stop embedded PostgreSQL before writing its shutdown snapshot, use --drain-required once to cross the broken boundary. After the fixed server is live, perform another ordinary hot restart. Require lostRunIds to be empty and every preflight run to appear in either adoptedRunIds or finalizedWhileDownRunIds; an ACP-backed original should be finalized and have a queued retry rather than be adopted.

Tailscale/private-auth dev mode:

pnpm dev --bind lan

This runs dev as authenticated/private with a private-network bind preset. On a fresh authenticated/private instance, open the app, sign in or create an account, and use the setup screen to claim the first instance admin from the browser. The CLI fallback remains:

pnpm paperclipai auth bootstrap-ceo

For Tailscale-only reachability on a detected tailnet address:

pnpm dev --bind tailnet

Legacy aliases still map to the old broad private-network behavior:

pnpm dev --tailscale-auth
pnpm dev --authenticated-private

Allow additional private hostnames (for example custom Tailscale hostnames):

npx paperclipai allowed-hostname dotta-macbook-pro