mirror of
https://github.com/paperclipai/paperclip.git
synced 2026-11-13 07:12:00 +01:00
DottaandClaude
be34a2d9d2
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work > - The restart-chaos harness (`tests/restart-chaos`) tests how runs survive control-plane faults > - Every wave runs one server, so no wave can reach a race between two servers on one database > - Hosted Cloud deploys by rolling overlap: a new server's startup recovery and sweeps run beside the old server's live runs > - #20508 fixed such a race (another server's orphan sweep failing a run whose owner took its lease back) with a database test only > - This pull request adds three two-server faults (MS1–MS3) and a cross-server invariant > - The benefit is that cross-server ownership races can be reproduced and graded in a live wave ## Linked Issues or Issue Description Refs #20508 (the race this harness can now reach live). **What is missing?** The harness starts one control plane per wave. A fault replaces or freezes that one server. Nothing starts a second server on the same database while the first still owns runs, so cross-server ownership bugs are found only by database tests. **How production runs (checked before choosing the scenarios)** - Hosted Cloud runs on Railway with Neon Postgres. A deploy is a rolling A/B overlap: B starts on the same database, A serves until B is ready, then A drains. Defaults are a 5 s overlap and up to 90 s of drain (`doc/plans/2026-10-31-zero-downtime-web-deploys.md`, lines 374–375). The chaos plan names the same shape as K12 (`doc/plans/2026-11-06-control-plane-restart-chaos.md`, line 103). It also records B's startup recovery hitting a lock on a run A still owned (lines 229–230). - This repository has no replica count or deploy strategy of its own. `docker/ecs-task-definition.json` describes one task, and the real Cloud deploy lives in another repository. The self-hosting docs do not describe several servers on one database. - Server-side ownership: per-boot controller lease of 90 s (`server/src/services/controller-instance-lease.ts`), legacy run lease of 60 s, native executor lease of 20 min, and the orphan sweep in `server/src/services/heartbeat/recovery.ts`. So MS1 models B's overlap beside a live A, MS2 models A paused (for example a host sleep) past its controller lease while B is up, and MS3 models the rolling upgrade itself. ## What Changed - New `tests/restart-chaos/multi-server.ts`: - MS1: B boots on the same database, instance directory, storage and secrets, on its own port. It stays up 60–150 s beside a live A, then gets SIGTERM. - MS2: B boots. A and its tree are frozen (TM2's freeze without aging) for 100–150 s, past the 90 s controller lease, then thawed. B stays up 30 s more, then stops. - MS3: B boots, then A gets K2 and a replacement on A's port. B stops 20–45 s after the replacement is ready. - B is always stopped once started, also when a step fails. - `checkCrossServer`, graded once per wave with an MS fault, over the wave database and the server process windows (pid, start, exit, freezes): - `MS_foreign_failure`: a run failed with `process_lost` or `native_execution_ownership_unverified` while its owner was alive and not frozen (#20508's shape included). The owner's own lease loss counts only without a prior freeze past the lease. - `MS_double_executor`: a run recorded a second runner identity before the first was shown stopped. - Unattributable ownership failures are listed, not charged. - `supervisor.ts`: records every server's process window and freezes, spawns and stops B (`<wave>/server-b.log`), and runs the MS steps. TM freezes are recorded too. - `types.ts`, `scheduler.ts`, `journey-matrix.ts`, `config.ts`: MS1–MS3 fault kinds, seeded holds, and validation (local only; no storm, crash point or K14). - `main.ts`: writes `<wave>/cross-server.json` and fails the wave on a violation. - README section "Two servers, one database (MS1–MS3)", with the production evidence and the limits. Test-only. No `package.json` or `.github` changes. ## Verification - `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts`: 38 files, 473 tests pass. - New `multi-server.test.ts` (14 tests): option validation and seeded holds; MS1/MS2/MS3 step order and timing with fake hooks; B stopped when a step fails; the invariant for a live owner, a frozen or exited owner, #20508's thaw-then-write shape, lease loss with and without a freeze past the lease, unattributed owners, and double executors. - New `multi-server-db.test.ts` (embedded Postgres): a run still `running` with `native_execution_ownership_unverified` (#20508's marker) is read at its write time and charged as `MS_foreign_failure`. - `node scripts/typecheck-test-projects.mjs tests/restart-chaos` passes. - Live wave `w-20261110T105448-s301` on head `a9ec5a4502` (master product code + this harness), `--adapters codex --workloads B5,Q1 --faults MS1,MS2 --concurrency 1 --tasks 3`: 3 of 3 pass, all crossed their fault. B booted on the shared database in about 10 s each time. MS2 froze A for 104 s; A stayed up after the thaw. Cross-server invariant: 0 violations, 0 unattributed. ## Risks - Test-harness code only. - B runs on the same host and PID namespace as A, so it can check A's PIDs directly, which a Railway replica cannot. These faults are a lower bound on cross-server exposure. The README says so. - MS2 outlasts the 90 s controller lease, not the 20 min native executor lease. - MS faults refuse TM2/TM3 in the same wave, and fail when B exits before its planned stop. - `MS_foreign_failure` uses the run's `finished_at` (or `updated_at` for a still-running row) and the owner recorded on the run. A run whose owner changed after the failure could be misattributed; the evidence file keeps the owner and windows for review. - `MS_double_executor` reads runner identity events; it cannot see a second executor that never recorded one. ## Model Used - Anthropic Claude, model ID `claude-opus-5-5`, used through Claude Code with tool use (shell, file edit, code execution). ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>