Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-10 01:50:13 +01:00
Code Issues Packages Projects Releases Wiki Activity

docs(plans): plan durable ownership and reconciliation for restart safety (#19831)

Browse Source 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs must survive a control-plane restart: they must continue
on their own, keep their work, and offer a Retry that works
> - The 2026-11-06/07 restart chaos campaign found about 40 restart
failures and merged about 69 fixes (#19469 through #19807)
> - Most of these failures have one shape: a restart stops a process
between two durable writes, and each subsystem then decides on its own
whether the leftover record is safe to release
> - Each fix closed one crash window, and about 12 fixes rebuilt the
same "this owner is dead" proof
> - This pull request adds a plan for one shared ownership record,
write-ahead intents, one reconciler, and an on-demand crash-point suite
> - The plan also makes time to first token and agent boot hard
requirements, so restart safety never makes turns slow
> - The benefit is that we fix the whole class of failure once, and we
stop fixing one window at a time

## Linked Issues or Issue Description

Refs #19696, #19713, #19717, #19724, #19741, #19747, #19784, #19807

**What existing behavior does this improve?**
Recovery after a control-plane restart. Today each subsystem keeps its
own in-progress records and its own release rules.

**Subsystem affected**
Native runtime, runner, environments and sandboxes, workspaces,
accounting and budgets, and the wake queue.

**Current behavior**
A restart can leave a lease, a fence, a claim, a workspace operation, a
receipt, or a deferred wake that belongs to a dead process. If no
subsystem can prove that the owner is dead, the system fails closed. The
task then needs a Retry, the company holds new work, or a run stays
`running`. About 40 server files read a controller boot id, and about 25
files check process liveness in different ways.

**Proposed behavior**
One owner stamp and one liveness proof for every in-progress record.
Intents are written before outside effects. One reconciler brings every
record to a terminal state. Admission never waits on cleanup. Accepted
work is delivered exactly once. Time to first token and agent boot are
measured and stay within budgets.

**Reason and benefit**
Fewer restart failures, less duplicate code, and a fast first response
after a restart.

## What Changed

- Added `doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`.
The plan contains:
- the five failure classes from the campaign, with example IDs and PR
counts
  - eight invariants
- a performance section with time-to-first-token, boot, and
restart-latency metrics and rules
  - the design (sections A–G)
  - an on-demand crash-point suite
  - six phases
  - risks
  - the decisions that Dotta made on 2026-11-07

## Verification

- This change adds documentation only. No code changes.
- Read the plan. Check that each failure class cites real merged PRs.
- Check the code claims:
- `git grep controller-process-identity -- server/src` shows three
users.
- `git grep task.time_to_first_token -- server/src` shows the existing
metric.

## Risks

- Low risk. The change adds one plan document and changes no code.
- The plan proposes later behavior changes. The largest is the removal
of the company-wide accounting gate. Each later phase will come as its
own pull request with tests.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, git, GitHub CLI). The model coordinated the chaos
campaign and wrote the plan from its evidence.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
DottaandClaude Opus 5.5 authored and GitHub committed 2026-11-07 08:36:05 -06:00
1 parent 2e52dfa864
commit 0f5ce5050e
1 file changed
+203
Download Patch File Download Diff File Expand all files Collapse all files
doc/plans/2026-11-07-durable-ownership-and-reconciliation.md
+203
View File
@@ -0,0 +1,203 @@
# Durable ownership and reconciliation for restart safety
Status: draft for review, 2026-11-07
Evidence: the 2026-11-06/07 control-plane restart chaos campaign (plan: `doc/plans/2026-11-06-control-plane-restart-chaos.md`). It merged about 69 PRs, #19469 through #19807. The "RC-" IDs below are that campaign's signature IDs.
## Recommendation
Stop fixing restart bugs one crash window at a time. Add one shared ownership primitive, one write-ahead intent rule for outside effects, and one reconciler that brings every in-progress record to a terminal state. Then move the existing hand-built mechanisms onto it, one at a time. Gate it in CI with a deterministic crash-point suite, not with random chaos timing.
## Why: the campaign's bugs share one shape
A control-plane restart can land between any two durable writes. It leaves an in-progress record owned by a process that is now dead. That record might be a lease, a startup fence, a coordinator claim, a workspace operation, an accounting receipt, a reservation or a deferred wake. Each subsystem decided alone whether it was safe to continue, and none could prove the owner was dead. They fail closed, so the leftover record became a Retry, a company-wide hold, or a run that stays `running` forever.
| Class | Examples (ledger IDs) | Approximate PRs |
|---|---|---|
| A dead owner's fence or lease that only a TTL releases | RC-16/16b/16c, 29d, 39, 41, 42, 47, 50, 57, 62, 64 | 12 |
| Accounting completeness gates company admission | RC-05/06 family, including regressions RC-06k and RC-06m | 8 |
| Work intents (wakes, Retries, comments) end terminally on a temporary refusal | RC-37, 45, 58, s7532 | 4 |
| Retry budgets charged per restart, not per unit of work | RC-22/22b/22c, 40, 60 | 5 |
| Events from an earlier generation break the current turn | RC-15, 20/20b/20c, 33/33b, 59, 61 | 10 |
The repeated reinvention shows up in the code:
- 40 server files reference a boot id.
- About 25 implement process liveness (`isProcessAlive`, `isProcessGroupAlive`, `controllerProcessExited`, `proveLocalWorkspaceOwnerStopped`, and others).
- 15 or more `*ownership*` / `*owner*` modules each carry their own rules.
The accounting fixes regressed each other three times. Twice the cause was an inferred-evidence check that was subtly wrong: key presence counted as proof, and an empty map was not null (RC-06m).
## Invariants the design must guarantee
These become reconciler checks and crash-suite assertions. They should also be added to `doc/SPEC-implementation.md`.
1. **Owned:** every in-progress durable record carries an owner stamp and a generation.
2. **Settled:** a record whose owner is proven gone reaches a terminal state within a bounded time (target: one reconciler pass after boot). Proof is by identity, never by age.
3. **Admission is independent of cleanup:** admitting new work never waits on another run's cleanup or accounting completeness. Only real spend reaching a hard stop, an explicit pause, or an inactive company refuses work.
4. **Exactly-once intents:** an accepted work intent (assignment, comment, Retry, recovery) is delivered exactly once, unless the task is cancelled or a hard stop applies. A temporary refusal defers it and never ends it terminally.
5. **Work-based budgets:** a restart consumes run budget only when the resumed attempt started provider work and made no progress.
6. **Generation fencing:** every runner and provider event carries its generation and turn. Events from an earlier generation are recorded as history and never fail the current turn.
7. **Visible next action:** every non-terminal task has a next action, either automatic or a working control. "Waiting" with no owner is a violation.
8. **Fast turns:** TTFT and agent boot stay within their measured budgets. Restart safety work never adds waiting or file transfer before `turn.start` (see "Performance constraint" below).
## Performance constraint: a turn must start as fast as possible
When someone messages an agent, the reply has to start as fast as possible. Restart safety must not make turns slow. A turn that spends minutes moving files between the sandbox and the host before the model starts fails this design, however correct it is.
### What we measure
These are measured for every run and recorded in the run log. With an OTLP endpoint set, they are also emitted as spans on the existing `run.startup.step` / `paperclip.sandbox.startup.*` path in `packages/adapter-utils/src/acpx-engine/startup-timing.ts`. They are never sent to Telemetry (AGENTS.md §5.7).
| Metric | From | To |
|---|---|---|
| **Time to first token (TTFT)** | user message or wake accepted | first deliverable model text. This extends the existing `task.time_to_first_token` (`native-session-executor.ts`) back to the message timestamp. |
| Agent boot | run claimed | provider session ready to accept `turn.start` |
| Critical-path breakdown | | queue wait, admission, workspace preparation and staging, environment acquire, runner launch, session open/resume, `turn.start` to the provider's first event |
| Restart recovery latency | control plane ready | recovered run's next provider event, for reattach and resume separately |
| Turn tail | provider turn completed | run terminal and task updated, including copyback and sync-out |
Each metric is reported by adapter (Codex, Claude/ACPX, OpenCode) and environment (local, Daytona), for warm and cold starts, as p50/p95.
### Rules for this design
1. **Nothing new goes on the critical path.** Intent records, owner stamps and generation stamps are small single-row writes in the transaction the path already uses. The reconciler runs only in the background. It never adds a wait before `turn.start`.
2. **File transfer must not block the next turn.** A follow-up turn on a still-valid sandbox lease uses the workspace that is already there. Sync-out, checkpoints and copyback run after the turn or alongside it, and incrementally. They never run as a full round trip before the model starts. A restart with a live runner reattaches; it does not restage.
3. **Fixed waits become event-driven.** Each one the campaign found is a target:
- RC-04: graceful drain waits the full 30 s;
- RC-09: a cold OpenCode attach takes 31–38 s against a 30 s bound;
- RC-36: `session.open` 30 s;
- RC-20b: a late Codex turn start of about 27 s;
- RC-27: a 5-minute orphan reaper.
4. **Phase 0 measures, then sets budgets.** It records a baseline for every metric above. Phase 1 sets p50/p95 budgets per adapter and environment. From then on, every phase's exit criteria include "no regression against the baseline". The on-demand crash-point suite and the chaos soak also report TTFT and restart recovery latency.
## Relationship to the zero-downtime upgrade work
Separate work moves the control plane toward zero-downtime upgrades on an ephemeral filesystem. PostgreSQL holds execution authority and durable protocol state. Object storage holds workspace archives, provider checkpoints, logs, traces and skill assets. Replacement servers reuse stable encryption and signing secrets. Most of the foundations are on master; #19830 is still open. Entry points:
- `server/src/services/ephemeral-server-contract.ts`
- `server/src/services/heartbeat/upgrade-readiness.ts`
- `server/src/services/native-runtime/native-runner-ownership.ts`
- `server/src/services/native-runtime/native-finalization-reconciler.ts`
- `server/src/services/native-runtime/native-harness-filesystem-independence.ts`
- in paperclip-cloud, `src/fleet/ephemeral-overlap.ts`
This plan builds on that work and must not fork it:
1. **The remote runner keeps running across a control-plane replacement.** The replacement server takes fenced ownership and reconciles durable checkpoints and results. It never starts a duplicate execution. So for remote runners, ownership moves by **fenced handoff** (lease and fence token), not by a process-liveness proof. A different host is never proof that an owner is dead. The `OwnerStamp` in section A records the fence token. `proveOwnerGone` applies only to local processes and to providers that give a platform proof.
2. **Graceful handoff and abrupt death are different paths.** Graceful replacement releases ownership explicitly. Abrupt death during a tool, result acceptance, finalization, or workspace export is recovered by proof or fence expiry. The crash-point suite covers both, including stale-owner fencing and duplicate result and accounting protection.
3. **Obligations stay separate.** Ownership, accepted results, workspace preservation and accounting are separate obligations. Each Phase 0 inventory row names exactly one of them, and settling one never settles another:
- preserving files never marks a failed run successful;
- preserving files never clears an unresolved export or recovery hold;
- preserving files never grants provider-resume authority.
WEB evidence receipts prove filesystem preservation, not successful remote recovery.
4. **Extend the existing pieces.** Section C extends `native-finalization-reconciler.ts`; it does not add a second loop. Section A starts from `native-runner-ownership.ts` and `controller-process-identity.ts`. Phase 0 includes the upgrade work's record kinds, and its owner reviews the inventory.
5. **External tool effects are not exactly-once.** The design guarantees exactly-once for Paperclip's own records (results, accounting, wakes). A tool's outside effect may run again after a crash. The agent checks before it repeats one; see #19804.
6. **Object storage must stay off the turn path.** Restoring an archive or checkpoint must not delay `turn.start` when a live workspace or runner exists. A cold restore that cannot be avoided counts against the time-to-first-token budget and is reported as its own step.
## Design
### A. One ownership primitive
Start from `server/src/services/controller-process-identity.ts`, which #19696 introduced. Today only `sandbox-creation-operations.ts`, `heartbeat/recovery.ts` and `environment-runtime.ts` (#19749) use it. #19713 (`native-dead-controller-result-lease.ts`) and #19717 (`native-workspace-owner-stop-proof.ts`) built their own proofs of the same fact in the same night.
- `OwnerStamp` = `{ host, pidNamespace, bootId, pid, processStartedAt, processGroupId?, generation, scope }`.
- One prover, `proveOwnerGone(stamp)`:
- A local proof requires the same host and pid namespace, plus either a dead pid or a reused pid (checked against the start time), plus an empty or older process group.
- A remote proof is delegated to the environment provider: lease termination, sandbox state, or a runner identity marker.
- The prover returns `gone | alive | unprovable`. `unprovable` never releases a record.
- Every record kind in the inventory below stores an `OwnerStamp` when it enters an in-progress state.
### B. Write-ahead intents for outside effects
The rule: record an intent before the effect, and record the outcome after it. Recovery reads the intent log, not inferred evidence. Several kinds already do this and should share one shape and one settle path:
- `sandbox_creation_operations` (#18357, #19696)
- the runner's `startupAttempt` with `beforeSpawn` (#19784)
- the controller's `turn.start` journal (#19772)
- workspace operations (#19807)
Effects still to cover:
- copyback stop
- runner release/destroy, partly done in #19749
- reservation hold and release
- workspace staging
### C. One reconciler
A registry maps each ownable record kind to `{ listInProgress, settle(record, proof), invariantChecks }`. One loop runs at startup and every 30 s:
- For each in-progress record with an owner stamp that isn't the current boot, call `proveOwnerGone`. If the owner is gone, call `settle`. If the proof is unprovable, surface an operator action.
- Run the invariant checks (I1–I7). Each violation writes a `recovery.invariant_violation` activity and a run-log event with the record and the rule. The loop repairs a violation when a safe repair is registered.
- Keep the loop sleep-aware: no work when nothing is in progress (see the `sleep-aware-background-work` skill).
This replaces the per-subsystem startup sweeps gradually, by deleting each one once its kind moves onto the registry.
### D. Separate admission from accounting
- Admission reads per-run exposure only: reserved plus observed cost. Hard stops use settled or estimated spend.
- A run with unfinished accounting adds its bounded estimate to exposure. It never adds a company-wide gate.
- This finishes the direction of #19497 → #19724 → #19826 and removes the `waitingRunCount` gate, rather than narrowing it again.
### E. Work-intent outbox
- Wake requests get a closed set of terminal reasons: delivered, consumed by a later run, task cancelled, or hard stop. Every temporary refusal becomes `deferred` with a reason and a wake trigger.
- Any event that releases a hold (accounting settled, lock released, pause lifted) promotes deferred intents through one function. The reconciler covers any trigger that is missed.
- Consumption is recorded by the comment or intent ID on the run that delivered it. That gives exactly-once delivery: no drops (RC-37) and no double delivery (RC-53).
### F. Work-based attempt budgets
Charge an attempt only on a recorded outcome that proves provider work started. An intent alone is not that proof. Use the outcome records from B:
- **Not charged:**
- a startup intent that is still marked `beforeSpawn`, because no process started;
- a startup that was spawned but never admitted;
- a `turn.start` that the provider did not accept.
- **Charged:** an attempt whose provider accepted a turn (the accepted `turn.start` outcome or `turn.started`), and which then made no progress before the next failure. Progress is defined as in #19720: completed work for that turn.
The rules from #19720 and #19747 then become one rule over these outcome records, instead of watermark heuristics. The on-demand crash-point suite asserts this at the pre-spawn and pre-accept crash points.
### G. Generation stamping
The runner stamps `generation` and `turnId` on every event. Driver and session code treat an older generation as history. #19718, #19722, #19772 and #19783 each special-cased one variant of this.
## Testing: enumerate crash points, don't sample them
- Add named crash points at each durable write boundary, for example `crashPoint("sandbox.create.after_provider_before_lease")`. In tests, an environment variable makes the process SIGKILL itself at the Nth hit. In production they compile to no-ops.
- An on-demand suite boots the stack once per crash point, runs one task per workload, restarts, and asserts invariants I1–I7. It runs locally or by manual CI dispatch, filtered to the crash points a change touches. It is not a required PR check.
- Keep the chaos harness as a soak test for timing interactions the enumeration misses. Its R1–R6 graders should map onto I1–I7.
- The random waves found 20 ms–1 s windows by luck. Enumeration finds them on purpose, and it finds the windows we haven't hit yet.
## Phases
| Phase | Scope | Exit criteria |
|---|---|---|
| 0. Inventory and baseline | List every in-progress record kind: table, column, writer, current release rule, current proof. Start from the 43 ownership, lease, sweep and fence modules. Separately, instrument and record the TTFT, boot and restart-latency baseline per adapter and environment, and list everything on the critical path before `turn.start`. | An inventory doc reviewed, with each kind mapped to A–G. A baseline report with the slowest critical-path steps ranked. |
| 1. Primitive and observe-only reconciler | `OwnerStamp`, `proveOwnerGone`, registry, reconciler running invariant checks in report-only mode. No behavior change. | The invariant violation rate is measured in chaos waves and staging. |
| 2. Move kinds over | One PR per kind or small group. Each switches release to the shared path and deletes its bespoke code, with a crash-point test. | Bespoke liveness code removed; crash-point suite green per kind. |
| 3. Admission and outbox | D and E. These change the most user-visible behavior, so they wait until Phase 1 data exists. | No company-wide gate; zero terminal skips for temporary reasons. |
| 4. Budgets and generations | F and G. | One attempt-charge rule; no per-variant stale-event code. |
| 5. On-demand suite | The crash-point suite is runnable locally and by manual CI dispatch, filtered by crash point or record kind. The chaos soak stays available too. | The suite is green on the release candidate; chaos soak 500-pass streak target. |
## Risks
- **Failing open.** A wrong "owner gone" proof releases live work and can duplicate side effects. Mitigation: `unprovable` never releases; a different host never counts as proof; every settle path is compare-and-swap; crash-point tests cover live-owner counter-cases for every kind.
- **Remote proof.** Daytona and other providers vary. Mitigation: the provider plugin contract returns the same `gone | alive | unprovable`.
- **Reconciler cost and idle sleep.** Mitigation: an indexed in-progress query per kind; skip when no kind has in-progress rows.
- **Migration churn in hot files** (`heartbeat.ts`, `native-session-executor.ts`, `budgets.ts`). Mitigation: Phase 1 adds only new code. Each Phase 2 PR is narrow and follows `rules/heartbeat-execution-boundaries.mdc`.
- **Latency regression.** New writes or proofs creep onto the turn path. Mitigation: invariant 8, a TTFT and boot budget check in every phase's exit criteria, and a rule that proofs needing I/O (process listing, provider API calls) run in the reconciler, never before `turn.start`.
- **Telemetry boundary.** Invariant violations go to the run log and activity log, not to Telemetry (AGENTS.md §5.7).
## Decisions (Dotta, 2026-11-07)
1. **Remove the company-wide accounting gate entirely.** Phase 3 deletes the gate; it does not narrow it further. Admission reads per-run exposure and hard stops only. This is a product rule and goes into `doc/SPEC-implementation.md`.
2. **The crash-point suite runs on demand and is not a required PR check.** It is available as a local command and as a manual (`workflow_dispatch`) CI run that can target selected crash points or record kinds. Phase 5's exit criterion changes to "the suite is runnable on demand and green on the release candidate".
3. **Inventory ownership:** one owning agent, with parallel per-subsystem readers (see "Phase 0 ownership" below).
4. **Land the open restart PRs first:** #19727, #19826 and #19829 merge as they are. They shrink Phase 2.
## Phase 0 ownership
One agent owns the inventory end to end. That keeps one schema, one set of definitions for "in progress", "owner" and "proof", and one reviewer of cross-subsystem cases. The owner fans out reading to a few parallel readers, one per subsystem:
- native runtime and runner
- environments and sandboxes
- workspaces
- accounting and budgets
- wake queue and recovery
Each reader fills fixed columns for its modules: table and column, writer, in-progress states, current release rule, current liveness proof, and which invariant it touches. The owner merges the rows, resolves overlaps (one record released by two paths, or two proofs of the same fact), and maps each kind to A–G. Dotta reviews the finished inventory once before Phase 1.