Commit Graph
3 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 3afb2c215b fix(server): name the blocking database sessions whenever a run's event writes stall (#21956)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server stores the events that a runner sends for each run, and
it records the identity of the runner process
> - Sometimes the server stops storing a run's events for minutes, and
the server log stays quiet
> - In two restart-chaos cases, the identity write of the run started
and then did not commit for about 197 s and 480 s
> - The existing database report only fires when the reattach phase ends
at its cap, so it did not fire for the cold-start case
> - This pull request logs the pending writes and the blocking database
sessions whenever any run write stays pending for 30 s
> - The benefit is that the next stall names the session and the query
that hold the run, so the fix can target one proven holder

## Linked Issues or Issue Description

Refs #21946

**What happened?**
Restart-chaos waves on master showed two stalls in event storage.
1. Cold start of an OpenCode run, with no restart. The runner emitted
`session.started` at 22:13:23.7Z. The server stored it at 22:16:40.97Z,
about 197 s later. The second `native.process_identity_recorded` write
of the run started at 22:13:23.706Z, and `runner.turn.submit` took
197270 ms. A second wave showed the same shape (about 191 s). The server
log had nothing in that window.
2. After a hard kill, every API GET timed out from 22:35:24Z to
22:43:23Z, server CPU was 0%, and no run events were stored. The process
identity transaction of the run began at 22:35:22.6Z and committed at
about 22:43:23Z.

**Expected behavior**
When the server cannot store a run's events, the server log names the
cause: which run writes are pending, and which database sessions hold or
wait for locks.

**Steps to reproduce**
1. Start a transaction that locks a `heartbeat_runs` row (`for update`)
and stays idle.
2. Record a process identity for that run
(`persistHeartbeatRunProcessMetadata`).
3. Before this change, nothing is logged while the write waits. After
this change, a warning names the idle holder and the waiting statement
after 30 s. The new database test does this with a 300 ms threshold.

**Relevant logs or output**
Example warning from the new test:
`runner event intake stalled; pending run writes, database sessions and
lock waits` with `operations: [{stage: "process_identity", pendingMs:
301}]` and `waits: [{state: "idle in transaction", query: "select id
from heartbeat_runs where id = ? ... for update"}, {waitEventType:
"Lock", blockedBy: [<holder pid>]}]`.

## What Changed

- New `server/src/services/run-intake-stall-watch.ts`. It keeps an
in-memory list of pending run operations: event receipts
(`PaperclipControlPlanePort.appendEvent`), event writes
(`appendHeartbeatRunEvent`, `appendHeartbeatRunEventBatch`), and process
identity writes (`persistHeartbeatRunProcessMetadata`).
- When an operation stays pending for 30 s, the watch logs one warning
per stalled run. The warning has: the pending operations and their age,
the pending count and oldest age per run, the pool size, the database
sessions per application name and state, the lock waits and holders
(`pg_blocking_pids`, query without literals, cut to 160 characters), and
the event-loop lag. Reports are at most one per 30 s. A second line logs
when a reported operation finishes.
- The watch arms a timer only while an operation is pending. It reads
the database only when it reports, through the private liveness pool
with a 5 s statement timeout.
- The server names its main pool `paperclip:<boot id>` unless
`DATABASE_APPLICATION_NAME` is set. This shows which server process owns
each session after a restart.
- `readDatabaseSessionWaits` moves from the reattach phase to
`server/src/services/database-session-waits.ts`. The reattach phase
re-exports it.
- `@paperclipai/db` exports `databaseClientOptionsFromEnv` and
`DatabaseClientOptions`.
- `doc/DATABASE.md` describes the new log lines and the pool name.
- I did not add `idle_in_transaction_session_timeout` to the main pool.
Some code holds a transaction open on purpose, and a blind timeout could
abort valid work. The new report names idle-in-transaction holders
instead.
- These are server log lines only. This change does not touch Telemetry
or OpenTelemetry.

## Verification

- `cd server && npx vitest run
src/services/run-intake-stall-watch.test.ts` (5 tests). The database
test holds a run row in an idle transaction, starts a process identity
write, and checks that the warning names the holder pid, its query, the
waiting statement, and the pool name. It is red without the tracking in
`run-lifecycle.ts` and `heartbeat-run-events.ts` (`expected [] to have a
length of 1`), and green with it.
- Unit tests with fake timers check one report per stalled run, the 30 s
throttle, the clear line, and that no timer stays armed when nothing is
pending.
- `cd server && npx vitest run
src/services/controller-process-identity.test.ts` checks the pool name
and the `DATABASE_APPLICATION_NAME` override.
- `npx tsc --noEmit` passes for `server` and `packages/db`.
- `pnpm verify` ran locally on the affected tests.

## Risks

- Low risk. Each run event adds one map insert and one map delete. No
database work happens while writes finish within 30 s.
- The main pool `application_name` changes from `paperclip` to
`paperclip:<boot id>`. A dashboard that filters on exactly `paperclip`
must use a prefix match, or the operator can set
`DATABASE_APPLICATION_NAME`.
- A report reads `pg_stat_activity` on the liveness pool. A report
during a full database stall can fail; the warning then has
`databaseReadError` and still lists the pending operations.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 22:15:40 -06:00
DottaandClaude Opus 5.5 a4dbde6181 refactor(recovery): judge native owner processes with the shared prover (#19997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs record the server and adapter processes that own them,
and recovery must prove an owner gone before it takes over or releases
its work
> - Phase 2 of the durable ownership plan gave the repository one
prover, `proveOwnerGone`. The native runtime still had four private
proofs of the same fact
> - The private proofs disagreed with the shared rules in edge cases:
some judged a PID from another PID space, and one accepted an
inconsistent start time as proof of reuse
> - This pull request moves those four proofs onto `proveOwnerGone` and
deletes the private liveness code
> - The benefit is one rule for "the owner is gone" across native
recovery, with the stricter shared behaviour where they differed

## Linked Issues or Issue Description

Refs #19933, #19950, #19953 (the shared ownership path and earlier Phase
2 kinds).

**What is the problem?**
Native recovery proved dead owners with four private checks instead of
the shared prover. The checks used different rules for PID spaces,
reused PIDs and unreadable start times.

**What should happen?**
Every native owner proof uses `proveOwnerGone`. Only a `gone` verdict
releases or takes over. The remote fence and the one-release legacy
fallback stay as they are.

**Why does it matter?**
Recovery decisions about the same process must agree. A dead PID in
another PID space must never be used as proof.

## What Changed

- `native-restart-recovery.ts` `evaluateNativeControllerTakeover`: the
local-process branch now calls `proveOwnerGone`. The remote fenced
branch (#19830), the `operator_attested` sentinel and the null-host
fallback (`processSpaceUnrecorded` with `legacyHostFallback`, recorded
by callers as `legacy_host_unknown_dead_pid_proof`) are kept. Reasons
are unchanged, plus a new `controller_pid_start_inconsistent`. Owners
built from receipts with no PID space are judged in this PID space:
their callers check the recorded host first.
- `native-failed-run-owner-release.ts`: `recordedProcessMayBeLive` is
deleted. `runProcessMayBeLive` and `controllerMayBeLive` release only on
a `gone` verdict. Each use of the legacy fallback is recorded as
`legacy_host_unknown_dead_pid_proof` when the release commits.
- `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` is
replaced by `recoveryProcessMayBeAlive`, which is false only when the
prover proves the runner gone. Signalling a recovered runner requires a
proven same process.
- `native-local-process-stop.ts` `recordedLocalExecutionStopped`: the
bound identity (launch event, host, PID, group, start time) is judged by
the prover. It uses `process_host_hash` when the row has one.
- New `native-runtime/native-owner-stamps.ts`: controller and
adapter-process stamp builders, shared with the ownership registry.

Not changed: `proveLocalWorkspaceOwnerStopped` (workspaces),
`evaluateNativeProviderProcesses`, the stop observation in
`recordNativeLocalProcessStop`, and `runProcessMayBeLive` in
`native-workspace-sync-retention.ts`.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-restart-recovery.test.ts
src/services/controller-process-identity.test.ts`: 51 passed.
- `cd server && npx vitest run
src/services/native-runtime/native-failed-run-owner-release.test.ts`: 14
passed.
- `cd server && npx vitest run
src/services/native-runtime/native-controller-host-recovery.test.ts`: 2
passed.
- `cd server && npx vitest run
src/services/quarantined-task-retry.test.ts -t "keeps quarantine"`: 18
passed. `explicit-native-continuation.test.ts`: 414 passed.
`ownership-reconciliation-report.test.ts`: 23 passed.
- Red with the master versions of the two modules: 3 new tests fail
("accepts a reused PID only when the live process started definitely
later", "never releases on a dead PID recorded in another PID space",
"keeps the one-release fallback for a run process saved before its PID
space, and records it"). The tests were written after the code.
- Two fixtures changed to match the shared rules:
- `controller-process-identity.test.ts`: the "recycled" case recorded a
start after the live process, which is an inconsistent stamp. It now
records an earlier start (a real reuse), and the inconsistent case is
asserted as alive.
- `quarantined-task-retry.test.ts` `worker_alive`: the live worker now
records this process's real start time. The old fixture was a reused
PID, which the prover correctly proves gone.
- `cd server && npx vitest run
src/services/native-local-process-stop.test.ts`: 5 passed (review round
1). With the master module, 2 fail: the other-PID-space case and the
reused-PID case.
- Not run locally: typecheck (CI runs it) and the wide suites.

## Risks

Behaviour changes, all from the shared rules:
- A live PID observed to start before the recorded controller is no
longer accepted as recycled (`controller_pid_start_inconsistent`; the
run waits for evidence).
- A failed-run owner whose recorded adapter process or controller is in
another PID space, or is a remote sandbox PID, is no longer released by
a local probe. It keeps the lease path.
- An unreadable start time is no longer exit evidence for a recovered
local runner.
- A local execution whose PID was reused by a later process now counts
as stopped. Before, any live PID blocked the proof.
- A local execution whose row records another PID space is no longer
counted as stopped by a dead local PID.
- A PID space that cannot be read here makes a hostless receipt owner
unprovable.

No persisted state changes. Rows saved by the previous release keep the
one-release null-host fallback, as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 02:35:15 -06:00
DottaandClaude Opus 5.5 630411af53 fix(recovery): require a 1 s start-time gap and treat EPERM as alive in owner proofs (#19848)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a restart, recovery code decides whether the process that owns
an in-progress run is gone
> - Two proofs compare process start times to the millisecond, but `ps
-o lstart` on macOS and BSD reports whole seconds
> - So a live owner can look like a reused PID, and recovery can release
it while it still runs
> - Signal-0 handling also disagrees: one check treats EPERM as dead,
while all others treat it as alive
> - This pull request puts one start-time rule and one EPERM rule in
`controller-process-identity.ts` and uses them at every site the Phase 0
ownership inventory named
> - The benefit is that no proof releases a live owner because of clock
precision or a permission error

## Linked Issues or Issue Description

**Bug description**
The Phase 0 ownership inventory (finding 1) found inconsistent "owner
gone" proofs. `native-failed-run-owner-release.ts`
(`recordedProcessMayBeLive`, `runProcessMayBeLive`) and
`native-session-executor.ts` (`verifiedRecoveryProcessIsAlive`) treat
any start-time difference of 1 ms or more as a reused PID.
`native-restart-recovery.ts` and `controllerProcessExited` require a gap
of at least 1 s. `local-service-supervisor.ts` (`isPidAlive`,
`isProcessGroupAlive`) treats EPERM as dead.

**Steps to reproduce**
1. On macOS, record a live process start time with millisecond
precision.
2. Run the abandoned-owner release pass, or the recovered-runner
liveness check, against that live PID.
3. The pass releases the owner, or the check reports the runner as gone.

**Expected behavior**
A sub-second start-time gap is ambiguous and keeps the owner. EPERM
means the process exists.

Refs #19846

## What Changed

- `controller-process-identity.ts`: export `processExists` (EPERM =
alive) and `processStartsDefinitelyDiffer` (gap >= 1 s; a missing time
proves nothing). `controllerProcessExited` uses the new helper.
- `native-failed-run-owner-release.ts`: the controller and run-process
checks use the shared helpers instead of exact millisecond equality.
- `native-session-executor.ts`: `verifiedRecoveryProcessIsAlive` accepts
a start time within 1 s as the same runner. Its signal path uses the
same check.
- `native-restart-recovery.ts`: `definitivelyDifferentProcessStart`
delegates to the shared helper. No behavior change.
- `local-service-supervisor.ts`: `isPidAlive` and `isProcessGroupAlive`
treat EPERM as alive.

## Verification

- New `controller-process-identity.test.ts`: the 1 s boundary, EPERM and
ESRCH for every signal-0 check, and a live spawned runner whose recorded
start is 500 ms off.
- `native-failed-run-owner-release.test.ts`: two new cases (controller
and run process) with a live child and a recorded start 500 ms off. The
owner stays `observed`.
- Red before the fix: with the old call sites, all 4 new cases fail
(`releaseAbandonedNativeOwners` returns 1 and releases the live owner;
EPERM returns false; the live runner reads as gone). Green after the
fix.
- `npx vitest run src/services/controller-process-identity.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/local-service-supervisor.test.ts`: 59 passed.

## Risks

- Low risk. The change only makes proofs stricter: a sub-second gap or
EPERM now keeps an owner that the old code released. Such a row waits
for the existing lease, TTL, or real process exit.
- A recycled PID whose new process started within 1 s of the recorded
one now stays held until it exits. That is the same rule the takeover
path already used.
- Previous-release story: no persisted shape changes. Rows saved by the
previous release are evaluated with the stricter rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used with tool use and
code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-07 10:06:35 -06:00