Commit Graph
2 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 648b238be1 perf(native-runtime): stop re-finalizing native results whose finalization error is recorded (#21674)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The native runtime releases owners of failed runs. When the owner's
controller is in a process space that Paperclip cannot verify, it shows
controller namespace attention on the task instead.
> - On a hosted instance with zero active runs, this writer made about
21% of all work signals. It inserted a new active attention row for the
same task about every 26 seconds.
> - The tasks were already done or cancelled. Recovery resolved each row
as `source_terminal`. The active slot was then free, so the next
owner-release pass inserted a new row and published a graph recovery
signal again. The loop did not stop.
> - A live task had the same defect: after the fresh-continuation
fallback resolved its attention, the next pass made the same incident
again.
> - This pull request stops the writer from surfacing attention for
finished tasks. It also stops the writer from making an incident again
when that run authority was resolved after its lease expired. Those
owners are no longer release candidates and do not set the release
deadline.
> - The benefit is that an idle instance does not write or publish
anything for these runs. Attention still shows once for a live task.

## Linked Issues or Issue Description

**What happened?**
The failed-run owner release (`settleFailedRunOwner`, expired-owner
path) called `surfaceNativeControllerNamespaceAttention` for every
expired foreign owner that had no active attention. It did this even
when the source task was `done` or `cancelled`. Recovery resolved that
attention as `source_terminal`. The next pass found no active attention
and inserted a new board row. Each insert published
`notifyIssueGraphRecovery` and the other execution-control signals. The
stranded-task check then ran again for the task graph. A hosted instance
collected about 45,000 rows on 13 finished tasks in this way.

**Expected behavior**
A pass that changes nothing writes nothing and publishes nothing. A
finished task never gets controller namespace attention. A live task
gets one incident for each exact run authority. A resolved incident
stays resolved until the authority changes.

**Steps to reproduce**
1. Make a native run that failed, with an `observed` coordinator whose
foreign lease has expired and whose controller host is not recorded.
2. Mark the task done or cancelled, or let the fresh-continuation
fallback resolve the attention of a live task.
3. Run the execution-finalization worker with the issue recovery workers
for 30 minutes.
4. On master, each task gets about 40 to 60 new attention rows. The
writer publishes thousands of signals.

## What Changed

- `surfaceNativeControllerNamespaceAttention` (shared helper): before it
creates attention, it checks for a resolved incident for this run that
was resolved at or after the owner's current lease expiry. If one
exists, it writes nothing.
- New `settledNativeControllerNamespaceAttention` SQL predicate. It is
shared by the helper, the release candidate page, and the release
deadline query, so all three use the same rule.
- `settleFailedRunOwner` (expired-owner path) does not surface attention
when the task is `done` or `cancelled`. The restart-recovery claim path
has the same guard.
- `releaseAbandonedNativeOwners` and `nextAbandonedNativeOwnerReleaseAt`
skip foreign owners whose task is finished or whose incident is settled.
A pass that has nothing to do selects nothing, and no immediate deadline
is set again. Local owners keep the normal proof-based release.
- The owner release proofs did not change. A foreign owner is still
never released without proof.
- Tests:
`server/src/__tests__/native-failed-run-owner-release-idle.test.ts`
(real PostgreSQL, real coordinator and workers, fake clock, statement
and signal counters) and three new cases in
`native-failed-run-owner-release.test.ts`.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/native-failed-run-owner-release-idle.test.ts
src/services/native-runtime/native-failed-run-owner-release.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/services/native-runtime/native-restart-recovery-isolation.test.ts`:
80 tests pass.
- Red on master, green with this change. The test seeds one done task
and one cancelled task (each with an earlier resolved `source_terminal`
incident) and one live task, then runs 30 idle minutes:
- master: 126 attention inserts, 249 attention writes, 7,024 signals,
8,025 statements.
- this branch: 1 attention insert (the live task, surfaced once), 2
attention writes (that insert and the fallback's resolution), 36
signals, 350 statements. All of this is startup work. Nothing repeats.
- After the fresh continuation settles a live task's attention, six
unrelated finalization wakes over 30 minutes:
- master: 3 attention inserts, 90 signals, 551 statements, and the
release deadline stays due.
- this branch: 0 attention writes, 0 run writes, 6 signals (the test's
own wakes), 153 statements, and the release deadline is null.
- New unit cases: a done or cancelled task never surfaces, and surfaces
once when it is reopened. A settled incident stays settled over repeated
passes. A renewed authority (a new lease expiry) surfaces one new
incident.
- The three new unit cases fail on master and pass on this branch.
- `tsc --noEmit` for `server` passes.

## Risks

- Low risk. The change only removes writes. It does not weaken a release
proof, an authorization, a budget, a pause, or an approval check.
- An incident that a person resolved or dismissed after the lease
expired is not made again for the same authority. A renewed authority (a
later lease expiry) still makes a new incident.
- A superseded incident (status `cancelled`) still shows again when the
superseding action clears. This keeps the current behavior.
- The historical rows that the loop made are not removed. This change
only stops new rows.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Refs #21621, #21616, #21617

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 17:50:45 -06:00
28e38da7e6 fix(server): wake native result commits from durable work (#17748)
## Thinking Path

> - Paperclip manages AI agents and preserves their results through
failures.
> - Native completion must save workspace output before it commits the
accepted result.
> - The orphan recovery pass repeatedly revisits this work even when the
subsystem is idle.
> - Accepted results, workspace receipts, retry dates and leases already
provide durable truth.
> - This PR wakes a scoped worker from those writes and schedules only
real unfinished work.
> - It preserves existing finalization authority and keeps later
projection and resource recovery separate.

## Linked Issues or Issue Description

Refs #17734 and #17717. This is slice 53 of the existing sleep stack,
targeting master after parent #17734 merged at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`.
A current search found no duplicate accepted-result commit PR.

**What problem does this solve?**
Native accepted-result recovery currently depends on periodic orphan
scanning. Quiet instances keep doing work to discover that no result
needs a commit.

**What improvement do you propose?**
Use the shared delivery coordinator with transaction-bound producer
signals. Reconstruct unfinished results at startup. Preserve saved retry
and lease deadlines. Remove only uncommitted-result execution from the
periodic pass.

**Why is this useful?**
The accepted-result worker becomes quiet with no work or retained
completed history. Real results and completed workspace operations
reactivate it without a polling delay. This does not claim that the
whole server or database can sleep yet.

## What Changed

- Track the initial result-commit sweep through shutdown without
blocking HTTP readiness on recovery locks.
- Keep ordinary company-only finalization hints from reconstructing
dormant result work. Existing company settings and dedicated owner
signals remain supported.

- Add a native result-commit worker. Bind candidates by company, issue,
run, accepted result and a successful workspace receipt. Exclude
committed, terminal, held and pending-effect work.
- Signal both accepted-result writers and workspace receipt writes
before their transaction changes. Extend the existing finalization
fanout for retry, lease, effect and completion transitions.
- Add an optional atomic due-date check to the existing finalizer claim.
A lease collision or later retry does not consume a failed attempt.
- Signal native retry cancellation so it removes the saved wake after
commit.
- Register through the existing heartbeat suppression and shared
delivery lifecycle. Preserve outer commit/rollback, drain/resume,
settings activation and shutdown behavior.
- Delegate only uncommitted-result execution from the periodic orphan
pass. Keep committed projection and resource settlement recovery
enabled.
- Add real producer/lifecycle tests and a detailed audit. No new tables,
provider branches or per-feature timers.

## Verification

- Current local source `2af88fa55f961a74340880240019f6cd31e21bf1`
incorporates the parent with a normal merge. All 1,203 parent-only, nine
owned-only and 12 automatic shared paths are preserved. The only manual
conflict keeps the new result fanout inside the existing ordinary
issue/run scope guard. The complete package tree matches parent profile
80 byte-for-byte.
- Controlled real-database regression: the original result-only unscoped
fanout produces three candidate SELECTs for routine company hints. The
same zero-query assertion passes after restoring the scoped fanout. A
real workspace receipt and accepted result then commit exactly one
decision and return to quiet. Red and green restore production bytes
exactly. Twenty cases outside the focused pattern are not counted as
passing there.
- The complete main selection passes **1,746 cases in 16 files**,
without skips, in 277.97 seconds. This includes all 21 result-worker
lifecycle cases and 82 startup cases, including held result recovery,
HTTP readiness and shutdown joining. Separate process recovery passes
410 cases in 162.42 seconds and external-chat response passes 273 in
53.41 seconds. Total: **2,429 distinct passes in 18 complete files**,
with no skips. Normal merge `998239b019defe52e7b5d64f884de4e871ccd736`
adopts the actual parent squash and preserves this complete tested child
tree.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Fifteen dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only calls
use the existing fresh-per-call recovery helper. All temporary imports
restore byte-exact, and every selected test hash matches after
completion. No assertion or timeout is relaxed.
- Runner validation belongs to the unchanged parent package tree: 130
passing installation/compatibility/transport cases, 22 installed
ACPX/Grok contract cases and 26 native-provider Rust cases. One existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. This does not claim live provider qualification.
- Earlier Greptile 4/5 on `656f6722be1aedf3d52b7f853956e15838204278`
correctly identified startup blocking on result recovery. The prior
branch `108cf0dc8296bf451b721ddac55ecbcd035d00f0` already tracks its
ready promise through the existing lifecycle owner. Current full startup
acceptance verifies the fix in the merged context. The later Greptile
service error is not a pass; current-head acceptance now passes: all 57
checks are clean, including protected CI and security. Greptile gives
5/5 on `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`, with no actionable
findings. No Ponytail verdict is present under the authorized
missing-review policy.
- The owner audit at
`doc/plans/2026-10-26-native-result-commit-sleep-audit.md` retains
earlier failures and their disposition. These checks do not claim a full
local repository suite, deployed app/database suspension, expanded Cloud
wake coverage or measured cost savings.

## Risks

- Process-local signals cover instrumented application writes in the
owning server process. They do not cover independent SQL writers or
another replica without supported ingress. Durable startup
reconstruction remains enabled.
- The initial native decision and its run projection are separate
transactions. The periodic reconciler still repairs already-committed
results and releases resources. This PR must not be used to infer that
committed means fully settled.
- Candidate discovery adds bounded, narrow reads at startup and after
signals. No large-fleet query benchmark is claimed.
- Existing Cloud alarm coverage, host sleep safety and resource blockers
are unchanged. No deployed sleep or savings claim is made.
- Parent #17734 has merged. This PR targets master at
`a019d00a393b57f531afe8dd8ab0b8a1820d1c4c`. Fresh current-head CI and
review remain required before merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 04:53:11 -05:00