Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-13 15:55:46 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,523 Commits 2,047 Branches 2,456 Tags
e2b337b330f2a8ea7f7ffb915cf19ca5faf712cf
Commit Graph
9 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 d33b3ac263 fix(wake-queue): save temporarily refused work intents and promote them through one path (#19956) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Users send work to agents as assignments, comments and Retries. The
wake queue delivers each one as a run
> - When an admission check refused a wake for a temporary reason, the
wake often ended as a terminal skip, so the work was lost
> - Restart-chaos waves found dropped comments and Retries (RC-37,
s7532) and double delivery (RC-53). Each fix covered one hold type
> - The durable ownership plan (section E) asks for one outbox rule: a
closed set of terminal outcomes, a saved deferral for every temporary
refusal, and one promotion path
> - This pull request adds that rule. It saves messages refused by a
pause, promotes deferred intents from every hold-release event through
one function, and adds a reconciler backstop
> - The benefit is that an accepted message is delivered exactly once
after a pause lifts, and the reconciler reports any intent that is stuck
or dropped

## Linked Issues or Issue Description

Refs #19831 (plan, section E), #19838 (inventory, wake-queue rows and
section 1b), #19927 (Phase 2 data: the I4 false positive), #19702,
#19741, #19829, #19826. Builds on #19944 (section D, merged).

**What happened?**
- A user commented while the agent was paused, or while the company was
paused. Admission refused the wake. Nothing saved the message as
pending, so when the pause lifted, nothing delivered it.
- Hold-release events woke different subsets of workers. A saved intent
then waited out a 30-second refusal backoff after its hold cleared.
- The I4 reconciler check matched free text with a regular expression.
Phase 2 found a false positive: a delivered retry whose error said
"retrying once".

**Expected behavior**
An accepted work intent ends only as delivered, consumed by a later run,
task cancelled, or hard stop. Every temporary refusal saves the intent
as `deferred` with its reason and its release trigger. Every
hold-release event promotes deferred intents through one function
(invariant 4).

**Steps to reproduce**
Pause an agent. Post a user comment on its task. Resume the agent. On
`master` the comment is never delivered as a wake.

## What Changed

- New `server/src/services/work-intent-outbox.ts`:
- `WORK_INTENT_TERMINAL_OUTCOMES`: the closed set of terminal outcomes.
- `TEMPORARY_REFUSAL_TRIGGERS`: maps each temporary refusal code to its
release trigger.
- `promoteDeferredWorkIntents`: the one promotion function. It removes
the 30-second refusal backoff from the matching deferred intents. It
then wakes run dispatch, the saved-receipt worker and stranded-work
recovery, and hints each saved intent of the issue.
- `heartbeat/queue.ts`: the #19829 save-as-deferred block is now a
shared helper. A user's message refused by a paused agent, a paused
company, or a budget pause that is not a reached hard stop is saved with
`executionWait: { reason, trigger, message }`. The caller still gets a
409. The message is saved only when the task's latest run is terminal,
because the release drain promotes a saved message behind that run. A
task that never ran keeps the message in the task, and stranded-work
recovery starts the task once the pause lifts. A `pending_approval`
agent and a task-tree hold are not changed here: the release drain still
fails a promoted wake for a pending approval, and tree holds have their
own resume path.
- Hold-release events call `promoteDeferredWorkIntents`:
  - accounting settled (`run-cost-accounting.ts`);
  - environment lease released (`environments.ts`, two sites);
  - native run projected terminal (`native-run-projection.ts`);
  - budget pause lifted (`budgets.ts`);
  - company reactivated (`companies.ts`);
  - task-tree hold released (`issue-tree-control.ts`);
  - agent lifecycle ready (`agent-lifecycle-driver.ts`).
- Reconciler (`recovery/ownership-registry.ts`,
`ownership-invariant-report.ts`, `native-finalization-reconciler.ts`):
- registers three new kinds: `wake_request_deferred`, `queued_run` and
`scheduled_retry_run`;
- I4 now reads the closed refusal code (the wake `reason`) of accepted
intents, not free text;
- a new I4 check reports a deferred pause refusal whose pause has lifted
for more than 5 minutes. The worker then promotes it as a backstop;
- a lifted pause inside its 5-minute grace window returns its deadline
through `nextDueAt`, and the worker also wakes on agent-lifecycle
signals (company and agent pause changes). So a missed release event is
still checked.
- `doc/SPEC-implementation.md`: adds the rule "Accepted work intents are
delivered exactly once".

## Verification

Each focused file ran alone in one vitest process (`--maxWorkers=1`) on
embedded Postgres. Results at `2875c85fd2`:

| Question | Loop | Elapsed | Result |
|---|---|---|---|
| Is a comment refused by a pause saved, then delivered once after the
pause lifts? | `heartbeat-held-comment-wake.test.ts -t "pause lifts"` |
8 s | 2 pass. With `queue.ts` from `master`, both fail: no intent is
saved |
| Does I4 use the closed code, and does it catch a stuck deferral? |
`ownership-reconciliation-report.test.ts` | 45 s | 20 pass. This
includes the Phase 2 "retrying once" false positive and the case
"paused, then lifted" |
| No regression on the paths that changed | held-comment (15),
archived-company guard (8), undispatched execution wait (32), deferred
receipt work (45), recovery pause durability (16), native run projection
work (55), tree-control routes and services (19, 1, 7) | 8–334 s each |
All pass |

- Exactly-once: each pause test runs two promotions (a release event,
then the reconciler backstop) and many dispatch passes. It asserts one
delivery. The existing consumption tests (RC-53) still pass.
- Server package typecheck passes.
- Speed: nothing new runs before `turn.start`. The new admission work
happens only on a refusal. Promotion runs in the release transaction and
in the background reconciler.
- Not run here: chaos waves. The coordinating session schedules them on
the shared harness.

## Risks

- `promoteDeferredWorkIntents` moves `updated_at` back for deferred
intents that were refused in the last 30 seconds. A busy company with
many release events can retry refused intents more often. Each retry
re-runs normal admission, and the refusal resets the backoff.
- More release sites now wake run dispatch and stranded-work recovery.
These notifications are coalesced per company.
- Lifting a pause is now the operator's explicit authorization to send
what the pause refused, as Send is after a Stop. When the pause stopped
the task's run, the lifted pause issues the task's next explicit wake,
which adopts the saved message once. Other messages queued behind an
acknowledged Stop keep the existing rule. If Stop recovery still holds
the task, the message waits for that recovery.
- An agent pause's stop sweep keeps the messages that the pause deferred
(`preservePauseDeferred`). Terminate and cleanup stops still end them.
- The backstop check reads only pause refusals (`company.inactive`,
`agent.not_invokable`), because those holds can be checked exactly.
Other deferred kinds keep their existing promotion paths.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 23:43:17 -06:00
DottaandClaude Opus 5.5 e620ce3678 fix(heartbeat): deliver each accepted follow-up exactly once through restarts and admission holds (#19741) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can comment, click Retry, or rely on automatic recovery while
an agent works on a task
> - Restart-chaos waves showed accepted follow-ups being dropped, or
delivered twice, around restarts and temporary company admission holds
(accounting still settling after a restart)
> - A dropped comment means the user's request is never acted on. A
duplicate reopens work that is already done
> - This pull request makes accepted work wait through a temporary hold
and resume when it clears. It also stops a comment from being promoted
again after a completed run already received it
> - The benefit is that each accepted follow-up is delivered exactly
once, eventually, unless the task is cancelled or a reached hard stop
applies

## Linked Issues or Issue Description

Refs #19702 (never-started tasks during a temporary hold), #19700
(bounded accounting hold), #14910 (stale reopen from a deferred wake; no
file overlap).

**What happened?**
Restart-chaos waves found these shapes:
1. **Comment wake cancelled.** A comment was saved behind a run and
promoted when the run ended. A temporary company hold refused the
unstarted queued run, and claim cancelled it terminally. The recovery
wake that followed was cancelled the same way, and the task rested
`blocked` without the comment.
2. **Dead Retry.** A Retry click during the hold stayed
`deferred_issue_execution` after the hold cleared. Nothing woke it, and
the issue held no execution lock.
3. **Refused assignment.** A new task's first wake was refused during
the hold. Nothing woke stranded-work recovery when the hold cleared, so
the task stayed `todo` with no run.
4. **Silent comment after recovery.** A comment was saved behind a run
that restart recovery finalized. A company-scoped hint waited out the 30
s deferred-receipt rescan, so the task looked done with the message
unread.
5. **Comment delivered twice.** A run whose provider process was lost
had a saved comment. Its retry started after the comment, received it in
its prompt (task history), did the work, and finished the task. The
saved wake was then promoted anyway, which reopened the done task.

**Expected behavior**
An accepted follow-up (comment, Retry, recovery) is delivered exactly
once, eventually, unless the task is cancelled. A temporary refusal
never ends in a terminal cancel, skip, or block. Reaching a hard stop
remains the board's decision.

## What Changed

- `heartbeat/queue.ts`: on a budget block at claim, an unstarted queued
run stays queued when the block is temporary. A reached hard stop or
blocked unpriced usage still cancels it.
- `recovery/service.ts`: stranded-work recovery waits out a temporary
hold for any task. It does not escalate to `blocked`. This extends
#19702 beyond never-started tasks.
- `budget-block-ownership.ts`: a shared `isBoardOwnedBudgetBlock`, moved
out of the recovery service. `budgets.ts` is unchanged.
- `run-cost-accounting.ts`: settling a run's accounting wakes queued-run
dispatch, the deferred-receipt worker, and stranded-work recovery. Held
work waits passively; this is what resumes it.
- `deferred-execution-notifications.ts`: new
`notifyDeferredWakesForIssue`. `native-run-projection.ts` (a run
reaching a terminal status) and `environments.ts` (lease release) now
hint each saved wake of the issue by id, so delivery does not wait out
the company rescan interval.
- `wake-queue/adapters/postgres.ts`: release treats a saved comment as
delivered only when a succeeded run of the wake's agent on that issue
started after the comment's latest revision, and either received the
comment as its own wake input (`wakeCommentIds`) or was a retry of the
interrupted turn whose task history included it. In that case it cancels
the saved wake with "Queued messages were already delivered to a
completed run" and leaves the task as it is. Task history alone never
consumes a message saved for its own turn. One query checks the whole
batch.
- `run-cost-accounting.ts`: settling late receipt debt also sends the
hold-release notifications.
- `doc/execution-semantics.md`: documents the rule.

## Verification

Red on `master` and green with this change, on embedded Postgres:
- `server/src/__tests__/heartbeat-held-comment-wake.test.ts`:
- A promoted comment refused by the hold stays pending, with no
cancelled run and no `blocked` task. It is delivered once after the hold
clears through real `accountRunCost`.
- A Retry clicked during the hold starts exactly once after accounting
settles, which wakes its owner.
- A new task whose assignment was refused during the hold starts once
accounting settles, which wakes stranded-work recovery.
- A comment that a later succeeded retry already received as history, or
that a later run received as wake input, is consumed without reopening
the task.
- Guard tests (green on both):
- A comment is still delivered in each of these cases: the later run
never received it, it was edited after that run started, it was posted
while the run was already running, or a non-retry run only saw it as
history.
  - A task whose continuation was refused waits out the hold.
  - A reached hard stop is still refused.
- `server/src/__tests__/native-run-projection-work.test.ts`: projecting
a recovered run terminal hints the saved comment's own wake (red on
`master`).
- Related suites pass locally (14 files, 482 tests): wake-queue, queued
comments routes, refused assignment wake, native run projection.
- Live replays (local, unmasked, kept instances):
- `--replay w-20261107T031005-s7501/t04` (Claude C1, K1): PASS. The
comment's run was created at 03:56:24 during the hold, stayed queued,
started at 04:06:24 when the hold cleared, and succeeded. It is the only
run carrying the comment.
- `--replay w-20261107T002911-s7401/t01` (before the per-wake hint): it
reproduced shape 4. Run 592d772f was killed 250 ms after
`run.result.accepted`, released and finalized at restart, and the task
rested `done` with the comment still saved. Calling the existing sweep
on the kept database promoted it, which confirmed that the missing piece
was the trigger.

## Risks

- A queued run can now sit queued for the length of a temporary hold,
which #19700 bounds. It does not hold the issue's execution lock, so
later comments still coalesce into it.
- The delivered-comment rule depends on the run's recorded prompt
context. A succeeded run that received the comment in its history counts
as delivery, even if the agent did not act on it. This matches
exactly-once delivery semantics.
- Settling accounting now emits three extra notifications per run. They
are coalesced per company.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, run as an agent in
Claude Code with tool use (shell, file editing, local test execution,
live restart-chaos replays).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-07 00:24:09 -06:00
DottaPaperclipClaude Opus 5.5
 		35d086213e fix(native): explain failed accepted results on the run (#19091) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run ends when the server projects the accepted result onto
the run row.
> - Operators read `errorCode` and `error` on a failed run to learn why
it failed.
> - Three native writers can set a run to `failed` without these fields.
#19063 repairs the finalizer retry-exhaustion writer.
> - The two result-projection writers still set `failed` with no cause
when the provider failure is not classified.
> - This pull request makes both projection writers record a cause when
they project `failed`.
> - The benefit is that, with #19063, no native writer can leave a
failed run unexplained.

## Linked Issues or Issue Description

Refs #19063.

**What happened?**

A staging soak found 11 failed native runs with `errorCode` null and
`error` null. In each run the provider turn succeeded. Workspace
copy-back then failed three times, and the finalizer retry-exhaustion
path set the run to `failed`. #19063 fixes that writer. An audit of
every `heartbeat_runs` status writer found two more writers with the
same defect. `finalizeNativeRun` and `projectCommittedRun` project an
accepted result with `runTerminalState: "failed"`. They write the error
fields only when a classified provider failure exists.

**Expected behavior**

A run with status `failed` always has an `errorCode` and an `error`. A
cause that is already recorded stays.

**Steps to reproduce**

1. Accept a native result with `runTerminalState: "failed"` and no
`turn.failed` event.
2. Call `finalizeNativeRun` with `projectRunStatus: true`, or project
the committed run through `projectCommittedRun`.
3. Read the run. On master its status is `failed`, and its `errorCode`
and `error` are null.

## What Changed

- Add `explainFailedRun` in
`server/src/services/native-runtime/native-run-failure-explanation.ts`.
It builds SQL SET expressions. They keep a non-empty existing
`errorCode` or `error`. Otherwise they write `native_run_failed` and a
readable message. By default the expressions leave a run that is already
`cancelled` unchanged, as the status writers do.
- `finalizeNativeRun` uses the helper when it projects `failed`. The
existing provider-failure and recovered-failure values remain the
preferred causes.
- `projectCommittedRun` uses the helper for a `failed` accepted result.
This writer sets status unconditionally, so it does not use the
cancelled branch.
- Writer audit: all legacy and heartbeat callers of `setRunStatus*` that
pass `failed` already set both fields. The other direct native writers
also set both fields.

## Verification

- Red, with the source change stashed: 3 new cases failed with
`errorCode: null`. These were the finalizer projection, recovery
projection with no cause, and recovery projection with an empty error.
The control case with a recorded cause passed.
- Green: `npx vitest run
src/__tests__/native-run-projection-work.test.ts
src/services/native-runtime/native-run-finalizer-telemetry.test.ts
src/__tests__/native-status-arbiter-corpus.test.ts` (225 passed).
- Wider run: all `server/src/__tests__/native-*.test.ts` and
`server/src/services/native-runtime/*.test.ts` (7416 passed). Two
failures come from the local environment. `plugin-runner-probe` needs
the Rust runner source. `native-ingress-bootstrap-retry` needs the
controller-host OpenCode proxy.
- This change was not exercised on staging. It changes only fields that
are written at terminal projection.

## Risks

- Low risk. The change affects only rows that are projected to `failed`.
A recorded cause is never replaced. Succeeded and cancelled projections
are unchanged.
- The staging failures need #19063 too. This PR alone does not change
their writer.
- The cause behind the staging copy-back failures is still unknown. The
sandbox command reports the redacted
`remote_workspace_operation_failed`, which also appears in a separate
signature. This PR does not change that diagnostic.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, browser reads of staging run records).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-04 17:40:05 -06:00
DottaandPaperclip fb6554ac62 fix(native): settle source wake with terminal projection (#18804) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Each agent wake request can bind to one heartbeat run.
> - Native recovery can finish and project a run after its live executor
has stopped.
> - The legacy completion callback settles the wake, but native terminal
projection did not.
> - A succeeded run could therefore leave its exact wake claimed with no
finish time.
> - This pull request settles that claim in each native terminal
projection transaction and reuses durable projection recovery for old
committed rows.
> - The benefit is consistent run and wake receipts without a provider
replay or a new sweeper.

## Linked Issues or Issue Description

Refs #13856 and #10109. These open PRs describe orphan claimed wakes and
cleanup after terminal persistence. This change fixes the native
projection owner and uses its existing durable worker. It does not
modify those contributor branches.

**What happened?**

A native run can be accepted and committed, then reach a terminal state
through recovery. Its source wake stays claimed because the live
completion callback does not return to settle it.

**Expected behavior**

Terminal projection must settle only the same company's exact source
wake bound to that agent and run. Completed, failed and cancelled
outcomes must retain their meaning. Other live, queued, cancelled,
unbound or newly rebound wake records must remain unchanged.

**Steps to reproduce**

Commit an accepted native result in the real database fixture. Leave its
source wake claimed. Call native finalization with run-status projection
enabled, without the live completion callback. Before this change, the
run becomes succeeded and finished while the exact wake remains claimed.

## What Changed

- Settle the exact source claim inside native finalization, committed
recovery, and exhausted-finalization projection transactions. Fence it
by wake ID, company, agent, run and current claimed state.
- Lock the matching claim and register settlement notifications before
the write. A failed transaction rolls back both run projection and wake
settlement.
- Let the existing durable projection worker discover a terminal run
with an unfinished exact claim. A repaired committed row leaves the
pending set. No new timer, sweep or table is added.
- Add real database regressions for native recovery, all three terminal
outcomes, old committed receipts, idempotence, scoped denial controls,
notification ordering and rollback. Cover provider/workspace
finalization exhaustion and denial when the caller does not own terminal
projection.
- Preserve the existing provider, accounting, workspace, permissions and
result contracts. No API, schema, user command, access-map entry or
documentation contract changes are needed.

## Verification

- The new native coordinator/finalizer regression failed before the
production change: the run was succeeded and committed, but the bound
wake was still claimed.
- The initial focused database run did not reach the assertion because
the SDK was unbuilt. A second attempt hit sandbox loopback restrictions.
A third exposed skipped package library hydration. These setup failures
were corrected before the meaningful red was captured.
- Real PostgreSQL projection and response suites passed 89 tests before
the independent review correction. The reviewer requested notification
intent registration before mutation. The correction adds a
notification-order and rollback case. All 90 tests now pass across both
suites.
- Server typecheck passed after building the isolated checkout's Runner
and SDK prerequisites. Fresh corrected-source server typecheck and
production server build also pass. Full repository checks run in
current-head CI. The earlier serialized-9 CI failure was the
independently repaired inbox snapshot fixture in #18798 (expected nine
reader calls, observed eight). A normal merge of the containing master
supplies that fix; the native suites plus the repaired inbox suite now
pass 96 tests together at merge head
`456c92ef8d60c3fcbeaf66ff8010a7ebd3d1109d`. Server typecheck passes.
That head was superseded by the reviewed retry-exhaustion correction
below. The subsequent review found a third terminal write in retry
exhaustion. Real database provider and workspace exhaustion regressions
failed with claimed/null-finished wakes before the fix. The same fenced
helper now runs inside that transaction. At final source head
`9718d4c2ec6066accf549dc6fea795891f6b90ec`, all 100 focused tests,
server typecheck and production server build pass, including
projection-disabled denial and rollback of run/coordinator/wake on
notification failure.
- Final current-head CI passed: 54 success, 2 skipped, and the neutral
Paperclip Review at 5/5. Greptile also reports 5/5. Required ci / verify
and ci / e2e passed. All review threads are resolved.
- No live task write, provider work, accounting repair, deployed replay
or manual rollout was performed.

## Risks

- A stale wake binding must never be settled. The update requires the
exact wake ID, company, agent, run and claimed state. Tests cover
unbound and newer-run bindings, foreign company and agent, queued and
cancelled records, and retained live ownership.
- Old claimed debt now participates in the existing projection
predicate. It uses the source wake primary key and stops matching after
settlement. Existing quiet-worker tests remain in the focused suite.
- Wake settlement must share the projection transaction. Notification
failure and outer rollback tests preserve both original rows.
- Interruption, timeout, retained historical terminal-failure
coordinators, and delayed response presentation keep their current
owners. This change does not infer or replay provider work.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, Extra High reasoning, with source
inspection, code execution and regression testing. Context-window
capacity is not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked existing public issues or described the issue in
this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket ID
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation where needed; existing
contracts are restored
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-03 13:17:16 -06:00
DottaandPaperclip baf651c8d0 perf(heartbeat): drive retry promotion from durable deadlines (#18387) 
Reviewed and verified at f1c48c924d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
 2026-10-31 09:07:52 -05:00
DottaPaperclipCodex
 		c8b7afee06 fix(server): schedule native environment lease settlement (#17801) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/24/changes/paperclip/24
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the resources used by their work.
> - Native run completion has several independent settlement duties.
> - The recurring run reaper repeated environment-release discovery for
completed native history.
> - Existing lease rows already record releases that still need work.
> - This pull request wakes the existing release owners from committed
changes and schedules only unfinished release debt.
> - Completed history becomes quiet while stop evidence, workspace
preservation, and retention policy keep their existing owners.

## Linked Issues or Issue Description

Refs: #17793, #17717.

**What problem does this solve?**

Periodic native finalization replay calls environment release even when
no lease needs work. This adds recurring database traffic to an idle
instance.

**What is the proposed solution?**

Register one scoped worker on the existing delivery coordinator.
Discover work from native finalizations and existing environment leases.
Wake it from native, lease, and settings transactions. Keep the release
owner's saved pending-cleanup deadlines. Keep retries only while a
concrete lease still needs settlement.

**What alternatives have you considered?**

A slower unconditional scan still queries idle history. A new job table
duplicates existing lease truth. Replacing the compound callback would
also change runtime-service and agent-status duties; those remain
separate slices.

This is slice 58. Parent #17793 has merged. This publication targets
master and preserves that landed prerequisite. Fresh current-head checks
and review remain required.

## What Changed

- Keep ordinary company-only finalization hints out of environment
settlement. Prove zero candidate SQL after quiet and preserve real lease
delivery.

- Reuse the existing durable destroy claim for run-owned cleanup,
excluding concurrent attempts and fencing late receipts while preserving
cleanup recovery.

- Scope lease cleanup notifications to their company/run so settlement
does not rediscover the entire backlog after each release.

- Read only `nativeWorkspaceSync` and `workspaceExportRetry` JSON fields
when discovering environment cleanup. Preserve saved run data and
existing release decisions.

- Reuse the environment runtime's release predicate for discovery. Add
no table or independent timer.
- Wake from existing finalization, projection, lifecycle, recovery,
lease, and settings transaction boundaries. Simple lease acquisition now
uses a transaction too.
- Preserve recorded retention policy, pending-cleanup deadlines,
original lease/provider ownership, and current release guards.
- Re-read durable lease state after a release. A logged or swallowed
per-lease failure does not discharge the obligation.
- Delegate only periodic environment release. Preserve default
live/manual behavior, runtime-service release, and agent-status
callbacks.
- Add an owner/producer/lifecycle audit and real database tests.

## Verification

Accepted production source: `9596fa022b57feea026c4728c80bdc36037b804f`.
Publication head: `fb252c8b0e52ce355e7d98db2e392e4831f99c61`. The actual
parent squash is adopted by a normal merge with the complete accepted
child tree unchanged. Only audit text follows.

- **4,172 distinct cases in 64 complete server files pass**, plus **20
selected chat cases** (4,192 selected total). The main batch reports
3,440 passes and 49 skips. Both complete Stop/read files then pass 51 +
6 cases on separate fresh databases through their supported override.
Their eight pure cases are counted once; skipped cases are never counted
as passes. Separate process recovery passes 410; review response passes
273. The other 1,794 chat cases were not run locally.
- **Quiet producer proof:** the controlled unscoped baseline fails with
three candidate SELECTs after three routine company-only hints. The
scoped case performs zero, releases a real eligible committed-run lease,
and returns to quiet. The same test hash spans red and green.
- **Static gates pass:** database build; server typecheck/build; module
boundaries; migration order; current Copilot profile identity. Packages
equal the accepted parent byte for byte.
- **Fresh fixtures:** 39 prepared PostgreSQL 17 databases plus fresh
helper-owned databases. The schema has all 412 journal entries through
index/tag 0413_tired_kronos. All 65 original test hashes restore
exactly; the indirect Stop fixture is also hashed. No assertion or
timeout changes.
- **Preserved records:** original 49 skips and their completed coverage
remain documented. Parent handoff timeout and prior chat-idle failure
records remain in their owning audits. This does not claim full
repository, full chat, live provider, or deployed sleep acceptance.
- [Current integration and native lease
audit](https://github.com/paperclipai/paperclip/blob/fb252c8b0e52ce355e7d98db2e392e4831f99c61/doc/plans/2026-10-26-native-environment-settlement-sleep.md#current-master-integration-and-native-lease-quiet-proof-2026-10-29).
Fresh CI, security, and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/6f3a0b8de149fb5620f15909473eb7c4455b9824/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ae9bff1c06e34702f64a399e58b45a657c133bd1/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `6f3a0b8de149fb5620f15909473eb7c4455b9824`. This remains a
single-duty PR stacked on #17793.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-native-environment-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/7e619453567e078a7d745f28bfe8e5ede571f282/doc/plans/2026-10-26-native-environment-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Existing lease owners retain destruction claims, provider identity,
stop evidence, retention policy, and pending-cleanup deadlines.
Incorrect changes here can release a live resource or lose a cleanup
obligation.
- Logged or swallowed release failures remain concrete debt. Real
retries must retain their saved floor across unrelated hints.
- In-process signals do not cover independent SQL or other replicas
without ingress.
- This delegates only periodic environment settlement. Runtime-service
and agent-status duties remain separate. Existing Cloud blockers remain;
no deployed sleep, database suspension, external alarm coverage, or cost
savings is claimed.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL tests, and GitHub tools. The exact runtime model identifier
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 06:37:48 -05:00
DottaPaperclipCodex
 		829f9e7eaf fix(server): settle Muse terminal assignments from committed changes (#17793) 
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/12/changes/paperclip/12
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its external-agent module owns Muse assignments, operation receipts,
and worker holds.
> - Running assignments and retained terminal history must not cause an
empty maintenance scan every five seconds.
> - Terminal run changes and operation results already have durable
database transactions.
> - This pull request wakes the existing maintenance worker from those
commits and reconstructs work at startup.
> - The worker retries only concrete settlement debt. It retains
qualification deadlines and authority checks.

## Linked Issues or Issue Description

Refs: #17772. This is slice 57. Parent #17772 has merged.
Related: #17000, #17525, #17752, #17758. The publication targets master
and preserves the merged parent.

**What existing behavior does this improve?**

Muse terminal assignment settlement and the existing Muse maintenance
worker.

**Current behavior**

A running assignment can keep maintenance scanning every five seconds.
Terminal reconciliation rewrites unchanged history. Native finalization
also calls that reconciliation during periodic replay. Discovery
excludes a revoked original binding even when its assignment still needs
settlement.

**Proposed behavior**

Committed run changes and operation receipts wake the existing worker
with the original company, run or binding identity. Startup reconstructs
actionable work from existing rows. Maintenance processes bounded
batches and disarms after settlement. Default live and manual callbacks
remain. The periodic callback delegates this one duty.

**Reason and benefit**

Configured but idle instances stop querying for this obligation. Late
receipts, restarts, drain and historical bindings still have recovery
paths. No table or independent timer is added.

**Breaking changes**

No schema, endpoint or provider protocol changes. Explicit manual
reconciliation retains its unbounded default. This does not alter Cloud
sleep admission or claim that all instance work can sleep.

## What Changed

- Add an owner predicate shared by discovery and locked settlement.
Fenced private-worker uncertainty remains held without an automatic
retry loop.
- Preserve original binding generation and company/agent/run identity,
including revoked bindings. Fence only the terminal assignment during
reconciliation.
- Extend existing maintenance with scoped hints, bounded batches,
concrete-debt retry, startup reconstruction and quiet drain behavior.
- Register notifications inside existing terminal, projection,
cancellation, operation-result and send-failure transactions. Keep
qualification and OAuth signals explicitly global when coalesced with
scoped changes.
- Serialize only four domain IDs. Real native-finalizer tests caught and
prevent a circular-object serialization failure at this boundary.
- Use existing restore/worktree suppression and settings reactivation.
Delegate both periodic finalizer branches while retaining resource
callbacks and default live/manual settlement.
- Add the producer audit and regression evidence in
`doc/plans/2026-10-26-muse-terminal-settlement-sleep.md`.

## Verification

Accepted production source: `584015018ed4a9d57d2b0f192b75847503b54e53`.
Publication head: `305014b030d6c5eb891c70bc57036f4192d4fdad`. Normal
adoption of the actual parent squash preserves the complete accepted
child tree. Only audit text, a migration-count clarification, and the
proved Photon test extension follow.

- **3,715 distinct cases in 52 complete server files pass**, plus **20
selected chat cases** (3,735 selected total). The main 49-file batch
passes 3,008; process recovery passes 410; review response passes 273;
the full Photon file passes 24. The other 1,794 chat cases were not run
locally.
- **Quiet producer proof:** moving only the Muse fanout outside its
parent guard produces six candidate SELECTs and fails the regression.
Keeping the guard yields zero and still settles a real terminal run. The
same test hash spans the controlled red/green pair.
- **Photon review evidence:** repeated active callbacks produce zero
settlement hints/candidate SQL. Three concurrent attention recoveries
produce exactly one of each. The existing locked attention guard was
correct; production needed no fix. The parent security check accepted
the resolved finding before merge. This PR keeps the additional
settlement queue in the existing regression.
- **Static checks pass:** database build; server typecheck/build; module
boundaries; migration order; current Copilot identity. Packages equal
the accepted parent byte for byte. No new live provider qualification is
claimed.
- **Fixtures and failures:** 34 fresh PostgreSQL 17 fixtures have 412
applied migration receipts through tag 0413_tired_kronos, plus fresh
per-call helper fixtures. All 52 original test hashes restore exactly.
The additional Photon test hash matches its passing proof. Earlier
remote chat passive-idle failure remains recorded; both passive/missing
cases pass in the new selection. Parent handoff timeout/fresh-file-pass
accounting remains in its audit. No assertion or timeout was weakened.
- [Integration and quiet producer
audit](https://github.com/paperclipai/paperclip/blob/305014b030d6c5eb891c70bc57036f4192d4fdad/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md#current-master-integration-and-quiet-producers-2026-10-29).
Fresh CI, security, and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/f7477933469fdc960f555425dce66eff25a94e54/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/bbd31dea5ef3a4875950a87b4636af8b6bde858e/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `f7477933469fdc960f555425dce66eff25a94e54`. This remains a
single-duty PR stacked on #17772.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-muse-terminal-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/ff196791db6dada2dc04ea9db0d58c5c3beb4a41/doc/plans/2026-10-26-muse-terminal-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- In-process signals do not capture independent SQL or other replicas.
Supported external events still need ingress.
- Pending assignments and operation receipts retain their original
binding identity. A new agent configuration cannot take their ownership.
- Genuine qualification, OAuth, and recovery deadlines remain
obligations. They must not be discarded merely to make the worker quiet.
- The parent coarse-hint guard prevents recurring Muse discovery; real
company/binding authority changes retain their signals.
- No full local repository suite, full chat suite, live Muse or Copilot
qualification, deployed sleep, database suspension, external alarm
coverage, or cost savings is claimed. Existing Cloud blockers remain.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, real
PostgreSQL tests, and GitHub tools. The exact runtime model identifier
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 06:18:40 -05:00
DottaPaperclipCodex
 		8ac5a128e6 fix(server): resolve native finalization notices from committed wakes (#17758) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native finalization can commit before its recovery notice is
resolved.
> - The periodic orphan pass revisits committed runs to resolve those
notices.
> - Retained completed history should not require repeated discovery.
> - This PR moves that one duty to transaction wakes and concrete
failure retries.
> - Existing notice rows reconstruct work after restart. The action
service still owns resolution.
> - Other completion and resource duties keep their current recovery
paths.

## Linked Issues or Issue Description

Refs #17752 (stack parent: committed run projection), #17748
(accepted-result commit), and #17734 (workspace finalization). These are
distinct recovery obligations. This slice incorporates parent #17752 at
`a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, now merged as
`50fe32684d2d4e1db129e3c1bb455571344a0c09`. Related #17724 retires
obsolete exact-session notices for terminal runs with no result and
preserves their replay hold. This slice resolves notices after a result
committed; it does not change that no-replay policy.

**Current behavior**
Committed native finalization recovery repeatedly inspects active
watchdog notices. A notice can also be updated in place to describe a
different run, so resolving a previously selected ID alone is unsafe.

**Proposed behavior**
Wake a scoped worker from relevant committed writes. Reconstruct active
finalization notices at startup. Resolve only the current notice after
the native owner rechecks its binding under lifecycle locks. Stop
scheduled work when no matching obligation remains.

**Reason and benefit**
Notice recovery becomes quiet after settlement and reacts to real
changes without a polling delay. Reused notices remain attached to their
current incident. This does not claim that every remaining server duty
or the host can sleep yet.

## What Changed

- Track the initial recovery-notice sweep through shutdown without
blocking HTTP readiness on task locks.
- Require issue/run scope for ordinary finalization fanout. Routine
company-only hints leave dormant notices quiet; dedicated owner and
explicit settings signals retain their scopes.

- Extract the native finalization notice predicate and resolution owner.
Keep the existing recovery-action service responsible for writing the
resolution.
- Bind candidates by company, issue and run. Require a committed result
and an active or escalated finalization notice. Exclude held ownership,
pending status effects and unfinished run projection.
- Recheck coordinator, issue, run and matching actions under locks. A
newer incident cannot be resolved from a stale candidate snapshot.
- Use the shared scoped worker, transaction signals, retry backoff and
lifecycle handling. Add no table, per-feature timer or provider branch.
- Signal action insertion, update, supersession and resolution;
finalization and effect changes; run projection and ownership release;
issue changes; settings activation; and the recovery route that reopens
an action.
- Preserve no-op resolution quiet. Serialize only the three supported
notification IDs, even when a caller supplies a larger internal object.
- Delegate only notice resolution from the periodic path. Keep
manual/default finalization and other completion/resource work.
- Add real PostgreSQL producer, concurrency and lifecycle coverage and a
detailed audit.

## Verification

- Current local source `730e00a7b007a36d188ca0a328ef050d3a25d1ee`
incorporates the parent with a normal merge. All 1,209 parent-only,
eight owned-only and 12 automatic shared paths are preserved. The two
manual paths retain both chat import sets and scope the new notice
fanout. The entire package tree matches parent profile 80 byte-for-byte.
- Controlled real-database regression: original notice-only unscoped
fanout produces three candidate SELECTs for routine company hints. The
same zero-query assertion passes after restoring scoped fanout. The real
notice writer then creates and resolves an action for a committed run
and returns to quiet. Both phases restore production bytes exactly; the
22 cases outside this focused pattern are not counted as passing there.
- The complete main command passes **1,873 cases in 21 files**, without
skips, in 309.69 seconds. It includes all 23 notice-worker cases, full
presentation/ownership read coverage, 88 startup cases, and the
projection/result/physical-work consumers. Process recovery passes 410
in 163.54 seconds and external-chat response passes 273 in 53.48
seconds. Total complete-file acceptance: **2,556 cases in 23 complete
files**, with no skips. The additional focused chat selection passes 18
cases in 13.44 seconds; 1,796 unselected chat cases do not count. Total
selected passes: **2,574**. Normal merge
`a0f2b3f4348b078219a53d8cc135c84202ba505e` adopts actual parent ancestry
with this complete tested child tree unchanged.
- The chat selection covers the real reply/notice row-lock regression,
Slack and Telegram response recovery, current-authority denial cases,
exact canonical response binding and internal GitHub attachment
navigation. Other cases in the giant chat file remain unselected; this
is not full chat-suite acceptance.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Twenty dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only
invocations use the existing fresh-per-call recovery helper. All
temporary imports restore exactly and all 24 selected test hashes match
after completion. No assertion or timeout is relaxed.
- Parent package acceptance remains applicable: 130 passing
installation/compatibility/transport cases, 22 installed ACPX/Grok
contract cases and 26 native-provider Rust cases. One existing opt-in
closed Pi snapshot case is skipped because its distribution root is
unset. This does not qualify a live provider.
- Failure history and scope are retained in
`doc/plans/2026-10-26-native-recovery-notice-sleep-audit.md`.
Current-head acceptance: all 57 checks are clean, including protected CI
and security. Greptile gives 5/5 on
`2a8777bbed8b1e10810e3d6d99aae3f25b4854ef`, with no actionable findings.
Prior scores do not substitute. No Ponytail verdict is present under the
authorized missing-review policy. No full local repository suite,
deployed sleep, expanded Cloud wake coverage or measured savings is
claimed.

## Risks

- Resolution takes coordinator, issue, run and action locks in the
native recovery order. Contention must preserve the current incident;
the real concurrent-upsert and competing-resolver cases exercise this
boundary.
- Process-local signals cover instrumented application writers. They do
not add support for independent SQL writers or replicas without
supported ingress. Startup reconstructs the existing durable rows.
- Completed finalization is not proof that external conversation
settlement, Muse reconciliation or physical cleanup finished. Those
obligations retain their owners and recovery paths.
- Discovery uses narrow keyset pages. No fleet benchmark, host
suspension or cost reduction is claimed. Cloud wake coverage remains
unchanged.
- Parent #17752 has merged. This PR targets master at
`50fe32684d2d4e1db129e3c1bb455571344a0c09`. Fresh current-head CI and
review must pass before the authorized protected merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 05:35:01 -05:00
DottaPaperclipCodex
 		50fe32684d fix(server): wake committed native run projection from durable changes (#17752) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run has separate result acceptance, task decision, run
status, and resource settlement steps.
> - A task decision can commit before its run status is saved.
> - Recovery currently revisits committed decisions through the periodic
orphan pass.
> - This PR gives the existing run projection repair its own
transaction-driven worker.
> - Existing records reconstruct unfinished work after restart.
Completed history becomes quiet.
> - Other completion and resource obligations keep their existing
recovery paths.

## Linked Issues or Issue Description

Refs #17748 (stack parent: accepted-result commit), #17734 (workspace
finalization), and #17717 (retained session cleanup). Related #17614
preserves terminal notifications. This slice is prepared on current
parent #17748 at `b2efdb2dbbbef5b2227ffa2bd2a9a6cfe3867fe2`; the parent
has merged as `28e38da7e6bdd6af4092dda56bc2be05bed47af5`.

**Current behavior**
The native recovery pass revisits committed decisions to repair the
heartbeat-run status and metadata. A crash between decision and run
writes creates real work, but periodic scanning also revisits retained
history.

**Proposed behavior**
Wake projection repair after committed producer changes. Reconstruct
mismatches on startup and settings activation. Retry concrete failures
through the shared scheduler. Keep the run projection before resource
settlement.

**Reason and benefit**
The repair worker performs no recurring SQL after its work settles. A
real decision, effect settlement, or ownership release reactivates it.
This is one worker migration; it does not establish host or database
suspension.

## What Changed

- Extract the existing projection predicate and write into their
native-runtime owner. The finalizer keeps initial metadata enrichment
and task decision authority.
- Discover accepted, committed, mismatched run projections with company,
issue and run bindings. Exclude pending status effects and held runner
ownership. Read narrow fields instead of whole result payloads.
- Use existing transaction hints, scoped delivery scheduling, failure
backoff and lifecycle handling. Add no tables or per-feature timers.
- Require issue/run scope for ordinary finalization fanout. Routine
company-only hints leave dormant projection quiet; explicit settings and
dedicated producer signals keep their existing contract.
- Connect finalization, pending effects, lifecycle status writes, issue
changes and worktree execution activation. Preserve outer transaction
commit and rollback behavior.
- Lock the current run before deciding which terminal event and
completion time to save. Concurrent live and recovery projection must
not announce the same transition twice.
- Delegate only already-committed projection from the periodic pass. Its
candidate scan uses the same pending predicate to defer remaining
consequences and cleanup until projection finishes.
- Add real PostgreSQL producer, race and lifecycle tests plus an owner
audit.

## Verification

- Current local source `a83b9449233a275176e637c765059f414305d7c1`
incorporates the parent with a normal merge. All 1,208 parent-only,
seven owned-only and ten automatic shared paths are preserved. The only
manual conflict keeps new projection fanout inside the ordinary
issue/run guard. The complete package tree matches parent profile 80
byte-for-byte.
- Controlled real-database regression: the original projection-only
unscoped fanout produces three candidate SELECTs for routine company
hints. The same zero-query assertion passes with scoped fanout. A real
committed result then repairs the run and emits one terminal event
before returning to quiet. Production bytes restore exactly in both
phases. Twenty cases outside that focused pattern are not counted as
passing there.
- The complete main command passes **1,830 cases in 19 files**, without
skips, in 301.14 seconds. It includes all 21 projection cases, 85
startup cases, required quality review and heartbeat run lifecycle, plus
physical-work and result-commit consumers. Separate process recovery
passes 410 in 163.58 seconds and external-chat response passes 273 in
53.26 seconds. Total: **2,513 distinct passes in 21 complete files**,
with no skips. Normal merge `d4b46fd29cb23126d556cc3aeb883e6b41f1a8a5`
adopts the actual parent squash with this entire tested child tree
unchanged.
- Production server typecheck, build, module boundaries, migration order
and source identity checks pass. Seventeen dedicated PostgreSQL 17
databases use the unchanged migration 413 schema. Embedded-only calls
use the existing fresh-per-call recovery helper. All temporary imports
restore byte-exact and all 21 selected test hashes match after
completion. No assertion or timeout is relaxed.
- Parent package acceptance remains applicable: 130 passing
installation/compatibility/transport cases, 22 installed ACPX/Grok
contract cases and 26 native-provider Rust cases. One pre-existing
opt-in closed Pi snapshot case is skipped because its distribution root
is not configured. No live provider qualification is claimed.
- The owner audit at
`doc/plans/2026-10-26-native-run-projection-sleep-audit.md` retains
prior failures and their disposition. Current-head acceptance: all 57
checks are clean, including protected CI and security. Greptile gives
5/5 on `a49a2dac13b0e1c19da5d3d6b435c781703b1c0b`, with no actionable
findings. Old scores do not substitute. No Ponytail verdict is present
under the authorized missing-review policy.
- These checks do not claim a full local repository suite, deployed
suspension, expanded Cloud wake coverage or measured cost savings.

## Risks

- Process-local signals cover instrumented application writers.
Independent SQL writers and replicas without supported ingress are not
newly covered. Durable startup reconstruction remains enabled.
- Projection must precede remaining committed resource settlement. The
real reconciler test proves the periodic path defers cleanup while
projection is pending and resumes it after repair.
- Status decisions, external conversation settlement, watchdog
resolution, Muse reconciliation and resource release keep their own
authority and recovery. A committed result alone is not proof of
completed cleanup.
- Candidate discovery uses narrow bounded pages. There is no large-fleet
performance or deployed sleep claim. Existing Cloud wake coverage and
sleep blockers remain unchanged.
- Parent #17748 has merged. This PR targets master at
`28e38da7e6bdd6af4092dda56bc2be05bed47af5`. Current-head CI and review
must pass before the authorized protected merge.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools and code execution.
The exact serving model ID and context-window size are not exposed in
this session. No subagent was used for this slice.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-29 05:10:16 -05:00