Commit Graph
7 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 c5c587f162 fix(issues): start a deferred children-completed wake the release drain missed (#20180)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent that delegates waits for its children, and is woken when
they finish (`issue_children_completed`)
> - Restart chaos t06 ended with both children done and the parent
blocked forever
> - The children finished while the parent's run was still going, so the
wake was deferred as a follow-up. That run then ended with the agent
setting the parent blocked, and the deferred wake was never started
> - The durable re-drivers for deferred wakes cover only
`issue_blockers_resolved`, so nothing else picks up a missed
`issue_children_completed`
> - This pull request makes both re-drivers cover children-completed
wakes too
> - The benefit is that a parent always hears that its children
finished, even when the release drain misses the deferred wake

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261109T041248-s9501`, task t06 (claude, M1, K2), Daytona per-turn,
master fa7950121d.

**Problem:** parent CHA-10 (all times UTC):
- 04:29:57: run 9cf49f52 started.
- 04:32:35 and 04:32:43: its children CHA-13 and CHA-14 finished.
- 04:32:43: `issue_children_completed` merged into the running turn.
- 04:32:53: that wake was deferred as a follow-up
(`issue_execution_deferred`, `deferred_issue_execution`, no run).
- 04:33:13: the turn ended with the agent setting the parent `blocked`,
a decision made before it saw the children's results.

The deferred wake stayed deferred for the rest of the wave, including
three later restarts. The parent sat blocked with no Retry. The
deferred-receipt worker (`deferred-receipt-work.ts`) and the
stranded-queue resume (`queue.ts`) select deferred dependency wakes with
`issueBlockersResolvedWakePayloadSql`, which matches only
`issue_blockers_resolved`.

## What Changed

- `server/src/services/issue-dependency-wakeups.ts`:
`dependencyResolvedWakePayloadSql` and `isDependencyResolvedWakePayload`
replace the blockers-only predicates. They match both
`issue_blockers_resolved` and `issue_children_completed` in
`_paperclipWakeContext.wakeReason`.
- The deferred-receipt worker and the stranded-queue resume use them, so
a deferred children-completed wake is started once the parent has no
live run, the same way a blockers-resolved one is.
- No code comments added.

## Verification

Red/green:
- `deferred-execution-receipt-work.test.ts` "starts a deferred
children-completed wake left behind when its parent's run blocked the
parent (restart chaos t06)". It sets up a blocked parent whose last run
succeeded, a done child, and the deferred `issue_children_completed`
receipt as t06 left it.
  - On master: the receipt stays `deferred_issue_execution` (red).
- With this PR: the receipt completes with exactly one
`issue_children_completed` run.
- `deferred-execution-receipt-work` and
`heartbeat-dependency-scheduling` pass (60 tests). `tsc --noEmit -p
server` is clean.

## Risks

- The workers now also select deferred children-completed wakes.
Admission is unchanged: they still pass the queue's own gates, and they
start only when the issue has no live run, the same as blockers-resolved
wakes.
- No schema change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069, #20108)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:14:47 -06:00
DottaandPaperclip 5d25117ffa fix(heartbeat): recover deferred dependency completion wakes (#19457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks wait for other tasks through saved dependency edges and wake
requests.
> - A dependency can finish while its assignee answers a question about
the blocked task.
> - Paperclip saves the completion wake behind that active turn.
> - Recovery did not find that saved wake unless it also contained a
comment or an interaction response.
> - This pull request restores that wake through the existing release
and dispatch checks.
> - The task can then continue after the active turn and its cleanup
finish.

## Linked Issues or Issue Description

**What happened?**

A task stayed blocked after its dependency finished. The completion wake
was saved while an audit turn was active. The wake stayed deferred after
that turn ended. It had no comment or interaction response, so both
recovery queries and the input guard excluded it.

**Expected behavior**

Recover the saved dependency wake when the prior turn and its cleanup
finish. Start one dependency continuation under the current execution
rules.

**Steps to reproduce**

1. Run a question response on a dependency-blocked task.
2. Finish the dependency while the response is active.
3. Save the completion wake behind that response.
4. Finish the response and its environment cleanup, or restart the
workers.
5. Observe that the saved wake has no run.

**Paperclip version or commit**

Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the
final regression fixture.

**Deployment mode**

Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated
embedded PostgreSQL.

Related work: #16673 added recovery for saved messages. #17288 and
#16966 address assignment wakes after cleanup. This change covers
dependency completion receipts with no message.

## What Changed

- Include saved `issue_blockers_resolved` receipts in worker
reconstruction and stranded queue selection.
- Let these receipts pass the input guard without a comment or
interaction response.
- Use the existing release transaction and dispatch checks. Add no
timer, migration, or direct task status change.
- Add process-backed regression cases for restart, cleanup release,
cancellation, agent pause, pending cleanup, and unresolved blockers.
- Update the dependency wake recovery document.

- Share the dependency-wake payload predicate and SQL fragment across
reconstruction and stranded receipt admission, so these eligibility
checks have one owner.

## Verification

- The final regression fails on the base code: the receipt stays
`deferred_issue_execution` and has no successor. The same test passes
with the fix.
- Six focused cases pass. They use real PostgreSQL and a real process
adapter. They check saved output, one dependency continuation, clean
task locks, repeat recovery, and execution gates. They do not call a
model provider.
- Three nearby suites pass: 125 tests across queue wiring, dependency
scheduling, and dependency recovery.
- Final-head server layered typechecking passes with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary
checks pass. Two parallel checker attempts reached the sandbox memory
limit; the completed low-memory check covers all source and test layers.
- The full deferred-receipt suite passes: 45 tests. Combined with the
nearby suites, 170 local tests pass. These results cover head
`8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in
progress.
- Local workspace typechecking and build stop at the Runner Rust step
because Cargo is absent. The full local test suite was not run. Required
CI must supply broad coverage.
- The hosted incident has not been recovered or deployed from this
branch.

- Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal
merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict
or history rewrite. All 54 receipt/helper cases pass on macOS arm64,
Node 26.4.0, real embedded PostgreSQL and process adapters. The six
dependency continuation and safety cases also pass separately. Module
boundaries and whitespace checks pass. No assertion, timeout, execution
gate or retry policy changed for this extraction.
- Current shared, SDK and adapter-utils source builds pass. The first
broad local server typecheck read stale borrowed adapter declarations;
rebuilding them exposed nominal type duplication between the borrowed
workspace and this checkout. It is not counted as a server typecheck
pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the
previously tested artifacts. Final revision
`182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable
CI checks (three skipped, one neutral review), including clean-workspace
typecheck, build and broad tests. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42262173960).
Greptile and Paperclip review both give 5/5 with no open findings.

## Risks

- Recovery now considers a saved dependency receipt with no message.
Current ownership, source finalization, cleanup, pause, cancellation,
and dependency checks still control execution.
- The scan remains bounded and uses existing transaction notifications
and restart reconstruction. The integration checks verify that it
becomes quiet after settlement.

## Model Used

- OpenAI Codex, based on GPT-6. The exact provider model ID and context
size are not exposed in this session. Used reasoning, tool calls,
TypeScript edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:38 -06:00
f2593b7905 feat: make Coordinator and Agent Chat standard and speed up navigation (#19360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People enter that work through the workspace, agent conversations,
and tasks.
> - Coordinator and Agent Chat still require experimental enrollment.
> - Task link prefetches do not warm the compact queries used by the
default task view. Chat also waits for a request when its navigation
list already holds the conversation.
> - This pull request makes Coordinator and Agent Chat standard and
reuses the reads needed for navigation.
> - People can choose Classic workspace in Experimental settings. Cached
work can appear while fresh data loads.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Workspace selection, Agent Chat availability, and navigation between
Chat, Tasks, and task details.

**Subsystem affected**

Cross-cutting: UI, workspace preferences, instance settings, and the
database migration.

**Current behavior**

People must enable Coordinator and Agent Chat in Experimental settings.
Opening a task can repeat requests because prefetch warms the full
detail and comment queries, while the default view reads compact task
data and a recent comment window. Opening Chat waits for a detail
request even when the current person's conversation list contains it.

**Proposed behavior**

Coordinator is the standard workspace for everyone. Experimental
settings offer Classic workspace, off by default. Old Coordinator
experiment values do not select Classic. Agent Chat is available to
everyone, including instances with an old saved or managed off value.
Task link intent warms the queries used by the mounted view. Chat can
use the current person's company-scoped conversation list during
refresh.

**Reason and benefit**

People can use the standard workspace without setup. Repeat navigation
can show cached content while the server refreshes it.

**Breaking changes**

Existing Coordinator experiment and Agent Chat off values no longer
disable these surfaces. Classic requires an explicit new opt-in. The
workspace API keeps its existing response shape and audit checks. The
migration retains legacy values and adds a separate Classic preference.
Execution permissions, company access, and chat privacy checks still
apply.

**Additional context**

Roadmap check: this graduates existing surfaces and improves their
navigation. It does not add a new agent execution capability. Duplicate
searches found no open Coordinator workspace PR. Related server cache
work, Refs: #17828, addresses compact issue-list invalidation; this PR
addresses client navigation queries.

## What Changed

- Add a default-off Classic workspace preference. Derive the existing
workspace API response from that choice. Keep personal persistence,
account isolation, and audited writes in the workspace preference
service.
- Replace the Coordinator experiment control with Classic workspace. Use
Coordinator while preferences load. Update the existing settings
examples and navigation recipe.
- Retire the Agent Chat experiment control. Ignore old stored and
managed off values. Retain the compatibility key and the existing
execution and access checks.
- Reuse account- and company-scoped chat list data with its original
freshness timestamp.
- Share compact task and recent-comment query options between prefetch
and the task view. Keep full-history prefetch for Classic tasks and
expanded history.
- Add upgrade, replay, cache, account isolation, and browser switching
regressions. Update product, database, development, and roadmap
documentation.

## Verification

- Targeted preference, settings, migration, cache, and UI regression
tests passed during implementation. The cache tests hold refresh
requests pending and check that content still renders. They also check
account isolation and stale refresh behavior.
- A real browser test verified fresh Coordinator defaults, Classic
opt-in persistence, and return to Coordinator. It used an isolated
source checkout with fresh data on macOS.
- Token, migration numbering and safety, map, and Storybook catalog
checks pass after the rebase.
- Repository typecheck and build pass. Typecheck and token gates also
pass after review repairs. The broad local run passed all six chat
integration shards (1,931 tests). It then found the host-CLI-dependent
recovery fixture described below. Its remaining duplicate work was
stopped after the full current-head CI matrix passed; this is not a
claim that the broad local command completed green.
- Review repairs passed 419 UI/migration fixture tests, 19 Coordinator
navigation tests, and all 214 tests in the five affected server suites.
The 24 Chat/workspace browser tests pass, including standard Coordinator
switching and explicit Classic persistence.
- The broader browser repair run passed 85 tests. It retained two failed
assertions and skipped five later serial cases. The affected files then
passed all 11 tests after the fixes: reopen the desktop panel after
reload and open More for Audit and Connectors. This includes the five
serial cases that had not run.
- The broad local suite exposed a recovery fixture that executed the
host Codex CLI during a simulated remote probe. On a host with Codex
0.154.0, model fallback broke its fixed-model assertion. The fake
sandbox now reports Codex 0.160.0. All 20 selected recovery/provider
compatibility cases pass after the repair.
- A CLI company import/export suite timed out during server startup in
the second CI run. The port fallback is intentional in that fixture. No
root cause was established. That suite passes on the final commit in CI.
This records recovery, not a claimed product fix for the earlier
timeout.
- The first CI run exposed remaining assumptions about the retired flags
and Classic UI. The repair commit updates those fixtures and navigation
steps. It also restores the missing enabled Cases destination in
Coordinator. Earlier local recovery attempts used the wrong fixture
notification and advanced fake time during live execution; the corrected
tests use the subscribed recovery queue and wall time during execution.
- Greptile reviewed commit `98972b030d4505d838427a2fca6b196b8cad6d47` at
**5/5** with no new comments. The original finding is resolved.
- Current-head CI is green: **56 successful checks**, with three checks
skipped by workflow filters. This includes typecheck, build, the
complete Vitest matrix, all eight browser shards, Runner verification,
release checks, and security checks. [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/42132651414).
- Typecheck caught missing `pid` and `startedAt` fields in the simulated
command result. The fixture now returns the complete process contract,
and the server test-layer typecheck passes.
- Storybook catalog alignment was checked. Storybook play tests were not
run. Hosted latency and cold bundle loading were not measured.

## Risks

- The new default changes the workspace for people who previously left
the Coordinator experiment off. They can opt into Classic in
Experimental settings.
- The additive migration must run before the new server code. It is safe
to replay and preserves explicit Classic choices.
- Cached content can be stale during background refresh. Existing query
invalidation and source timestamps still control freshness.
- These changes reduce avoidable loading waits. They do not establish a
measured improvement in hosted request latency or initial bundle
download time.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution, and browser automation. The exact serving revision and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-05 16:28:25 -08:00
DottaandClaude Opus 5.5 512fbea0bb fix(native): deliver messages saved while a native run finalizes (#19183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A user can reply on a task while the agent's run is still finishing
> - A remote native run first commits its result and its status decision
(for example `done`). Then its sandbox environment settles (harness
backup, warm retention).
> - A message that arrives after the final turn took its input is saved
as a deferred receipt behind that run
> - The saved-message recovery path holds the receipt while the source
run still owns its environment lease or its workspace copy-back owner
> - Nothing notified that path when the lease or the owner settled, so
the messages waited for an unrelated company event
> - This pull request notifies the existing deferred-receipt work when a
run's environment lease settles or its workspace finalization owner is
released
> - The benefit is that messages sent while a run finalizes are
delivered once, soon after the run settles, and a task closed in that
window is reopened for them

## Linked Issues or Issue Description

**What happened?**
A user posted three plain comments on a task while the agent's remote
native run was finalizing. The final turn had already called its finish
tool. The comments were saved in one deferred receipt. The run then
committed `done` and its status became `succeeded`. No follow-up run
started. The task stayed `done` with three unanswered requests. About 20
minutes later, an unrelated event in the same company woke the
deferred-receipt worker, and the messages were delivered.

**Expected behavior**
A message that arrives after the last turn took its input, but before
the run releases, gets one follow-up turn as soon as the run settles. If
finalization marked the task `done` in that window, the task is reopened
for the message. Messages that the finished turn already consumed are
not replayed.

**Steps to reproduce**
1. Start a native run with a remote (sandbox) environment.
2. After the agent calls its finish tool, post a user comment on the
task.
3. Let finalization commit `done` and project the run as `succeeded`.
4. Observe that the comment's deferred receipt stays
`deferred_issue_execution` after the environment lease settles.

**Root cause**
A remote native run is finished by native finalization, which projects
the terminal status. The in-process run completion
(`releaseIssueExecutionAndPromote`) does not run for it. Its saved
messages depend on the stranded-queue sweep in
`resumeDeferredExecutionReceipts`, which calls the release with
`resumeStrandedQueue`. That release correctly waits while
`hasPendingRunCleanup` is true (an unsettled environment lease or a
workspace finalization owner). Finalization notifies the
deferred-receipt worker when it projects the run, but the cleanup is
still pending at that time. The worker then drops the entry. Lease
settlement and owner release did not notify the worker again.

Refs #19122 (saves a redirect sent while a stopped run settles; this
change reuses the same deferred-receipt delivery path for normally
completing runs).

## What Changed

- `environmentService.releaseLease` notifies deferred-execution work for
the company when a run's lease leaves `active` or `pending_cleanup`
(released, retained, or failed).
- `environmentService.releaseLeasesForRun` notifies deferred-execution
work in the same way.
- `withNativeWorkspaceFinalizationOwnership` notifies deferred-execution
work when it releases its owner.
- Delivery uses the existing path: `resumeDeferredExecutionReceipts` →
stranded-queue release → deferred wake promotion (with the existing
reopen rule for user messages). No new delivery mechanism.
- New regression in `deferred-execution-receipt-work.test.ts` with the
real deferred-receipt worker, the real release adapter, and the real
lease service.
- `doc/execution-semantics.md`: one additive paragraph for messages sent
while a run finalizes.

## Verification

- Staging evidence (read-only): the task's saved queue held the three
comments as one `deferred_issue_execution` receipt after the run
succeeded. The run had no `run succeeded` lifecycle event (it was
finished by native finalization, not run completion). The receipt was
promoted about 20 minutes later with `reopenedFrom: done` by the same
recovery path.
- Red before the fix: the new test failed with `expected null to be
truthy` (the receipt stayed deferred with no run after the lease
settled).
- Green after the fix: `cd server && npx vitest run
src/__tests__/deferred-execution-receipt-work.test.ts -t "finishing
native run"`.
- Safety cases in the new test:
- The receipt stays deferred while the environment lease is active
(cleanup holds still apply).
- After settlement, exactly one run carries both messages in order, and
the done task is reopened once.
  - A second settlement signal does not deliver the messages again.
- Related suites: `deferred-execution-receipt-work`,
`environment-service`, `environment-cleanup-work`,
`native-environment-settlement-work`, `idle-sleep-safety`.
- Not done: a live staging reproduction at the fixed revision.
Deployment is out of scope.

## Risks

- The deferred-receipt worker reloads the company's deferred receipts
after each run lease settlement. Each reload is bounded and runs only
when deferred receipts exist. The notification is transactional, so a
rolled-back release does not wake the worker.
- Low risk: no schema change, no API change, and no change to admission
or reopen rules.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, browser for read-only staging diagnostics).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 22:26:42 -06:00
DottaandPaperclip 0eaabe5c0b refactor(accounting): remove company serialization from paid settlement (#18917)
Settle paid operations through operation-scoped transactions. Preserve exact ledger evidence and replay while moving aggregate budget decisions to durable deferred work.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-04 07:29:28 -06:00
DottaandPaperclip e0c6992426 fix(heartbeat): prevent saved input starvation during queue reloads (#18819)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Accepted messages and task Retry requests use durable wake receipts.
> - The recovery worker loads receipts in pages of 100 and checks
batches of 25.
> - Company activity can reload those pages before the pending batch
drains.
> - Reloads reset ready receipts' dispatch times and can repeatedly
favor receipts already checked.
> - This pull request preserves the priority of unchecked ready receipts
so saved input can reach normal admission.

## Linked Issues or Issue Description

**What happened?**

Saved input can remain deferred after the preceding run finishes.
Repeated company eligibility changes reset the dispatch times of pending
receipts during paginated reconstruction. A PostgreSQL regression checks
only 75 of 178 receipts on the original code despite twelve eligibility
changes.

**Expected behavior**

Each ready receipt must reach its admission check. Existing wait
deadlines, retry backoffs, authorization, and execution gates must still
apply.

**Steps to reproduce**

1. Seed more than one page of deferred receipts.
2. Keep those receipts pending after their checks.
3. Send company eligibility notifications between scheduler passes.
4. Check that every receipt is reconsidered, including receipts later in
the backlog.

**Paperclip version or commit**

Reproduced on `367f29478d`.

**Deployment mode**

Built from source with PostgreSQL. The defect occurs before provider
execution.

Related: #10195 reports a similar starvation symptom in a different
promotion loop. #18621 added saved-input Retry after a successful run.
This change fixes fairness in the deferred-receipt worker.

## What Changed

- Add an optional entry merge callback to scoped deadline
reconstruction.
- Use it in the deferred-receipt worker to retain unread ready receipts'
priority while refreshing their state.
- Place receipts whose wait has expired into the ready queue during
reconstruction.
- Keep real wait, claim, and dispatch deadlines authoritative.
- Add a paginated starvation regression, real process dispatch coverage,
and GitHub lock-collision recovery coverage.
- Document the recovery fairness rule.

## Verification

- The PostgreSQL starvation regression checks only 75 of 178 receipts
before the fix. It checks all 178 afterward, both with ordinary receipts
and with expired execution waits.
- Seven focused scheduler cases pass, including moved interrupt grace,
moved native error deadline, fresh claim/wait deadlines, and
unchanged-gate grace.
- GitHub endpoint lock-collision recovery, retained board authority,
HTTP Retry, and five current admission-gate cases pass.
- Real process acceptance passes on the final code: task Retry crosses
the repeatedly invalidated backlog, reconstructs its persisted queued
run after a worker restart, saves `receipt-proof` in run output, and has
one successor for the Retry receipt. The test holds executor capacity
while advancing queue time, then starts fresh workers at PostgreSQL time
before releasing a real process lease. The defect is in admission
scheduling before provider execution; this acceptance uses PostgreSQL
and a local process adapter.
- The full workspace build and typecheck (`node
scripts/typecheck-low-memory.mjs`) pass locally. The affected worker
suites pass 52 of 53 cases together; the existing native multi-company
batch case times out in that combined run and passes in isolation
without code changes. All new regression and acceptance cases pass.
- Local `pnpm test:run` stops in its first chat shard on three 15-second
timeouts (Teams long native HTTP picture delivery, direct final-response
materialization, and contended inbound notices). All three pass in
isolation without code changes. Later full-suite groups did not run
locally after that shard failure.
- [The full GitHub Actions matrix
passes](https://github.com/paperclipai/paperclip/actions/runs/41697489374)
on `ac91df2cc6db6c77529d923177b375b8c92efd45`, including all
general-test, serialized-server, browser, build, and typecheck jobs.
Server shard 8 initially failed an upstream process-adoption test and
passed its one rerun without code changes.
- [Greptile is
5/5](https://github.com/paperclipai/paperclip/pull/18819#issuecomment-6612091535)
on the same current head, with no actionable findings or unresolved
review threads. The separate Paperclip Review check is neutral; its
previous request was bound to an older commit.

Focused command:

```sh
pnpm exec vitest run server/src/__tests__/deferred-execution-receipt-work.test.ts server/src/__tests__/scoped-deadline-work.test.ts
```

## Risks

- Ready receipt order changes during repeated reconstruction. Admission
still reads current durable state.
- A ready receipt must not inherit an obsolete priority after it
acquires a future deadline. The merge applies only when both pending and
loaded entries are immediate. Deadline safety tests cover that boundary.
- The merge callback is optional. Other scoped workers retain their
current reconstruction behavior.

## Model Used

OpenAI `gpt-6.1-sol` in Codex, max reasoning effort, with tool use and
local code execution. The runtime does not expose the context-window
size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused regressions and
acceptance; full-run timing limitations documented above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 13:17:00 -06:00
DottaandPaperclip 3ee4c5b613 Make saved execution receipt recovery react to durable work (#18409)
Reviewed and verified at 8f306e0a6e.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 10:03:59 -05:00