Commit Graph
6 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 c5c587f162 fix(issues): start a deferred children-completed wake the release drain missed (#20180)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent that delegates waits for its children, and is woken when
they finish (`issue_children_completed`)
> - Restart chaos t06 ended with both children done and the parent
blocked forever
> - The children finished while the parent's run was still going, so the
wake was deferred as a follow-up. That run then ended with the agent
setting the parent blocked, and the deferred wake was never started
> - The durable re-drivers for deferred wakes cover only
`issue_blockers_resolved`, so nothing else picks up a missed
`issue_children_completed`
> - This pull request makes both re-drivers cover children-completed
wakes too
> - The benefit is that a parent always hears that its children
finished, even when the release drain misses the deferred wake

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261109T041248-s9501`, task t06 (claude, M1, K2), Daytona per-turn,
master fa7950121d.

**Problem:** parent CHA-10 (all times UTC):
- 04:29:57: run 9cf49f52 started.
- 04:32:35 and 04:32:43: its children CHA-13 and CHA-14 finished.
- 04:32:43: `issue_children_completed` merged into the running turn.
- 04:32:53: that wake was deferred as a follow-up
(`issue_execution_deferred`, `deferred_issue_execution`, no run).
- 04:33:13: the turn ended with the agent setting the parent `blocked`,
a decision made before it saw the children's results.

The deferred wake stayed deferred for the rest of the wave, including
three later restarts. The parent sat blocked with no Retry. The
deferred-receipt worker (`deferred-receipt-work.ts`) and the
stranded-queue resume (`queue.ts`) select deferred dependency wakes with
`issueBlockersResolvedWakePayloadSql`, which matches only
`issue_blockers_resolved`.

## What Changed

- `server/src/services/issue-dependency-wakeups.ts`:
`dependencyResolvedWakePayloadSql` and `isDependencyResolvedWakePayload`
replace the blockers-only predicates. They match both
`issue_blockers_resolved` and `issue_children_completed` in
`_paperclipWakeContext.wakeReason`.
- The deferred-receipt worker and the stranded-queue resume use them, so
a deferred children-completed wake is started once the parent has no
live run, the same way a blockers-resolved one is.
- No code comments added.

## Verification

Red/green:
- `deferred-execution-receipt-work.test.ts` "starts a deferred
children-completed wake left behind when its parent's run blocked the
parent (restart chaos t06)". It sets up a blocked parent whose last run
succeeded, a done child, and the deferred `issue_children_completed`
receipt as t06 left it.
  - On master: the receipt stays `deferred_issue_execution` (red).
- With this PR: the receipt completes with exactly one
`issue_children_completed` run.
- `deferred-execution-receipt-work` and
`heartbeat-dependency-scheduling` pass (60 tests). `tsc --noEmit -p
server` is clean.

## Risks

- The workers now also select deferred children-completed wakes.
Admission is unchanged: they still pass the queue's own gates, and they
start only when the issue has no live run, the same as blockers-resolved
wakes.
- No schema change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069, #20108)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:14:47 -06:00
DottaandPaperclip 5d25117ffa fix(heartbeat): recover deferred dependency completion wakes (#19457)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tasks wait for other tasks through saved dependency edges and wake
requests.
> - A dependency can finish while its assignee answers a question about
the blocked task.
> - Paperclip saves the completion wake behind that active turn.
> - Recovery did not find that saved wake unless it also contained a
comment or an interaction response.
> - This pull request restores that wake through the existing release
and dispatch checks.
> - The task can then continue after the active turn and its cleanup
finish.

## Linked Issues or Issue Description

**What happened?**

A task stayed blocked after its dependency finished. The completion wake
was saved while an audit turn was active. The wake stayed deferred after
that turn ended. It had no comment or interaction response, so both
recovery queries and the input guard excluded it.

**Expected behavior**

Recover the saved dependency wake when the prior turn and its cleanup
finish. Start one dependency continuation under the current execution
rules.

**Steps to reproduce**

1. Run a question response on a dependency-blocked task.
2. Finish the dependency while the response is active.
3. Save the completion wake behind that response.
4. Finish the response and its environment cleanup, or restart the
workers.
5. Observe that the saved wake has no run.

**Paperclip version or commit**

Reproduced against `70d23612dd`. Rechecked against `2f35eb1080` with the
final regression fixture.

**Deployment mode**

Source checkout on Linux x64, Node 25.9.0, pnpm 9.15.4, and isolated
embedded PostgreSQL.

Related work: #16673 added recovery for saved messages. #17288 and
#16966 address assignment wakes after cleanup. This change covers
dependency completion receipts with no message.

## What Changed

- Include saved `issue_blockers_resolved` receipts in worker
reconstruction and stranded queue selection.
- Let these receipts pass the input guard without a comment or
interaction response.
- Use the existing release transaction and dispatch checks. Add no
timer, migration, or direct task status change.
- Add process-backed regression cases for restart, cleanup release,
cancellation, agent pause, pending cleanup, and unresolved blockers.
- Update the dependency wake recovery document.

- Share the dependency-wake payload predicate and SQL fragment across
reconstruction and stranded receipt admission, so these eligibility
checks have one owner.

## Verification

- The final regression fails on the base code: the receipt stays
`deferred_issue_execution` and has no successor. The same test passes
with the fix.
- Six focused cases pass. They use real PostgreSQL and a real process
adapter. They check saved output, one dependency continuation, clean
task locks, repeat recovery, and execution gates. They do not call a
model provider.
- Three nearby suites pass: 125 tests across queue wiring, dependency
scheduling, and dependency recovery.
- Final-head server layered typechecking passes with
`PAPERCLIP_TYPECHECK_LOW_MEMORY=1`. Whitespace and module boundary
checks pass. Two parallel checker attempts reached the sandbox memory
limit; the completed low-memory check covers all source and test layers.
- The full deferred-receipt suite passes: 45 tests. Combined with the
nearby suites, 170 local tests pass. These results cover head
`8f9d3aa6ac19a6d2532bd869eb8adda7cf122a5a`. Required CI remains in
progress.
- Local workspace typechecking and build stop at the Runner Rust step
because Cargo is absent. The full local test suite was not run. Required
CI must supply broad coverage.
- The hosted incident has not been recovered or deployed from this
branch.

- Review follow-up at `182385e9bc433adac06440997de7e4cfbf889a72`: normal
merge of master `0d03610d2f73b7cd799547e43929bb2303319024`; no conflict
or history rewrite. All 54 receipt/helper cases pass on macOS arm64,
Node 26.4.0, real embedded PostgreSQL and process adapters. The six
dependency continuation and safety cases also pass separately. Module
boundaries and whitespace checks pass. No assertion, timeout, execution
gate or retry policy changed for this extraction.
- Current shared, SDK and adapter-utils source builds pass. The first
broad local server typecheck read stale borrowed adapter declarations;
rebuilding them exposed nominal type duplication between the borrowed
workspace and this checkout. It is not counted as a server typecheck
pass. All 30 SDK and 169 adapter runtime JavaScript outputs match the
previously tested artifacts. Final revision
`182385e9bc433adac06440997de7e4cfbf889a72` now passes all 55 applicable
CI checks (three skipped, one neutral review), including clean-workspace
typecheck, build and broad tests. [CI
result](https://github.com/paperclipai/paperclip/actions/runs/42262173960).
Greptile and Paperclip review both give 5/5 with no open findings.

## Risks

- Recovery now considers a saved dependency receipt with no message.
Current ownership, source finalization, cleanup, pause, cancellation,
and dependency checks still control execution.
- The scan remains bounded and uses existing transaction notifications
and restart reconstruction. The integration checks verify that it
becomes quiet after settlement.

## Model Used

- OpenAI Codex, based on GPT-6. The exact provider model ID and context
size are not exposed in this session. Used reasoning, tool calls,
TypeScript edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 11:49:38 -06:00
8d9d2f9ddf fix(recovery): preserve deliberate stops and dependency waits
Preserve deliberate assignee blocks and operator stops during dependency and run-release recovery. Attribute native waits to their bound owner. Ignore unrelated edits and shared-workspace exports from another issue. Continue after genuine dependency changes while preserving independently authorized user input.

Keep the original contributor's implementation and review history. Add native ownership, unrelated-edit, later readiness, and shared-workspace regressions. The shared-workspace loop was reproduced before the fix.

Co-Authored-By: Devin Foley <devin@paperclip.ing>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-15 00:18:23 -07:00
zach-hermesandPaperclip 8db826d18a fix(issues): cycle-aware issue_blockers_resolved after terminal reset (#11979)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents resume blocked work through the `issue_blockers_resolved`
wake when every durable blocker is `done`
> - That wake is level-triggered: one ready state produces one wake,
shared by the issue update route, workspace-finalize backstop, and
periodic liveness backstop
> - The ready-state key hashed only the dependent id and blocker set, so
it ignored a later reset from a terminal status back into `blocked`
> - After that reset, completing the same blockers found the previous
cycle's completed wake and suppressed the new continuation
> - This pull request folds the dependent's `blockedTransitionAt` into
the ready-state key, with compatibility for old no-cycle keys
> - The benefit is that a reset blocked issue receives exactly one new
wake without watchdog status repair or a change to blocker edges

## Linked Issues or Issue Description

Refs: https://github.com/paperclipai/paperclip/issues/5985
Refs: https://github.com/paperclipai/paperclip/issues/6555
Related: https://github.com/paperclipai/paperclip/pull/8009
Related: https://github.com/paperclipai/paperclip/pull/11570

This change does not auto-flip `blocked` to `todo`. The wake is the
continuation. It also does not treat cancelled blockers as resolved.

**What happened?**
A blocked assigned issue that was previously `done` or `cancelled`, then
reset to `blocked` on the same blocker set, did not receive
`issue_blockers_resolved` when those blockers later returned to `done`.
A completed wake from the previous cycle reused the same level-triggered
state key and suppressed the new wake. Route-time emit,
workspace-finalize backstop, and periodic liveness backstop all used
that helper.

**Expected behavior**
When every durable blocker is `done`, a currently `blocked` assigned
issue must receive exactly one valid `issue_blockers_resolved`
continuation for the current blocked cycle. A completed wake from an
earlier cycle must not suppress it. Watchdog `blocked` → `todo` repair
must not be required.

**Steps to reproduce**
1. Assign issue B, block it on issue A, mark A `done`, and let B receive
`issue_blockers_resolved`.
2. Mark B `done`.
3. Reset A to `todo` and reset B from `done` to `blocked` on the same A
id. This refreshes `blockedTransitionAt`.
4. Mark A `done` again.
5. Observe that B stays `blocked` with no new `issue_blockers_resolved`
wake.

**Paperclip version or commit**
`master` at `cc42a67e7e9e8eb183097afc8ff4ebfa694fb3e0`

**Deployment mode**
Self-hosted server

## What Changed

- Extend `buildIssueBlockersResolvedWakeStateKey` so the digest includes
the dependent's `blockedTransitionAt` as UTC ISO-8601, or `none`
- Thread `blockedTransitionAt` through `listWakeableBlockedDependents`,
both route emit sites, and both backstop candidate selects
- Keep compatibility: new cycle-aware keys suppress in idempotent
statuses; old no-cycle state keys suppress when in-flight, or when
completed and `requestedAt >= blockedTransitionAt` (or the cycle is
null); legacy per-edge keys stay in-flight-only
- Do not rewrite `blockedByIssueIds`, auto-flip `blocked` → `todo`, or
delete historical wake rows
- Add helper, route, restore, chained dependent, and backstop tests for
the reset cycle

## Verification

```
pnpm --filter @paperclipai/server exec vitest run \
  src/__tests__/issue-dependency-wakeups-routes.test.ts \
  src/__tests__/heartbeat-issue-liveness-escalation.test.ts \
  src/services/issue-dependency-wakeups.ts \
  src/services/issue-dependency-wakeups.test.ts
```

Local result: all named tests passed (helper 9, routes 8, liveness 26).

## Risks

- Deploy overlap: in-flight and same-cycle completed wakes still exist
under the old no-cycle key. The lookup keeps those as suppressors so
this change does not enqueue a duplicate in the current cycle.
- A completed old-key wake from before the current `blockedTransitionAt`
no longer suppresses. That is the intended fix.
- No schema migration. Rollback is revert of this PR.
- This does not change cancelled-blocker semantics or watchdog `blocked`
→ `todo` repair.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Provider: xAI
- Model: Grok 4.6
- Tool use and code execution: yes
- Human-authored: no

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` /
`Closes: #` / `Refs: #` OR (b) described the issue in-PR following the
relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-23 08:50:38 -07:00
Nicky LeachandPaperclip e0e503e1bc fix(server): make blockers-resolved wake dedup level-triggered (#11732)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip uses issue dependencies to pause work until blockers reach
a ready state.
> - A blocked issue with several blockers can miss its wake when the
final blocker completes.
> - The wake deduplication used a historical per-edge key, so an old
completed wake hid the current ready state.
> - This pull request adds a level-triggered key for the sorted set of
blocker issue ids and uses one helper for all wake paths.
> - The benefit is that the final blocker wake can repair a missed wake,
while repeated reconciliation stays bounded.

## Linked Issues or Issue Description

Refs #8009, #7853, and #6719. These public pull requests cover related
dependency-wake and deduplication behavior. This pull request fixes a
separate multi-blocker state-key gap.

**What happened?**
A blocked issue with multiple blockers received no
`issue_blockers_resolved` wake when the final blocker completed. An
earlier completed per-edge wake suppressed the wake for the current
all-ready state.

**Expected behavior**
The final blocker completion must emit one wake for the current ready
state. A later reconciliation pass must not emit a second wake for the
same state.

**Steps to reproduce**
1. Create a blocked issue with at least two blocker issues.
2. Complete one blocker and record its completed per-edge wake.
3. Complete the final blocker.
4. Run the route-time or reconciliation wake path.
5. Confirm that one level-triggered wake exists for the sorted blocker
set.

**Paperclip version or commit**
`eed1e5cad91a37547e1b521232da04b9ddb316f0`

**Deployment mode**
Local dev from source.

## What Changed

- Add a SHA-256 level-triggered idempotency key from the sorted blocker
issue ids.
- Share one deduplication helper across route-time, finalize-time, and
periodic wake paths.
- Treat state-key rows with idempotent statuses as duplicates.
- Treat legacy per-edge rows as duplicates only while they remain in
flight.
- Record skipped route-time wakes without suppressing later
finalize-time or periodic wakes.
- Add regression coverage for a completed earlier-blocker wake and a
second reconciliation pass.

## Verification

- Run `npx vitest run
server/src/__tests__/issue-dependency-wakeups-routes.test.ts`.
- Run `npx vitest run
server/src/__tests__/heartbeat-issue-liveness-escalation.test.ts`.
- Run `npx vitest run
server/src/__tests__/heartbeat-dependency-scheduling.test.ts`.
- Run `npx vitest run server/src/__tests__/issue-rewake-throttle.test.ts
server/src/__tests__/recovery-stale-issue-lock-sweep.test.ts`.
- Run `tsc --noEmit` on the touched files.

## Risks

The change alters wake deduplication for dependency reconciliation. The
new key uses the full sorted blocker set, so a change in that set
permits a new wake. The regression tests cover the missed-final-blocker
case and repeated reconciliation.

## Model Used

OpenAI Codex, GPT-5, tool-use model with code execution and repository
review support.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-08-19 12:52:50 -07:00
Nicky Leach 7bfaaadcb8 Add dependency wake reconciliation backstop (#8943) 2026-07-03 18:30:51 -07:00