DottaandPaperclip
8dfed893d8
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Decisions can depend on strict references to other tasks. > - A cancelled, deleted, or completed target can make a decision obsolete. > - The server currently scans open decisions on every maintenance tick to find those changes. > - The owning issue writes already have transaction boundaries that can signal the change. > - This pull request reconciles affected decisions after commit and reconstructs open targets at startup. > - Valid open decisions and completed history stop causing periodic SQL. ## Linked Issues or Issue Description **What existing behavior does this improve?** Decision target invalidation. This is slice 12 of the recurring-server sleep stack, based on #17482. TTL expiry is #17479 and continuation delivery is #17478. Retention remains separate work. **Subsystem affected** Server decision policy, issue mutation and deletion, and startup wiring. **Current behavior** The aggregate sweep repeatedly reads open decisions and their strict targets. It shares a maintenance tick with unrelated jobs even when every target remains valid. **Proposed behavior** Committed decision creation and terminal issue updates request a scoped check. Deletion requests a bounded full scan because its target links can cascade away. Startup reconstructs open target state. Once reconciliation settles, no target scan remains scheduled. **Reason and benefit** Remove normal empty polling without losing target invalidation, transactional delivery, or restart recovery. Reuse existing decision rows and target links instead of adding a table. **Breaking changes** No API or schema change. The main tick stops calling the combined decision sweep after its four duties have separate owners. The public combined method retains its default behavior. **Additional context** The initial decision implementation stored secondary links as well as primary links. This migration does not assume a new legacy backfill. Independent SQL writers still require an explicit signal integration. ## What Changed - Replace the periodic target scan with committed-change activation and bounded startup reconstruction. Keep the existing decision policy and expiry/activity owner. - Require production notifications to identify an issue or decision. Before deletion, lock the issue and capture its linked open decisions in the same company; emit their IDs before links cascade. This avoids resetting full scans and fences concurrent proposal creation. - Reuse that owner helper for issue removal and routine failure cleanup. No linked decisions means no target signal. Remove company deletion's ineffective global signal; its pre-existing restrictive decision foreign keys remain unchanged. - Retain PostgreSQL timestamp text in paging cursors so rows within one millisecond cannot repeat indefinitely. - Remove the unused combined internal decision sweep after its final duty is split; exercise each owner directly in the existing behavioral tests. - Keep query policy and notification discovery in `decision-targets.ts`. The coordinator registration contains lifecycle wiring only. Add no queue table, per-route recovery wrapper or external-wake exemption. ## Verification Current candidate normally integrates landed effect recovery `e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to tested target head `85ce6545bf`; the only new source files are four independently merged Runner E2E accounting files, preserved exactly. The 248-case/type/build evidence comes from test revision `54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by documentation-only changes. Fresh current-head CI and reviews are required. This remains the target-invalidation duty only. - **248 cases pass** in all seven complete suites: 25 target, 80 routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. Database suites use separate migrated PostgreSQL fixtures. Server production typecheck and build pass. - **Four regressions failed before repair:** paging across microsecond timestamps, deletion behind a full valid startup page, unnecessary global work from unrelated deletion, and unnecessary work after deleting an empty company. All four pass. - A real concurrent proposal is observed waiting on the issue-row lock between pre-delete discovery and deletion commit. Removing that lock made the test fail because the proposal committed instead of waiting. Restoring the implementation passes. The mutation was restored before commit. - Existing real producer tests cover commit/rollback, deletion with scoped hints in one outer transaction, secondary targets, re-opened tasks, valid/lenient history quietness, startup paging, failure retry, drain/resume and shutdown. The routine failure case now requires no signal when its cleanup has no linked decision. - The earlier dropped-hint and drain-fixture failures remain in [the behavior audit](doc/idle-decision-targets.md). Extra upstream embedded-only tests attempted on the parent could not initialize locally; CI must cover that environment. No new schema migration or table is added. The final review cleanup removes the combined internal `sweepExpired` method, cursor, flags and unused target-policy export. A repository/package/SDK audit found no remaining production caller or public exposure. Behavioral tests now call the dedicated workers; isolation tests prove TTL does not recover stale effects and effect recovery does not expire open decisions. HTTP and plugin SDK contracts are unchanged. The full 248-case selection passes after this cleanup and parent reconciliation; types/build pass with the updated Runner vendor. Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI group: company import/export startup exceeded its 30-second health deadline, and the dense ZIP case exceeded its unchanged five-second timeout. The preceding UI group passed all 4,169 cases. Both CLI files and the transfer implementation are unchanged from the landed parent. The complete ZIP-transfer file passes all 17 cases locally on this exact head; the CLI startup E2E is skipped because the host's embedded PostgreSQL bootstrap fails. This does not prove Linux startup locally. The failure logs remain recorded; no timeout or assertion is changed. One bounded retry of the failed jobs is requested after the workflow completed. Its result and the current-head gates remain required before merge. Commands: server typecheck/build and Vitest for `decision-target-work`, `routines-service`, `decisions-service`, `decision-effect-recovery-work`, `decision-expiry-work`, `server-startup-feedback-export` and `scoped-deadline-work`. Local targeted acceptance does not establish deployed suspension or fleet savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates. ## Risks - Deletion now holds the issue row lock while discovering dependent decisions and completing the existing removal transaction. This is needed to serialize new FK references. Provider calls are not placed inside that transaction. - Company deletion with decision-origin references still has the pre-existing restrictive FK limitation. Removing the futile notification does not implement a new company-data cleanup policy. - Supported writers share the application DB owner and maintain existing target links. Independent SQL writers, damaged links or multiple active processes need their own wake contract; startup still reconstructs all open targets. - Cloud sleep blockers and durable external alarms are unchanged. Worker quiet is not proof of deployed host/database suspension. ## Model Used OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact serving model identifier and context window were not exposed. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [ ] All Paperclip CI gates are green - [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
9.9 KiB
Idle decision target invalidation
Open decisions no longer require a periodic target scan. The target worker reconciles committed issue changes and decision creation. Startup reconstructs open decisions in bounded pages. After reconciliation, valid open decisions and completed history cause no recurring SQL. TTL expiry, continuation delivery, interrupted effects, and retention remain separate responsibilities.
Domain policy
Only strict effect references participate. The existing shared
decisionEffectTargetIssueIds helper includes primary and secondary references.
A missing or cancelled target expires a decision with target_gone. If all
strict targets are done and at least one was not done in the proposal snapshot,
it expires with target_completed. Lenient decisions and proposals intentionally
acting on already-done targets remain open. A target that reopened before
reconciliation is checked in its current state; the notification is not authority
to expire it. The existing open-status compare-and-set arbitrates expiry against
other decision transitions.
decision-targets.ts owns this policy and scoped reconciliation. It delegates
expiry, activity, and continuation to the existing expiry owner. Startup
registers the target worker when scheduling is enabled. The combined internal
sweepExpired method is removed after the final duty is split. No new table or
migration is added.
Producer audit
| Producer | Intent before write | Scope |
|---|---|---|
| Decision creation, including bundles and caller-owned transactions | Existing creation transaction, before insertion | Decision ID |
issueService.update entering/reasserting done or cancelled |
Existing issue update transaction, before the patch | Issue ID |
| Issue removal | Lock the issue and capture linked open decision IDs in the existing transaction, before deletion | Affected decision IDs; no signal when none |
| Routine dispatch failure deleting its created issue | Same locked pre-delete lookup in the existing dispatch transaction | Affected decision IDs; usually no signal |
The existing target link table supports ordinary issue scopes. Before a target is deleted, the decision owner locks its issue row, reads distinct linked open decisions in the same company, and signals their IDs. The links may then cascade without losing notification scope. The row lock closes the race in which a new proposal could add its target reference after discovery but before deletion. This remains part of the caller's existing transaction; no provider operation is wrapped in a new transaction. Production hints require an issue or decision ID, so routine cleanup and repeated removals do not restart full startup paging.
Company deletion no longer signals target work. With no decision rows it cannot create a same-company invalidation obligation. Pre-existing restrictive origin run/issue foreign keys still prevent deleting some companies with decisions; that separate limitation is unchanged. This migration does not claim to implement complete decision-data deletion for those companies.
Other direct issue writers were inspected: checkout/adoption enter in-progress; conversation writers enter todo/in-progress/in-review; recovery enters blocked; release preserves existing terminal status; administrative release changes locks and assignment. Tree restoration excludes terminal snapshots. Route aliases and raw privacy SQL do not write terminal status. These paths do not create new invalidation obligations. They can restore a target before the worker runs; the worker reads its current status.
The initial decision feature (0197 / commit 9c1f8e7887) already stored primary
and secondary target links in the creation transaction. The later shared-helper
extraction ef33c1d9ed preserved that behavior. There is no newly assumed legacy
secondary-link backfill. Direct SQL writers or damaged/incomplete link data are
outside the supported live writer contract; startup still examines all open rows.
Scheduling and lifecycle
The owning service holds pending scopes and cursors, not a second durable queue. The database decisions, options, snapshots, and target rows remain truth. Each pass examines at most 100 rows/scopes, ordered by expiry and ID using the existing open-decision index. The cursor retains PostgreSQL timestamp text, including microseconds, rather than rounding through a JavaScript Date. An empty scope consumes one unit of the pass budget. It rotates unfinished scopes so a long startup scan does not starve new changes.
A settled notification replaces its scope with a fresh cursor. An admitted pass
cannot delete that replacement when it completes. Failed rows receive individual
retry scopes while later rows continue; query failures retain their scope.
Retries use the coordinator's existing 30-second cadence only while reconciliation
is unfinished. Unknown hints request a bounded full scan. Once all scopes settle,
hasPending returns from memory and no future target scan is scheduled.
Intent registration is awaited before writes. Nested writes belong to the outer transaction. Rollback does not publish a change; ambiguous outcomes retain the existing XID reconciliation contract. Notifications cover the application's DB owner and child pools, not arbitrary SQL writers or independent active processes.
Drain preserves pending scopes without passive target SQL. Resume processes them. Shutdown checks cancellation between rows and awaits admitted expiry/delivery. Startup reconstructs anything left after process exit. Cloud's conservative sleep guards are unchanged; this establishes worker quiet, not permission to sleep with unsupported obligations or proof of host suspension.
Current verification
All 248 cases in seven complete suites pass: 25 target, 80 routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. PostgreSQL suites use separate migrated external fixtures. Server typecheck and build pass. No new migration or table is added. The normal parent merge preserves landed expiry and all 21 independent master files. Server types and build use the updated Runner vendor (1,524 inputs and 1,563 outputs).
Four real regressions failed before repair: sub-millisecond paging, deletion behind a full valid startup page, unrelated deletion notifications and successful empty-company deletion notifications. They all pass. The new concurrency test observes the proposal's actual PostgreSQL lock wait while deletion is held. Temporarily removing the issue-row lock made the test fail because the proposal was accepted instead of waiting; restoring it passes. The mutation was restored before commit. No timeout or production hold was weakened.
The combined internal method had no production caller after this split. The server package exposes only its startup module; neither its exports nor the plugin SDK exposes the decision service. Its tests now call the dedicated TTL, target, effect and continuation workers, preserving their outcome assertions. The obsolete cursor, scan options and exported target-policy method are removed. HTTP and plugin SDK contracts are unchanged.
Fresh current-head CI and both automated reviews remain required. Historical fixture and CI failures below remain part of the verification record. Extra upstream embedded-only tests attempted during parent integration could not initialize locally; CI must cover that environment.
Earlier verification
The initial 17-case real-PostgreSQL target suite passed. The combined selection passed 233 tests: those 17, 17 effect-recovery cases, 14 TTL cases, 50 decision service cases, 39 startup cases, 80 routine-service cases, and 16 routine-route cases. The final target run passed all 18 cases, including a change to the same scope during reconciliation. In total, 234 selected cases pass across these runs.
Coverage includes actual issue update/removal, decision creation after cancellation, secondary targets, all-target completion, intentional done targets, lenient history, outer commit/rollback, startup after deleted links, first change and failed expiry through drain, reopened targets, scope isolation, more than one page of valid open rows, failure fairness, creation during delivery, and shutdown. The original routine failure test observed intent and settlement around its cleanup transaction. The current test requires no target signal when that created issue has no linked decisions. Quiet assertions observe SQL through one hour of fake time after settlement.
Workspace typecheck/build passed before the final test-only addition. No runtime test failure was suppressed or timeout increased. Existing warnings from startup fixture signal listeners remain visible. The routine suite now supports the validated disposable external test database path, so local embedded PostgreSQL unavailability does not silently skip its 80 cases.
A local query probe with 40,000 completed decisions, one open decision, and 40,001 target links used existing indexes: scoped lookup read five buffers (0.012 ms), full open-page lookup two (0.003 ms). The probe was rolled back. No new schema index is needed for this change. Full PR CI and fresh review remain required.
Earlier review and CI repair
Review found that an absent deletion hint was dropped when the outer transaction
also carried a scoped hint. Both delete-before-update and update-before-delete
regressions reproduced the missing expiry after the target links cascaded away.
That revision sent the explicit all hint alongside scoped hints; the current
pre-delete decision-ID contract below supersedes the global hint.
The repaired target suite passes 20 cases; the TTL suite passes 14 cases.
The initial CI run failed the TTL worker's drain assertion. It measured SQL before already-admitted sweeps had finished. The test now joins those specific sweep promises before measuring passive SQL; unresolved-intent holds remain intact. Waiting for the entire idle count to reach zero was tested and rejected: that count correctly includes the retained recovery holds. No production drain or retry behavior changed for this test repair.