fix(server): wake decision target checks from committed changes (#17486)

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Decisions can depend on strict references to other tasks.
> - A cancelled, deleted, or completed target can make a decision
obsolete.
> - The server currently scans open decisions on every maintenance tick
to find those changes.
> - The owning issue writes already have transaction boundaries that can
signal the change.
> - This pull request reconciles affected decisions after commit and
reconstructs open targets at startup.
> - Valid open decisions and completed history stop causing periodic
SQL.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Decision target invalidation. This is slice 12 of the recurring-server
sleep stack, based on #17482. TTL expiry is #17479 and continuation
delivery is #17478. Retention remains separate work.

**Subsystem affected**
Server decision policy, issue mutation and deletion, and startup wiring.

**Current behavior**
The aggregate sweep repeatedly reads open decisions and their strict
targets. It shares a maintenance tick with unrelated jobs even when
every target remains valid.

**Proposed behavior**
Committed decision creation and terminal issue updates request a scoped
check. Deletion requests a bounded full scan because its target links
can cascade away. Startup reconstructs open target state. Once
reconciliation settles, no target scan remains scheduled.

**Reason and benefit**
Remove normal empty polling without losing target invalidation,
transactional delivery, or restart recovery. Reuse existing decision
rows and target links instead of adding a table.

**Breaking changes**
No API or schema change. The main tick stops calling the combined
decision sweep after its four duties have separate owners. The public
combined method retains its default behavior.

**Additional context**
The initial decision implementation stored secondary links as well as
primary links. This migration does not assume a new legacy backfill.
Independent SQL writers still require an explicit signal integration.

## What Changed

- Replace the periodic target scan with committed-change activation and
bounded startup reconstruction. Keep the existing decision policy and
expiry/activity owner.
- Require production notifications to identify an issue or decision.
Before deletion, lock the issue and capture its linked open decisions in
the same company; emit their IDs before links cascade. This avoids
resetting full scans and fences concurrent proposal creation.
- Reuse that owner helper for issue removal and routine failure cleanup.
No linked decisions means no target signal. Remove company deletion's
ineffective global signal; its pre-existing restrictive decision foreign
keys remain unchanged.
- Retain PostgreSQL timestamp text in paging cursors so rows within one
millisecond cannot repeat indefinitely.
- Remove the unused combined internal decision sweep after its final
duty is split; exercise each owner directly in the existing behavioral
tests.
- Keep query policy and notification discovery in `decision-targets.ts`.
The coordinator registration contains lifecycle wiring only. Add no
queue table, per-route recovery wrapper or external-wake exemption.

## Verification

Current candidate normally integrates landed effect recovery
`e9e84bb8a1`. Its runtime and acceptance tests are byte-identical to
tested target head `85ce6545bf`; the only new source files are four
independently merged Runner E2E accounting files, preserved exactly. The
248-case/type/build evidence comes from test revision
`54936a919f36c2ea421e7ce8e15912b1105b8dcc`, followed by
documentation-only changes. Fresh current-head CI and reviews are
required. This remains the target-invalidation duty only.

- **248 cases pass** in all seven complete suites: 25 target, 80
routine, 54 decision, 26 effect-recovery, 20 expiry, 42 startup and one
scoped-deadline case. Database suites use separate migrated PostgreSQL
fixtures. Server production typecheck and build pass.
- **Four regressions failed before repair:** paging across microsecond
timestamps, deletion behind a full valid startup page, unnecessary
global work from unrelated deletion, and unnecessary work after deleting
an empty company. All four pass.
- A real concurrent proposal is observed waiting on the issue-row lock
between pre-delete discovery and deletion commit. Removing that lock
made the test fail because the proposal committed instead of waiting.
Restoring the implementation passes. The mutation was restored before
commit.
- Existing real producer tests cover commit/rollback, deletion with
scoped hints in one outer transaction, secondary targets, re-opened
tasks, valid/lenient history quietness, startup paging, failure retry,
drain/resume and shutdown. The routine failure case now requires no
signal when its cleanup has no linked decision.
- The earlier dropped-hint and drain-fixture failures remain in [the
behavior audit](doc/idle-decision-targets.md). Extra upstream
embedded-only tests attempted on the parent could not initialize
locally; CI must cover that environment. No new schema migration or
table is added.

The final review cleanup removes the combined internal `sweepExpired`
method, cursor, flags and unused target-policy export. A
repository/package/SDK audit found no remaining production caller or
public exposure. Behavioral tests now call the dedicated workers;
isolation tests prove TTL does not recover stale effects and effect
recovery does not expire open decisions. HTTP and plugin SDK contracts
are unchanged. The full 248-case selection passes after this cleanup and
parent reconciliation; types/build pass with the updated Runner vendor.


Current-head CI run 40602159553 (attempt 1) failed in the workspace CLI
group: company import/export startup exceeded its 30-second health
deadline, and the dense ZIP case exceeded its unchanged five-second
timeout. The preceding UI group passed all 4,169 cases. Both CLI files
and the transfer implementation are unchanged from the landed parent.
The complete ZIP-transfer file passes all 17 cases locally on this exact
head; the CLI startup E2E is skipped because the host's embedded
PostgreSQL bootstrap fails. This does not prove Linux startup locally.
The failure logs remain recorded; no timeout or assertion is changed.
One bounded retry of the failed jobs is requested after the workflow
completed. Its result and the current-head gates remain required before
merge.

Commands: server typecheck/build and Vitest for `decision-target-work`,
`routines-service`, `decisions-service`,
`decision-effect-recovery-work`, `decision-expiry-work`,
`server-startup-feedback-export` and `scoped-deadline-work`. Local
targeted acceptance does not establish deployed suspension or fleet
savings. Fresh exact-head CI and Greptile/Ponytail remain merge gates.

## Risks

- Deletion now holds the issue row lock while discovering dependent
decisions and completing the existing removal transaction. This is
needed to serialize new FK references. Provider calls are not placed
inside that transaction.
- Company deletion with decision-origin references still has the
pre-existing restrictive FK limitation. Removing the futile notification
does not implement a new company-data cleanup policy.
- Supported writers share the application DB owner and maintain existing
target links. Independent SQL writers, damaged links or multiple active
processes need their own wake contract; startup still reconstructs all
open targets.
- Cloud sleep blockers and durable external alarms are unchanged. Worker
quiet is not proof of deployed host/database suspension.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
This commit is contained in:
DottaandPaperclip authored and GitHub committed 2026-10-27 18:53:50 -05:00
1 parent e9e84bb8a1
commit 8dfed893d8
15 files changed
+852 -76

No files matched your search

+164
View File
@@ -0,0 +1,164 @@
# Idle decision target invalidation
Open decisions no longer require a periodic target scan. The target worker
reconciles committed issue changes and decision creation. Startup reconstructs
open decisions in bounded pages. After reconciliation, valid open decisions and
completed history cause no recurring SQL. TTL expiry, continuation delivery,
interrupted effects, and retention remain separate responsibilities.
## Domain policy
Only strict effect references participate. The existing shared
`decisionEffectTargetIssueIds` helper includes primary and secondary references.
A missing or cancelled target expires a decision with `target_gone`. If all
strict targets are done and at least one was not done in the proposal snapshot,
it expires with `target_completed`. Lenient decisions and proposals intentionally
acting on already-done targets remain open. A target that reopened before
reconciliation is checked in its current state; the notification is not authority
to expire it. The existing open-status compare-and-set arbitrates expiry against
other decision transitions.
`decision-targets.ts` owns this policy and scoped reconciliation. It delegates
expiry, activity, and continuation to the existing expiry owner. Startup
registers the target worker when scheduling is enabled. The combined internal
`sweepExpired` method is removed after the final duty is split. No new table or
migration is added.
## Producer audit
| Producer | Intent before write | Scope |
| --- | --- | --- |
| Decision creation, including bundles and caller-owned transactions | Existing creation transaction, before insertion | Decision ID |
| `issueService.update` entering/reasserting done or cancelled | Existing issue update transaction, before the patch | Issue ID |
| Issue removal | Lock the issue and capture linked open decision IDs in the existing transaction, before deletion | Affected decision IDs; no signal when none |
| Routine dispatch failure deleting its created issue | Same locked pre-delete lookup in the existing dispatch transaction | Affected decision IDs; usually no signal |
The existing target link table supports ordinary issue scopes. Before a target
is deleted, the decision owner locks its issue row, reads distinct linked open
decisions in the same company, and signals their IDs. The links may then cascade
without losing notification scope. The row lock closes the race in which a new
proposal could add its target reference after discovery but before deletion.
This remains part of the caller's existing transaction; no provider operation
is wrapped in a new transaction. Production hints require an issue or decision
ID, so routine cleanup and repeated removals do not restart full startup paging.
Company deletion no longer signals target work. With no decision rows it cannot
create a same-company invalidation obligation. Pre-existing restrictive origin
run/issue foreign keys still prevent deleting some companies with decisions;
that separate limitation is unchanged. This migration does not claim to implement
complete decision-data deletion for those companies.
Other direct issue writers were inspected: checkout/adoption enter in-progress;
conversation writers enter todo/in-progress/in-review; recovery enters blocked;
release preserves existing terminal status; administrative release changes locks
and assignment. Tree restoration excludes terminal snapshots. Route aliases and
raw privacy SQL do not write terminal status. These paths do not create new
invalidation obligations. They can restore a target before the worker runs; the
worker reads its current status.
The initial decision feature (0197 / commit `9c1f8e7887`) already stored primary
and secondary target links in the creation transaction. The later shared-helper
extraction `ef33c1d9ed` preserved that behavior. There is no newly assumed legacy
secondary-link backfill. Direct SQL writers or damaged/incomplete link data are
outside the supported live writer contract; startup still examines all open rows.
## Scheduling and lifecycle
The owning service holds pending scopes and cursors, not a second durable queue.
The database decisions, options, snapshots, and target rows remain truth. Each
pass examines at most 100 rows/scopes, ordered by expiry and ID using the existing
open-decision index. The cursor retains PostgreSQL timestamp text, including
microseconds, rather than rounding through a JavaScript Date. An empty scope consumes one unit of the pass budget. It
rotates unfinished scopes so a long startup scan does not starve new changes.
A settled notification replaces its scope with a fresh cursor. An admitted pass
cannot delete that replacement when it completes. Failed rows receive individual
retry scopes while later rows continue; query failures retain their scope.
Retries use the coordinator's existing 30-second cadence only while reconciliation
is unfinished. Unknown hints request a bounded full scan. Once all scopes settle,
`hasPending` returns from memory and no future target scan is scheduled.
Intent registration is awaited before writes. Nested writes belong to the outer
transaction. Rollback does not publish a change; ambiguous outcomes retain the
existing XID reconciliation contract. Notifications cover the application's DB
owner and child pools, not arbitrary SQL writers or independent active processes.
Drain preserves pending scopes without passive target SQL. Resume processes them.
Shutdown checks cancellation between rows and awaits admitted expiry/delivery.
Startup reconstructs anything left after process exit. Cloud's conservative
sleep guards are unchanged; this establishes worker quiet, not permission to
sleep with unsupported obligations or proof of host suspension.
## Current verification
All 248 cases in seven complete suites pass: 25 target, 80 routine, 54 decision,
26 effect-recovery, 20 expiry, 42 startup and one scoped-deadline case. PostgreSQL
suites use separate migrated external fixtures. Server typecheck and build pass.
No new migration or table is added. The normal parent merge preserves landed
expiry and all 21 independent master files. Server types and build use the
updated Runner vendor (1,524 inputs and 1,563 outputs).
Four real regressions failed before repair: sub-millisecond paging, deletion
behind a full valid startup page, unrelated deletion notifications and successful
empty-company deletion notifications. They all pass. The new concurrency test
observes the proposal's actual PostgreSQL lock wait while deletion is held.
Temporarily removing the issue-row lock made the test fail because the proposal
was accepted instead of waiting; restoring it passes. The mutation was restored
before commit. No timeout or production hold was weakened.
The combined internal method had no production caller after this split. The
server package exposes only its startup module; neither its exports nor the
plugin SDK exposes the decision service. Its tests now call the dedicated TTL,
target, effect and continuation workers, preserving their outcome assertions.
The obsolete cursor, scan options and exported target-policy method are removed.
HTTP and plugin SDK contracts are unchanged.
Fresh current-head CI and both automated reviews remain required. Historical
fixture and CI failures below remain part of the verification record. Extra
upstream embedded-only tests attempted during parent integration could not
initialize locally; CI must cover that environment.
## Earlier verification
The initial 17-case real-PostgreSQL target suite passed. The combined selection
passed 233 tests: those 17, 17 effect-recovery cases, 14 TTL cases, 50 decision
service cases, 39 startup cases, 80 routine-service cases, and 16 routine-route
cases. The final target run passed all 18 cases, including a change to the same scope
during reconciliation. In total, 234 selected cases pass across these runs.
Coverage includes actual issue update/removal, decision creation after cancellation,
secondary targets, all-target completion, intentional done targets, lenient history,
outer commit/rollback, startup after deleted links, first change and failed expiry
through drain, reopened targets, scope isolation, more than one page of valid
open rows, failure fairness, creation during delivery, and shutdown. The original routine
failure test observed intent and settlement around its cleanup transaction. The
current test requires no target signal when that created issue has no linked
decisions.
Quiet assertions observe SQL through one hour of fake time after settlement.
Workspace typecheck/build passed before the final test-only addition. No runtime
test failure was suppressed or timeout increased. Existing warnings from startup
fixture signal listeners remain visible. The routine suite now supports the
validated disposable external test database path, so local embedded PostgreSQL
unavailability does not silently skip its 80 cases.
A local query probe with 40,000 completed decisions, one open decision, and 40,001
target links used existing indexes: scoped lookup read five buffers (0.012 ms),
full open-page lookup two (0.003 ms). The probe was rolled back. No new schema
index is needed for this change. Full PR CI and fresh review remain required.
## Earlier review and CI repair
Review found that an absent deletion hint was dropped when the outer transaction
also carried a scoped hint. Both delete-before-update and update-before-delete
regressions reproduced the missing expiry after the target links cascaded away.
That revision sent the explicit `all` hint alongside scoped hints; the current
pre-delete decision-ID contract below supersedes the global hint.
The repaired target suite passes 20 cases; the TTL suite passes 14 cases.
The initial CI run failed the TTL worker's drain assertion. It measured SQL
before already-admitted sweeps had finished. The test now joins those specific
sweep promises before measuring passive SQL; unresolved-intent holds remain
intact. Waiting for the entire idle count to reach zero was tested and rejected:
that count correctly includes the retained recovery holds. No production drain
or retry behavior changed for this test repair.
@@ -16,7 +16,8 @@ Do not mark an item done from a timer removal or a green check alone.
| 02 | Remaining chat maintenance | Provider runtime discovery; voice reconciliation; milestones; wake notices; interaction continuations; terminal confirmations; GitHub recovery; Teams maintenance, separately by owner | Pending |
| 03 | Run and task recovery | Orphan runs and active leases; shutdown workspace settlement; scheduled retries; queued runs; stranded assignments; resolved dependencies; task watchdogs; silent-run monitoring; stale locks | Pending |
| 04 | Decision retention and notifications | Retention deadlines and notification delivery are separate reasons | Retention pending; archive delivery [#17475](https://github.com/paperclipai/paperclip/pull/17475) merged after exact-head 5/5 reviews and passing current CI |
| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482), and targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates |
| 05 | Decision expiry and interrupted effects | Expiry, effect recovery, continuation recovery | Continuations [#17478](https://github.com/paperclipai/paperclip/pull/17478) merged; expiry [#17479](https://github.com/paperclipai/paperclip/pull/17479) merged; effects [#17482](https://github.com/paperclipai/paperclip/pull/17482) merged; targets [#17486](https://github.com/paperclipai/paperclip/pull/17486) await current-head review/CI gates |
| 06 | Tool connection health | Configuration/use activation and defined proactive health deadlines | Pending |
@@ -630,3 +631,140 @@ integrates that landed parent and all 21 independent master files. All 143 cases
in its five complete suites pass. Its preceding CI shard interruption remains
recorded in the owning effect-recovery audit; it is not a passing receipt. Fresh
CI/reviews are required on the reconciled head.
Slice 11 is [#17482](https://github.com/paperclipai/paperclip/pull/17482), head
`a5b5994c1d`, based on #17479. It is attached and Greptile review was requested.
Its CI is active with no failure in the latest snapshot. Slice 10's one
serialized-job retry passed; all checks now pass at `63b58cc26f`, with Greptile
5/5 and no unresolved findings. The recorded original timeout remains part of
its verification history.
## Next owner audit: decision target invalidation
Branch `codex/sleep-12-decision-target-invalidation` starts from #17482. No target
migration is implemented or ready yet. This is the final responsibility inside
the current combined decision sweep, separate from retention and notifications.
Preserve its exact policy: only strict effect references count. A missing target
or any cancelled target expires with `target_gone`. If all strict targets are
done and at least one was not done in the saved proposal snapshot, expire with
`target_completed`. Lenient decisions and proposals intentionally targeting
already-done tasks remain open. Include secondary effect references through the
existing `decisionEffectTargetIssueIds` helper. Do not duplicate provider/effect
payload interpretation inside the scheduler or broaden invalidation to every
updated timestamp. TTL validity remains separately enforced on use.
The direct-writer audit found the main `issueService.update` boundary can enter
done/cancelled and already owns an outer transaction. Release preserves an
existing terminal status; checkout/adoption enter in-progress; conversation
writers enter todo/in-progress/in-review; recovery writers enter blocked.
Tree restoration explicitly excludes terminal snapshots. Administrative release
only changes locks/assignment. Route aliases `issueRows` write source-trust or
owner-notification metadata. Raw SQL updates found in issue/project privacy
propagation change privacy only. These are audit findings, not a substitute for
producer integration tests or final inspection.
Three direct deletion paths need accounting: issue removal, routine dispatch
failure cleanup, and company deletion. Routine creation is visible through its
service while dispatch holds a separate transaction, so do not assume its failed
issue could never have become a target. Company deletion currently encounters
restricting decision/run references if decisions exist; do not silently fix or
relax those foreign keys in this slice. Single-target links cascade on issue
deletion, so an after-commit lookup through those links cannot identify deleted
targets. Use pre-write intent and either preserve the affected decision scope or
request a bounded startup-style scan for this rare path.
A candidate design uses the existing coordinator's scoped hints: decision
creation signals its decision ID; terminal issue writes signal their issue ID;
deletion or unknown hints request a bounded full scan. Existing decision-target
links select affected open rows; the domain helper computes invalidation from
current targets and saved snapshots. Stable valid open decisions must become
quiet after their finite reconciliation, even with long retention or no near TTL.
No per-decision timer or new queue table is required.
Keep scoped scan state in the owner, with bounded pages and fair rotation.
Changes arriving during a scan must reset/requeue that scope without being lost
when the admitted pass completes. Failed candidates retain retry debt while
allowing later scopes to progress. Startup reconstructs all open targets from
durable state; rollback and uncertain writes use the existing transaction
protocol. `hasPending` must not perform passive SQL during drain. Shutdown must
finish admitted expiry and leave later candidates for restart. A changed target
can reopen before reconciliation; inspect current state rather than blindly
expiring from the notification itself.
Required tests include the actual issue update/removal producers, creation after
cancellation, intentional done-target proposals, secondary targets, lenient
history, no recurring SQL with valid open decisions, outer transactions,
rollback, restart with deleted target links, failed expiry, concurrent scope
changes, fair paging, drain/resume, and shutdown. Exercise relevant route/service
consumers when adding notification calls to their existing transaction boundary.
Do not remove the periodic target sweep before this coverage exists.
## Slice 12 implementation: decision targets
The target owner, scoped notification adapter, and audited producer calls are
implemented. Startup reconstructs open targets; ordinary terminal writes select
linked decisions; deletion requests a bounded full scan after cascading links.
The aggregate decision scan is removed now that its four separate owners exist.
See [target invalidation](../idle-decision-targets.md) for scope and policy.
The combined real-database/consumer selection passes 233 tests. Workspace
typecheck and build pass. The final target run also passes all 18 cases, including the additional
same-scope concurrent-change regression (234 unique selected cases in total). No new table or migration is introduced. Existing
indexes support both lookup shapes in the 40,000-row completed-history probe.
Slice 11 #17482 has Greptile 5/5 at `a5b5994c1d` and no unresolved findings.
Its first browser shard 3/8 failed in agent-chat fixture cleanup: cancelling a
run hit a PostgreSQL lock timeout, and the following test saw the experimental
flag left enabled. No CI retry has been used. Reproduce the two tests on the
exact parent head before classifying a retry or repair. This parent is not yet
claimed green. The full original inventory remains in scope.
Slice 11 #17482 now has all checks passing at `a5b5994c1d`, Greptile 5/5,
no unresolved findings, and no merge conflict. Its one browser-shard retry passed
after both affected cases passed locally on the exact head (2/2). The original
fixture-cleanup lock timeout remains documented in the PR.
Slice 12 is #17486 at initial head `5062efb368`. Review found that deletion's
absent hint could be dropped when the same outer transaction included a scoped
hint. Both operation orders failed new regressions before the explicit `all`
hint repair. The repaired target/TTL selection passes 34 cases. CI also exposed
a TTL test race: passive-SQL measurement now joins already-admitted sweeps,
without dropping unresolved recovery holds. Fresh CI/review remain required.
The fresh master audit at `991c47bc6e` identified an additional recurring-server
owner: the Grok routine dispatcher starts a one-second settings/dispatch scan
when public MCP is configured, even when Grok is disabled. Include this in the
full stack as slice 13, separately from decision retention. The disabled baseline
issued three settings queries in 3.2 seconds. Its durable prepared dispatches,
atomic claim, and existing uncertain-outcome policy must remain authoritative.
Slice 12 review repair: deletion captures affected open decision IDs under an
issue-row lock before links cascade. Ordinary cleanup without linked decisions
and empty-company deletion no longer signal global work. PostgreSQL text cursors
preserve microsecond precision. Four real regressions failed before repair;
removing the new lock also failed a controlled concurrent-proposal test. All
246 target/routine/decision/effect/expiry/startup/deadline cases and server
types/build pass. The existing restrictive company-deletion FKs and explicit
combined-service compatibility path remain documented. Fresh head gates required.
Slice 12 final combined-sweep cleanup: the repository/package/SDK audit found no
production consumer or public exposure of `decisionService.sweepExpired` after
the final duty split. Removed that method, its cursor/flags/imports and the
unneeded target-policy export. Its behavioral tests now use the four dedicated
workers. Real isolation tests show TTL does not recover effects and effect
recovery does not expire open decisions. All 248 cases in seven complete suites,
server types and build pass after integrating parent `c304273b75`. This supersedes
the earlier manual-caller compatibility claim. Both creation signals remain:
expiry-only creation and scoped target invalidation. Fresh head gates required.
Slice 11 merged at `c304273b75` → `e9e84bb8a1` after exact-head Greptile/Ponytail
5/5 and all 58 checks passed or were neutral/skipped. Slice 12 normally merges
that landed dependency and preserves four independent Runner E2E accounting
files. All runtime and acceptance test sources are byte-identical to its tested
`85ce6545bf` tree (248 cases, server types/build). Fresh final-head CI and reviews
remain required after retargeting to master.
@@ -316,12 +316,15 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
expect(errors.every(error => String(error).includes("wake unavailable"))).toBe(true); errors.length = 0;
});
it("keeps aggregate maintenance from recovering effects when the worker owns them", async () => {
const row = await create(); await claim(row.id, new Date(Date.now() - 120_000));
expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false }))
.toEqual({ expired: 0, resumed: 0 });
it("does not expire overdue open decisions during effect recovery", async () => {
const row = await create();
await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(decisions.id, row.id));
expect(await service().recoverEffects()).toEqual({ resumed: 0 });
expect(await service().get(row.id)).toMatchObject({ status: "open" });
expect(await comments()).toHaveLength(0);
expect(await service().sweepExpired()).toEqual({ expired: 0, resumed: 1 }); expect(await comments()).toHaveLength(1);
expect(wake).not.toHaveBeenCalled();
expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 });
expect(wake).toHaveBeenCalledTimes(1);
});
it.each(["missing_decider", "missing_option"] as const)("settles %s as durable failure and then becomes quiet", async reason => {
@@ -62,8 +62,10 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining,
canReconcile: () => true, onError: error => errors.push(error) });
await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready;
await registerDecisionExpiryWork(coordinator, service(), () => !draining).ready;
const continuation = service(), expiry = service();
await registerDecisionContinuationWork(coordinator, continuation, () => !draining).ready;
await registerDecisionExpiryWork(coordinator, expiry, () => !draining).ready;
return { continuation, expiry };
}
async function quiet() {
await vi.waitFor(async () => {
@@ -218,12 +220,14 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
});
it("keeps a failed expiry retry through drain and preserves pending state", async () => {
const row = await create(new Date(Date.now() + 30_000)); await start();
const row = await create(new Date(Date.now() + 30_000)); const workers = await start();
const sweeps = [vi.spyOn(workers.expiry, "sweepTtlExpiry"), vi.spyOn(workers.continuation, "sweepContinuations")];
await db.execute(sql`create function decision_expiry_fail() returns trigger language plpgsql as $$ begin
if NEW.status = 'expired' then raise exception 'expiry unavailable'; end if; return NEW; end $$`);
await db.execute(sql`create trigger decision_expiry_fail before update on decisions for each row execute function decision_expiry_fail()`);
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors.length).toBeGreaterThan(0); });
draining = true; expect((await service().get(row.id))?.status).toBe("open");
await Promise.allSettled(sweeps.flatMap(sweep => sweep.mock.results.filter(result => result.type === "return").map(result => result.value)));
const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(60_000);
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
await db.execute(sql`drop trigger decision_expiry_fail on decisions`);
@@ -247,12 +251,15 @@ const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTes
expect(wake).not.toHaveBeenCalled(); expect(await service().nextExpiryAt()).toBeNull();
});
it("keeps TTL expiry out of the aggregate sweep when its worker owns the duty", async () => {
it("does not recover stale running effects during TTL expiry", async () => {
const row = await create();
await db.update(decisions).set({ expiresAt: new Date(Date.now() - 1000) }).where(eq(decisions.id, row.id));
expect(await service().sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false })).toEqual({ expired: 0, resumed: 0 });
expect((await service().get(row.id))?.status).toBe("open");
expect(await service().sweepTtlExpiry()).toEqual({ expired: 1 }); expect(wake).toHaveBeenCalledTimes(1);
await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "yes",
decidedByUserId: null, updatedAt: new Date(Date.now() - 120_000), expiresAt: new Date(Date.now() - 1000) })
.where(eq(decisions.id, row.id));
expect(await service().sweepTtlExpiry()).toEqual({ expired: 0 });
expect(await service().get(row.id)).toMatchObject({ status: "decided", executionStatus: "running" });
expect(wake).not.toHaveBeenCalled();
expect(await service().nextExpiryAt()).toBeNull();
});
it("pages past a failed candidate and retries it without repeating completed expirations", async () => {
@@ -0,0 +1,366 @@
import { randomUUID } from "node:crypto";
import { mkdtemp, rm } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { eq, sql } from "drizzle-orm";
import { afterAll, afterEach, beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
import { activityLog, agents, authUsers, companies, companyMemberships, createDb, subscribeDatabaseWork, decisionEffectExecutions,
decisions, decisionRetention, decisionTargetIssues, heartbeatRuns, issueComments, issues } from "@paperclipai/db";
import { getEmbeddedPostgresTestSupport, startEmbeddedPostgresTestDatabase } from "./helpers/embedded-postgres.js";
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { decisionService } from "../services/decisions.js";
import { registerDecisionTargetWork } from "../services/decision-target-work.js";
import { registerDecisionContinuationWork } from "../services/decision-continuation-work.js";
import { createDeliveryWorkCoordinator } from "../services/delivery-work-coordinator.js";
import { issueService } from "../services/issues.js";
import { idleWorkSnapshot } from "../services/task-admission.js";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { deleteCompany } from "../services/company-deletion.js";
import { notifyDecisionTargetIssueRemoval } from "../services/decision-targets.js";
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_DECISION_TARGETS_TEST_DATABASE_URL");
const support = externalUrl ? { supported: true } : await getEmbeddedPostgresTestSupport();
(support.supported ? describe : describe.skip)("decision target worker", () => {
let temporary: Awaited<ReturnType<typeof startEmbeddedPostgresTestDatabase>> | undefined;
let db: ReturnType<typeof createDb>;
let companyId: string, agentId: string, originId: string, targetId: string, runId: string, userId: string;
let coordinator: ReturnType<typeof createDeliveryWorkCoordinator> | undefined;
let draining: boolean;
let recovery: ReturnType<typeof decisionService>;
const errors: unknown[] = [];
const wake = vi.fn(async () => {});
const service = () => decisionService(db, { wakeOriginAgent: wake });
beforeAll(async () => {
if (!externalUrl) temporary = await startEmbeddedPostgresTestDatabase("decision-targets-");
db = createDb(externalUrl ?? temporary!.connectionString, { maxConnections: 6, idleTimeoutSeconds: 0, maxLifetimeSeconds: 0 });
}, 30_000);
beforeEach(async () => {
vi.stubEnv("PAPERCLIP_DECISION_SIGNING_SECRET", "0123456789abcdef0123456789abcdef");
companyId = randomUUID(); agentId = randomUUID(); originId = randomUUID(); targetId = randomUUID(); runId = randomUUID(); userId = randomUUID();
await db.insert(companies).values({ id: companyId, name: "Recovery", issuePrefix: `R${companyId.slice(0, 6)}`, requireBoardApprovalForNewAgents: false });
await db.insert(authUsers).values({ id: userId, name: "Decider", email: `${userId}@example.test`, createdAt: new Date(), updatedAt: new Date() });
await db.insert(companyMemberships).values({ companyId, principalType: "user", principalId: userId, status: "active", membershipRole: "member" });
await db.insert(agents).values({ id: agentId, companyId, name: "Proposer", role: "engineer", status: "active", adapterType: "codex_local", adapterConfig: {}, runtimeConfig: {}, permissions: {} });
await db.insert(issues).values([
{ id: originId, companyId, title: "Origin", status: "in_progress", priority: "medium", assigneeAgentId: agentId, responsibleUserId: userId },
{ id: targetId, companyId, title: "Target", status: "todo", priority: "medium", responsibleUserId: userId },
]);
await db.insert(heartbeatRuns).values({ id: runId, companyId, agentId, status: "running", responsibleUserId: userId, contextSnapshot: { issueId: originId } });
await Promise.all(Array.from({ length: 6 }, () => db.execute(sql`select pg_sleep(0.01)`)));
draining = false; errors.length = 0; wake.mockReset();
vi.useFakeTimers({ toFake: ["Date", "setTimeout", "clearTimeout", "setInterval", "clearInterval"], shouldAdvanceTime: true });
});
afterEach(async () => {
await coordinator?.stop(); coordinator = undefined;
vi.useRealTimers(); vi.restoreAllMocks(); vi.unstubAllEnvs();
await db.execute(sql`drop trigger if exists decision_target_expire_fail on decisions`);
await db.execute(sql`drop function if exists decision_target_expire_fail()`);
await db.delete(decisionEffectExecutions); await db.delete(decisionTargetIssues); await db.delete(decisions);
await db.delete(decisionRetention); await db.delete(activityLog); await db.delete(issueComments); await db.delete(heartbeatRuns);
await db.delete(issues); await db.delete(agents); await db.delete(companyMemberships); await db.delete(authUsers); await db.delete(companies);
expect(errors).toEqual([]);
});
afterAll(async () => { await db?.$client.end({ timeout: 1 }); await temporary?.cleanup(); });
const actor = () => ({ type: "agent" as const, companyId, agentId, runId, source: "agent_jwt" as const,
onBehalfOfUserId: userId, onBehalfOfMemberships: [{ companyId, membershipRole: "member", status: "active" }] });
const board = () => ({ type: "board" as const, userId, companyIds: [companyId], source: "session" as const,
memberships: [{ companyId, membershipRole: "member", status: "active" }] });
const create = (staleness: "strict" | "lenient" = "strict") => service().create({ companyId, agentId, runId, actor: actor(), title: "Comment?", body: "Body",
continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [
{ type: "comment_on_issue", targetIssueId: targetId, staleness, bodyMarkdown: "hello" },
] }] });
async function start() {
coordinator = createDeliveryWorkCoordinator({ owner: db, canRun: () => !draining, canReconcile: () => true,
onError: error => errors.push(error) });
await registerDecisionContinuationWork(coordinator, service(), () => !draining).ready;
recovery = service();
await registerDecisionTargetWork(coordinator, recovery, () => !draining).ready;
}
async function quiet() {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(30_000);
expect(idleWorkSnapshot().active).toBe(0); expect(coordinator!.nextWakeAt()).toBeNull();
}, { timeout: 5000 });
const reads = vi.spyOn(db, "select"), writes = vi.spyOn(db, "update"), executes = vi.spyOn(db, "execute");
await vi.advanceTimersByTimeAsync(60 * 60_000);
expect(reads).not.toHaveBeenCalled(); expect(writes).not.toHaveBeenCalled(); expect(executes).not.toHaveBeenCalled();
reads.mockRestore(); writes.mockRestore(); executes.mockRestore();
}
async function expired(id: string, expiredReason = "target_gone") {
await vi.waitFor(async () => {
await vi.advanceTimersByTimeAsync(30_000);
expect(await service().get(id)).toMatchObject({ status: "expired", metadata: { continuationPending: false, expiredReason } });
}, { timeout: 5000 });
}
async function failExpiry() {
await db.execute(sql`create function decision_target_expire_fail() returns trigger language plpgsql as $$ begin
if NEW.status = 'expired' then raise exception 'expiry unavailable'; end if;
return NEW; end $$`);
await db.execute(sql`create trigger decision_target_expire_fail before update on decisions for each row execute function decision_target_expire_fail()`);
}
const change = (status: "done" | "cancelled") => issueService(db).update(targetId, { status, actorUserId: userId });
it("stays quiet with valid open decisions and wakes from the real terminal update", async () => {
await start(); await quiet(); const row = await create(); await quiet();
await change("done"); await expired(row.id, "target_completed"); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("finishes startup paging across submillisecond expiry values", async () => {
const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions);
await db.insert(decisions).values(Array.from({ length: 101 }, () => ({ ...seed, id: randomUUID() })));
await db.execute(sql`update decisions set expires_at = date_trunc('milliseconds', now() + interval '1 day') + interval '456 microseconds'`);
const owner = service();
await owner.sweepInvalidTargets();
expect(owner.hasDecisionTargetWork()).toBe(true);
await owner.sweepInvalidTargets();
expect(owner.hasDecisionTargetWork()).toBe(false);
expect(wake).not.toHaveBeenCalled();
});
it("reaches a deleted target behind a full page of valid startup decisions", async () => {
const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions);
const invalidId = "ffffffff-ffff-4fff-afff-ffffffffffff";
const valid = Array.from({ length: 100 }, (_, index) => ({ ...seed,
id: `01000000-0000-4000-8000-${String(index).padStart(12, "0")}`,
options: [{ id: "yes", label: "Yes", effects: [{ type: "comment_on_issue" as const, targetIssueId: originId, staleness: "strict" as const, bodyMarkdown: "hello" }] }],
}));
await db.insert(decisions).values([...valid, { ...seed, id: invalidId }]);
await db.insert(decisionTargetIssues).values({ companyId, decisionId: invalidId, issueId: targetId });
const owner = service();
await owner.sweepInvalidTargets();
expect(owner.hasDecisionTargetWork()).toBe(true);
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, (event, hints) => {
if (event === "settled") owner.decisionTargetsChanged(hints);
});
try {
await issueService(db).remove(targetId);
expect(await owner.sweepInvalidTargets()).toEqual({ expired: 1 });
expect((await owner.get(invalidId))?.metadata).toMatchObject({ expiredReason: "target_gone" });
expect(owner.hasDecisionTargetWork()).toBe(false);
expect(wake).toHaveBeenCalledTimes(1);
} finally { unsubscribe(); }
});
it("fences new target references between deletion discovery and commit", async () => {
let entered!: () => void, release!: () => void;
const discovered = new Promise<void>(resolve => { entered = resolve; });
const gate = new Promise<void>(resolve => { release = resolve; });
const removing = db.transaction(async tx => {
await notifyDecisionTargetIssueRemoval(tx, targetId);
entered(); await gate;
await tx.delete(issues).where(eq(issues.id, targetId));
});
let proposing: Promise<{ accepted: boolean }> | undefined;
try {
await discovered;
proposing = create().then(() => ({ accepted: true }), () => ({ accepted: false }));
await vi.waitFor(async () => {
const blocked = await db.execute(sql`select pid from pg_stat_activity where datname = current_database()
and pid <> pg_backend_pid() and wait_event_type = 'Lock' and query like '%decision_target_issues%'`);
expect(blocked).toHaveLength(1);
});
release(); await removing;
await expect(proposing).resolves.toEqual({ accepted: false });
expect(await db.select().from(decisions)).toEqual([]);
expect(wake).not.toHaveBeenCalled();
} finally { release(); await Promise.allSettled([removing, ...(proposing ? [proposing] : [])]); }
});
it("does not signal target work for removal with no linked decisions", async () => {
await create();
const unrelatedId = randomUUID();
await db.insert(issues).values({ id: unrelatedId, companyId, title: "Unrelated", status: "todo", responsibleUserId: userId });
const settled = vi.fn();
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { if (event === "settled") settled(); });
try { await issueService(db).remove(unrelatedId); expect(settled).not.toHaveBeenCalled(); }
finally { unsubscribe(); }
});
it("does not launch global target work after deleting an empty company", async () => {
const emptyCompanyId = randomUUID();
await db.insert(companies).values({ id: emptyCompanyId, name: "Empty", issuePrefix: `E${emptyCompanyId.slice(0, 6)}` });
const home = await mkdtemp(join(tmpdir(), "decision-target-company-"));
vi.stubEnv("PAPERCLIP_HOME", home);
const settled = vi.fn();
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { if (event === "settled") settled(); });
try {
expect(await deleteCompany(db, emptyCompanyId)).toMatchObject({ id: emptyCompanyId });
expect(settled).not.toHaveBeenCalled();
} finally { unsubscribe(); await rm(home, { recursive: true, force: true }); }
});
it("keeps lenient and intentional already-done proposals open and quiet", async () => {
const lenient = await create("lenient"); await change("done"); const intentional = await create();
await start(); await quiet();
expect((await service().get(lenient.id))?.status).toBe("open");
expect((await service().get(intentional.id))?.status).toBe("open"); expect(wake).not.toHaveBeenCalled();
});
it("expires a decision created after its target was cancelled", async () => {
await change("cancelled"); await start(); await quiet(); const row = await create();
await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("reacts to the real removal producer after target links cascade away", async () => {
const row = await create(); await start(); await quiet(); await issueService(db).remove(targetId);
expect(await db.select().from(decisionTargetIssues).where(eq(decisionTargetIssues.decisionId, row.id))).toHaveLength(0);
await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it.each(["cancelled", "deleted"] as const)("reconstructs %s target invalidation at startup", async action => {
const row = await create();
if (action === "deleted") await db.delete(issues).where(eq(issues.id, targetId));
else await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetId));
await start(); await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("includes a strict secondary effect reference in notification scope", async () => {
const blockerId = randomUUID();
await db.insert(issues).values({ id: blockerId, companyId, title: "Blocker", status: "todo", responsibleUserId: userId });
const row = await service().create({ companyId, actor: actor(), agentId, runId, title: "Follow up?", body: "Body",
continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [
{ type: "create_issue", targetIssueId: targetId, staleness: "strict", draft: { title: "Next", blockedByIssueIds: [blockerId] } },
] }] });
await start(); await quiet(); await issueService(db).update(blockerId, { status: "cancelled", actorUserId: userId });
await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("waits for the last strict target to complete", async () => {
const blockerId = randomUUID();
await db.insert(issues).values({ id: blockerId, companyId, title: "Blocker", status: "todo", responsibleUserId: userId });
const row = await service().create({ companyId, actor: actor(), agentId, runId, title: "Follow up?", body: "Body",
continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [
{ type: "create_issue", targetIssueId: targetId, staleness: "strict", draft: { title: "Next", blockedByIssueIds: [blockerId] } },
] }] });
await start(); await quiet(); await change("done"); await quiet();
expect((await service().get(row.id))?.status).toBe("open");
await issueService(db).update(blockerId, { status: "done", actorUserId: userId });
await expired(row.id, "target_completed"); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("waits for the actual issue update's outer transaction and ignores rollback", async () => {
const row = await create(); await start(); await quiet();
await expect(db.transaction(async tx => {
await issueService(db).update(targetId, { status: "cancelled", actorUserId: userId }, tx);
await vi.advanceTimersByTimeAsync(120_000); expect(wake).not.toHaveBeenCalled(); throw new Error("rollback");
})).rejects.toThrow("rollback");
await quiet(); expect((await service().get(row.id))?.status).toBe("open");
await db.transaction(async tx => {
await issueService(db).update(targetId, { status: "cancelled", actorUserId: userId }, tx);
await vi.advanceTimersByTimeAsync(120_000); expect(wake).not.toHaveBeenCalled();
});
await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it.each([true, false])("preserves a deletion mixed with a scoped hint in one transaction, deleteFirst=%s", async deleteFirst => {
const row = await create(); const otherId = randomUUID();
await db.insert(issues).values({ id: otherId, companyId, title: "Unrelated", status: "todo", responsibleUserId: userId });
await start(); await quiet();
await db.transaction(async tx => {
const scoped = issueService(tx as unknown as typeof db);
if (deleteFirst) await scoped.remove(targetId);
await scoped.update(otherId, { status: "done", actorUserId: userId });
if (!deleteFirst) await scoped.remove(targetId);
});
expect(await db.select().from(decisionTargetIssues).where(eq(decisionTargetIssues.decisionId, row.id))).toHaveLength(0);
await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("retains a first target change through drain without passive SQL", async () => {
const row = await create(); await start(); await quiet(); draining = true; await change("cancelled");
const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("checks current state if a cancelled target reopened before reconciliation", async () => {
const row = await create(); await start(); await quiet(); draining = true; await change("cancelled");
await issueService(db).update(targetId, { status: "todo", actorUserId: userId });
draining = false; await quiet(); expect((await service().get(row.id))?.status).toBe("open"); expect(wake).not.toHaveBeenCalled();
});
it("retains failed expiry through drain and retries after the database recovers", async () => {
const row = await create(); await start(); await quiet(); await failExpiry(); await change("cancelled");
await vi.waitFor(async () => { await vi.advanceTimersByTimeAsync(30_000); expect(errors.length).toBeGreaterThan(0); });
draining = true;
const reads = vi.spyOn(db, "select"); await vi.advanceTimersByTimeAsync(120_000);
expect(reads).not.toHaveBeenCalled(); reads.mockRestore();
await db.execute(sql`drop trigger decision_target_expire_fail on decisions`);
draining = false; await expired(row.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
for (const error of errors) expect(error).toMatchObject({ cause: { message: "expiry unavailable" } }); errors.length = 0;
});
it("does not inspect unrelated open decisions on an ordinary scoped change", async () => {
const unrelated = await create();
const otherId = randomUUID(); await db.insert(issues).values({ id: otherId, companyId, title: "Other", status: "todo", responsibleUserId: userId });
const affected = await service().create({ companyId, actor: actor(), agentId, runId, title: "Other?", body: "Body",
continuationPolicy: "wake_origin_agent", options: [{ id: "yes", label: "Yes", effects: [
{ type: "comment_on_issue", targetIssueId: otherId, staleness: "strict", bodyMarkdown: "hello" },
] }] });
await start(); await quiet();
await db.update(decisions).set({ options: sql`'null'::jsonb` }).where(eq(decisions.id, unrelated.id));
await issueService(db).update(otherId, { status: "cancelled", actorUserId: userId });
await expired(affected.id); await quiet(); expect(wake).toHaveBeenCalledTimes(1);
});
it("bounds startup pages and continues after more than one page of valid decisions", async () => {
const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions);
await db.insert(decisions).values(Array.from({ length: 101 }, () => ({ ...seed, id: randomUUID() })));
await start(); expect(recovery.hasDecisionTargetWork()).toBe(true);
await quiet(); expect(await db.select().from(decisions).where(eq(decisions.status, "open"))).toHaveLength(101);
expect(wake).not.toHaveBeenCalled();
});
it("continues past a failing first page and retains its failed candidate", async () => {
const seed = await create(); await db.delete(decisionTargetIssues); await db.delete(decisions);
const ids = Array.from({ length: 101 }, (_, i) => `12000000-0000-4000-8000-${String(i).padStart(12, "0")}`);
await db.insert(decisions).values(ids.map(id => ({ ...seed, id })));
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetId));
await db.execute(sql`create function decision_target_expire_fail() returns trigger language plpgsql as $$ begin
if NEW.id = '12000000-0000-4000-8000-000000000000' and NEW.status = 'expired' then raise exception 'expiry unavailable'; end if;
return NEW; end $$`);
await db.execute(sql`create trigger decision_target_expire_fail before update on decisions for each row execute function decision_target_expire_fail()`);
await start(); expect(wake).toHaveBeenCalledTimes(99); expect((await service().get(ids.at(-1)!))?.status).toBe("open");
await expired(ids.at(-1)!); expect((await service().get(ids[0]))?.status).toBe("open");
await db.execute(sql`drop trigger decision_target_expire_fail on decisions`);
await expired(ids[0]); await quiet(); expect(wake).toHaveBeenCalledTimes(101);
for (const error of errors) expect(error).toMatchObject({ cause: { message: "expiry unavailable" } }); errors.length = 0;
});
it("retains a change to the same issue scope while its earlier pass is still running", async () => {
await change("done"); const intentional = await create();
await db.update(decisions).set({ expiresAt: new Date(Date.now() + 24 * 60 * 60_000) }).where(eq(decisions.id, intentional.id));
await issueService(db).update(targetId, { status: "todo", actorUserId: userId });
const ordinary = await create(); await start(); await quiet();
wake.mockImplementationOnce(async () => { await change("cancelled"); });
await change("done"); await expired(ordinary.id, "target_completed"); await expired(intentional.id); await quiet();
expect(wake).toHaveBeenCalledTimes(2);
});
it("learns a decision created during invalidation delivery", async () => {
const first = await create(); await change("cancelled"); let nextId = "";
wake.mockImplementationOnce(async () => { nextId = (await create()).id; });
await start(); await expired(first.id); await vi.waitFor(() => expect(nextId).not.toBe(""));
await expired(nextId); await quiet(); expect(wake).toHaveBeenCalledTimes(2);
});
it("finishes admitted invalidation and leaves later rows for restart during shutdown", async () => {
const first = await create(), second = await create(); await change("cancelled");
let release!: () => void; const gate = new Promise<void>(resolve => { release = resolve; });
wake.mockImplementationOnce(async () => gate);
const starting = start(); let stopped = false; let stopping: Promise<void>;
try {
await vi.waitFor(() => expect(wake).toHaveBeenCalledTimes(1));
stopping = coordinator!.stop().then(() => { stopped = true; });
await vi.advanceTimersByTimeAsync(60_000); expect(stopped).toBe(false);
} finally { release(); }
await starting; await stopping!; coordinator = undefined;
expect(wake).toHaveBeenCalledTimes(1);
const states = await Promise.all([service().get(first.id), service().get(second.id)]);
expect(states.map(row => row!.status).sort()).toEqual(["expired", "open"]);
await start(); await quiet(); expect(wake).toHaveBeenCalledTimes(2);
});
});
+15 -14
View File
@@ -225,12 +225,12 @@ describePg("decisionService", () => {
...original(source, options), archiveReviewedManifest: async () => { throw new Error("archive_failure"); },
})) : undefined;
try {
expect(await service().sweepExpired()).toMatchObject({ resumed: 1 });
expect(await service().recoverEffects()).toMatchObject({ resumed: 1 });
const saved = await service().get(row.id);
expect(saved).toMatchObject({ executionStatus: "failed", metadata: { archiveProposalError: failure, continuationPending: false } });
expect(saved!.metadata.continuationDeliveredAt).toEqual(expect.any(String));
expect(wakes).toHaveLength(1);
await service().sweepExpired();
await service().recoverEffects();
expect(wakes).toHaveLength(1);
} finally { factory?.mockRestore(); }
});
@@ -512,7 +512,7 @@ describePg("decisionService", () => {
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1);
});
it("recovers stale running decisions from the bounded server sweep", async () => {
it("recovers stale running decisions from the effect worker", async () => {
process.env.PAPERCLIP_DECISIONS_RECOVERY_GRACE_MS = "0";
const created = await createCommentDecision();
await db.update(decisions).set({ status: "decided", executionStatus: "running", chosenOptionId: "yes", decidedByUserId,
@@ -520,7 +520,7 @@ describePg("decisionService", () => {
await db.insert(decisionEffectExecutions).values({ decisionId: created.id, effectIndex: 0, effectType: "comment_on_issue",
targetIssueId, status: "claimed" });
expect(await service().sweepExpired()).toEqual({ expired: 0, resumed: 1 });
expect(await service().recoverEffects()).toEqual({ resumed: 1 });
expect((await service().get(created.id))?.executionStatus).toBe("succeeded");
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1);
expect(wakes).toEqual([{ companyId, agentId, issueId: originIssueId, decisionId: created.id, outcome: "decided" }]);
@@ -538,7 +538,7 @@ describePg("decisionService", () => {
expect((await service().get(created.id))?.metadata).toMatchObject({ continuationPending: true });
expect(await db.select().from(issueComments).where(eq(issueComments.issueId, targetIssueId))).toHaveLength(1);
await service().sweepExpired();
await service().sweepContinuations();
expect(wakes).toEqual([{ companyId, agentId, issueId: originIssueId, decisionId: created.id, outcome: "decided" }]);
expect((await service().get(created.id))?.metadata).toMatchObject({ continuationPending: false });
@@ -613,8 +613,8 @@ describePg("decisionService", () => {
const second = await createCommentDecision("lenient", { idempotencyKey: "batch-2", expiresAt: nearFutureExpiry() });
await expireDecisionNow(first.id);
await expireDecisionNow(second.id);
expect((await service().sweepExpired()).expired).toBe(1);
expect((await service().sweepExpired()).expired).toBe(1);
expect((await service().sweepTtlExpiry()).expired).toBe(1);
expect((await service().sweepTtlExpiry()).expired).toBe(1);
});
it("falls back to the default sweep batch size for invalid configuration", async () => {
@@ -624,7 +624,7 @@ describePg("decisionService", () => {
await expireDecisionNow(first.id);
await expireDecisionNow(second.id);
await expect(service().sweepExpired()).resolves.toMatchObject({ expired: 2 });
await expect(service().sweepTtlExpiry()).resolves.toMatchObject({ expired: 2 });
});
it("expires TTL and target-gone decisions and wakes the origin agent", async () => {
@@ -633,7 +633,8 @@ describePg("decisionService", () => {
await db.update(decisions).set({ expiresAt: new Date(0) }).where(eq(decisions.id, ttl.id));
await db.update(issues).set({ status: "cancelled" }).where(eq(issues.id, targetIssueId));
await expireDecisionNow(ttl.id);
expect((await service().sweepExpired()).expired).toBe(2);
expect((await service().sweepTtlExpiry()).expired).toBe(1);
expect((await service().sweepInvalidTargets()).expired).toBe(1);
const rows = await db.select().from(decisions);
expect(rows.find((row) => row.id === ttl.id)?.metadata).toMatchObject({ expiredReason: "ttl" });
expect(rows.find((row) => row.id === gone.id)?.metadata).toMatchObject({ expiredReason: "target_gone" });
@@ -645,7 +646,7 @@ describePg("decisionService", () => {
const lenient = await createCommentDecision("lenient", { idempotencyKey: "lenient-survives" });
await db.update(issues).set({ status: "done" }).where(eq(issues.id, targetIssueId));
expect((await service().sweepExpired()).expired).toBe(1);
expect((await service().sweepInvalidTargets()).expired).toBe(1);
expect((await service().get(completed.id))?.metadata).toMatchObject({ expiredReason: "target_completed" });
expect((await service().get(lenient.id))?.status).toBe("open");
@@ -656,7 +657,7 @@ describePg("decisionService", () => {
await db.update(issues).set({ status: "done" }).where(eq(issues.id, targetIssueId));
const reopen = await createCommentDecision("strict", { idempotencyKey: "already-done" });
expect((await service().sweepExpired()).expired).toBe(0);
expect((await service().sweepInvalidTargets()).expired).toBe(0);
expect((await service().get(reopen.id))?.status).toBe("open");
});
@@ -691,7 +692,7 @@ describePg("decisionService", () => {
});
await db.update(issues).set({ status: "done" }).where(eq(issues.id, blockerId));
expect((await service().sweepExpired()).expired).toBe(1);
expect((await service().sweepInvalidTargets()).expired).toBe(1);
expect((await service().get(created.id))?.metadata).toMatchObject({ expiredReason: "target_completed" });
});
@@ -716,7 +717,7 @@ describePg("decisionService", () => {
await service().decide({ id: acceptedAgain.id, optionId: "assign", decidedByUserId, userActor: boardActor() });
await service().dismiss(rejected.id, decidedByUserId, boardActor(), "Not this time");
await expireDecisionNow(stale.id);
await service().sweepExpired();
await service().sweepTtlExpiry();
const stats = await service().stats(companyId, { originAgentId: agentId });
expect(stats.filters).toEqual({ originAgentId: agentId, since: null });
@@ -825,7 +826,7 @@ describePg("decisionService", () => {
if (action === "decide") await svc.decide({ id: row.id, optionId: "yes", decidedByUserId, userActor: boardActor() });
if (action === "cancel") await svc.cancel(row.id, { actorType: "agent", actorId: agentId });
if (action === "dismiss") await svc.dismiss(row.id, decidedByUserId, boardActor());
if (action === "expire") { await expireDecisionNow(row.id); await svc.sweepExpired(new Date(), { includeContinuations: false }); }
if (action === "expire") { await expireDecisionNow(row.id); await svc.sweepTtlExpiry(); }
await quiet(); await delivered(row.id);
expect(wake).toHaveBeenCalledTimes(1);
expect(wake.mock.calls[0][0]).toMatchObject({ decisionId: row.id, companyId, agentId, issueId: originIssueId });
@@ -1,3 +1,4 @@
import { resolveExternalTestDatabaseUrl } from "./helpers/external-test-database.js";
import { subscribeDatabaseWork } from "@paperclipai/db";
import { DELIVERY_QUEUES } from "../services/delivery-work-notifications.js";
import { createHmac, randomUUID } from "node:crypto";
@@ -38,7 +39,8 @@ import * as providerRegistry from "../secrets/provider-registry.ts";
import { routineService } from "../services/routines.ts";
import { secretService } from "../services/secrets.ts";
const embeddedPostgresSupport = await getEmbeddedPostgresTestSupport();
const externalUrl = resolveExternalTestDatabaseUrl("PAPERCLIP_ROUTINES_TEST_DATABASE_URL");
const embeddedPostgresSupport = externalUrl ? { supported: true, reason: undefined } : await getEmbeddedPostgresTestSupport();
const describeEmbeddedPostgres = embeddedPostgresSupport.supported ? describe : describe.skip;
const originalSecretsProviderEnv = process.env.PAPERCLIP_SECRETS_PROVIDER;
const originalPaperclipApiUrlEnv = process.env.PAPERCLIP_API_URL;
@@ -55,8 +57,8 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => {
beforeAll(async () => {
process.env.PAPERCLIP_API_URL = "http://localhost:3100";
tempDb = await startEmbeddedPostgresTestDatabase("paperclip-routines-service-");
db = createDb(tempDb.connectionString);
if (!externalUrl) tempDb = await startEmbeddedPostgresTestDatabase("paperclip-routines-service-");
db = createDb(externalUrl ?? tempDb!.connectionString);
}, 20_000);
afterEach(async () => {
@@ -2167,7 +2169,10 @@ describeEmbeddedPostgres("routine service live-execution coalescing", () => {
},
});
const run = await svc.runRoutine(routine.id, { source: "manual" });
const events: string[] = [];
const unsubscribe = subscribeDatabaseWork(db, DELIVERY_QUEUES.decisionTargets, event => { events.push(event); });
const run = await svc.runRoutine(routine.id, { source: "manual" }).finally(unsubscribe);
expect(events).toEqual([]);
expect(run.status).toBe("failed");
expect(run.failureReason).toContain("queue unavailable");
@@ -348,7 +348,9 @@ vi.mock("../services/index.js", () => ({
list: vi.fn(async () => ({ items: [], nextCursor: null })),
})),
decisionService: vi.fn(() => ({
sweepExpired: vi.fn(async () => ({ expired: 0 })),
sweepInvalidTargets: vi.fn(async () => ({ expired: 0 })),
decisionTargetsChanged: vi.fn(),
hasDecisionTargetWork: vi.fn(() => false),
recoverEffects: vi.fn(async () => ({ resumed: 0 })),
nextEffectRecoveryAt: vi.fn(async () => null),
sweepTtlExpiry: vi.fn(async () => ({ expired: 0 })),
@@ -926,7 +928,8 @@ describe("startServer feedback export wiring", () => {
expect(decisions.recoverEffects).toHaveBeenCalledTimes(1);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(true);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(true);
expect(decisions.sweepExpired).toHaveBeenCalledWith(expect.any(Date), { includeContinuations: false, includeExpiry: false, includeEffects: false });
expect(decisions.sweepInvalidTargets).toHaveBeenCalledTimes(1);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionTargets)).toBe(true);
} finally {
setIntervalSpy.mockRestore();
}
@@ -966,6 +969,7 @@ describe("startServer feedback export wiring", () => {
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionContinuation)).toBe(false);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionExpiry)).toBe(false);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionEffectRecovery)).toBe(false);
expect(deliveryWorkRegisterMock.mock.calls.some(([queue]) => queue === DELIVERY_QUEUES.decisionTargets)).toBe(false);
} finally {
setIntervalSpy.mockRestore();
}
+3 -4
View File
@@ -1,3 +1,4 @@
import { registerDecisionTargetWork } from "./services/decision-target-work.js";
import { registerDecisionEffectRecoveryWork } from "./services/decision-effect-recovery-work.js";
import { registerDecisionExpiryWork } from "./services/decision-expiry-work.js";
import { registerDecisionContinuationWork } from "./services/decision-continuation-work.js";
@@ -1845,7 +1846,8 @@ async function startServerWithDatabaseTeardown(
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
await step("decision_effect_recovery_worker", () => registerDecisionEffectRecoveryWork(deliveryWork, decisionExecutor,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
await step("decision_expiry", () => decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false }));
await step("decision_target_worker", () => registerDecisionTargetWork(deliveryWork, decisionExecutor,
() => !isIdleTaskDrainActive() && !isWarmStandby() && !heartbeatSchedulerStopped).ready);
const loginSchedule = loginCleanupSchedule(db, config.heartbeatSchedulerIntervalMs);
await step("login_cleanup_workers", () => registerLoginCleanupWork(deliveryWork, [
{ queue: DELIVERY_QUEUES.adapterLoginCleanup, run: () => adapterLoginReaper.sweep().then(logAdapterLoginReaperResult), next: loginSchedule.adapter },
@@ -1866,9 +1868,6 @@ async function startServerWithDatabaseTeardown(
trackHeartbeatSchedulerWork((async () => {
if (isIdleTaskDrainActive() || heartbeatSchedulerStopped) return;
scheduleRoutineStatusReconcile();
trackHeartbeatSchedulerWork(decisionExecutor.sweepExpired(new Date(), { includeContinuations: false, includeExpiry: false, includeEffects: false }).catch((err: unknown) => {
logger.error({ err }, "decision expiry sweep failed");
}));
trackHeartbeatSchedulerWork(runRetentionSweep().catch((err: unknown) => {
logger.error({ err }, "decision retention sweep failed");
}));
@@ -0,0 +1,17 @@
import type { createDeliveryWorkCoordinator } from "./delivery-work-coordinator.js";
import { DELIVERY_QUEUES } from "./delivery-work-notifications.js";
import type { decisionService } from "./decisions.js";
export function registerDecisionTargetWork(
coordinator: ReturnType<typeof createDeliveryWorkCoordinator>,
service: Pick<ReturnType<typeof decisionService>, "sweepInvalidTargets" | "decisionTargetsChanged" | "hasDecisionTargetWork">,
canRun: () => boolean,
) {
return coordinator.register(DELIVERY_QUEUES.decisionTargets, {
retryMs: 30_000,
pausePassiveMaintenance: true,
onSettled: service.decisionTargetsChanged,
run: signal => canRun() ? service.sweepInvalidTargets(new Date(), signal) : Promise.resolve(),
hasPending: async () => service.hasDecisionTargetWork(),
});
}
+93
View File
@@ -0,0 +1,93 @@
import { and, asc, eq, getTableColumns, inArray, sql } from "drizzle-orm";
import { decisions, decisionTargetIssues, issues, type Db } from "@paperclipai/db";
import { decisionEffectTargetIssueIds } from "@paperclipai/shared";
import type { decisionExpiryService } from "./decision-expiry.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
type Decision = typeof decisions.$inferSelect;
type Cursor = { expiresAt: string; id: string };
type Scope = { kind: "all" | "issue" | "decision"; id?: string; cursor?: Cursor };
const open = sql`${decisions.status} = 'open'`;
export function notifyDecisionTargetWork(transaction: object, hint: `issue:${string}` | `decision:${string}`) {
return notifyDeliveryWork(transaction, DELIVERY_QUEUES.decisionTargets, hint);
}
export async function notifyDecisionTargetIssueRemoval(transaction: Pick<Db, "select" | "selectDistinct">, issueId: string) {
const [issue] = await transaction.select({ companyId: issues.companyId }).from(issues)
.where(eq(issues.id, issueId)).for("update");
if (!issue) return;
const linked = await transaction.selectDistinct({ decisionId: decisionTargetIssues.decisionId }).from(decisionTargetIssues)
.innerJoin(decisions, eq(decisions.id, decisionTargetIssues.decisionId))
.where(and(eq(decisionTargetIssues.companyId, issue.companyId), eq(decisionTargetIssues.issueId, issueId), eq(decisions.companyId, issue.companyId), open));
for (const row of linked) await notifyDecisionTargetWork(transaction, `decision:${row.decisionId}`);
}
export function decisionTargetService(db: Db, expire: ReturnType<typeof decisionExpiryService>["expire"]) {
const scopes = new Map<string, Scope>([["all", { kind: "all" }]]);
function changed(hints?: ReadonlySet<string>) {
if (!hints?.size) { scopes.set("all", { kind: "all" }); return; }
for (const hint of hints) {
const [kind, id] = hint.split(":");
if ((kind === "issue" || kind === "decision") && id) scopes.set(hint, { kind, id });
else scopes.set("all", { kind: "all" });
}
}
async function reason(decision: Decision): Promise<"target_gone" | "target_completed" | null> {
const strictIds = new Set(decision.options.flatMap(option => option.effects
.filter(effect => effect.staleness === "strict").flatMap(decisionEffectTargetIssueIds)));
if (strictIds.size === 0) return null;
const targets = await db.select({ id: issues.id, status: issues.status }).from(issues)
.where(and(eq(issues.companyId, decision.companyId), inArray(issues.id, [...strictIds])));
if (targets.length !== strictIds.size || targets.some(target => target.status === "cancelled")) return "target_gone";
const snapshots = decision.targetSnapshots as Record<string, { status?: string }>;
return targets.every(target => target.status === "done") && targets.some(target => snapshots[target.id]?.status !== "done")
? "target_completed" : null;
}
async function sweep(now = new Date(), signal?: AbortSignal) {
let expired = 0, inspected = 0;
const errors: unknown[] = [];
for (const [key, scope] of [...scopes]) {
if (signal?.aborted || inspected >= 100) break;
if (scopes.get(key) !== scope) continue;
scopes.delete(key);
const limit = 100 - inspected;
try {
const rows = await db.select({ ...getTableColumns(decisions), cursorExpiresAt: sql<string>`${decisions.expiresAt}::text` }).from(decisions).where(and(open,
scope.kind === "decision" ? eq(decisions.id, scope.id!) : undefined,
scope.kind === "issue" ? inArray(decisions.id, db.select({ id: decisionTargetIssues.decisionId })
.from(decisionTargetIssues).where(eq(decisionTargetIssues.issueId, scope.id!))) : undefined,
scope.cursor ? sql`(${decisions.expiresAt}, ${decisions.id}) > (${scope.cursor.expiresAt}::timestamptz, ${scope.cursor.id}::uuid)` : undefined,
)).orderBy(asc(decisions.expiresAt), asc(decisions.id)).limit(limit);
inspected += Math.max(1, rows.length);
for (const row of rows) {
if (signal?.aborted) {
if (!scopes.has(key)) scopes.set(key, scope);
return { expired };
}
try {
const invalid = await reason(row);
if (invalid) expired += await expire(row, invalid, now);
} catch (error) {
errors.push(error);
const retry = `decision:${row.id}`;
if (!scopes.has(retry)) scopes.set(retry, { kind: "decision", id: row.id });
}
scope.cursor = { expiresAt: row.cursorExpiresAt, id: row.id };
}
if (rows.length === limit && !scopes.has(key)) scopes.set(key, scope);
} catch (error) {
errors.push(error); inspected++;
if (!scopes.has(key)) scopes.set(key, scope);
}
}
if (errors.length === 1) throw errors[0];
if (errors.length) throw new AggregateError(errors, "Decision target reconciliation failed");
return { expired };
}
return { sweep, changed, hasPending: () => scopes.size > 0 };
}
+8 -38
View File
@@ -1,5 +1,6 @@
import { decisionTargetService, notifyDecisionTargetWork } from "./decision-targets.js";
import { randomUUID } from "node:crypto";
import { and, asc, count, desc, eq, gt, gte, inArray, sql } from "drizzle-orm";
import { and, asc, count, desc, eq, gte, inArray, sql } from "drizzle-orm";
import type { Db } from "@paperclipai/db";
import { databaseWorkScopeKey, companyMemberships, decisionBundles, decisionEffectExecutions, decisionRetention, decisions, decisionTargetIssues, heartbeatRuns, issueRelations, issues } from "@paperclipai/db";
import { ATTENTION_SOURCE_KINDS, decisionEffectTargetIssueIds } from "@paperclipai/shared";
@@ -15,7 +16,7 @@ import {
import { decisionRetentionService, hashAttentionArchiveManifest } from "./decision-retention.js";
import { decisionContinuationService, type DecisionContinuationWake } from "./decision-continuations.js";
import { decisionEffectRecoveryService, decisionSweepBatchSize } from "./decision-effect-recovery.js";
import { decisionEffectRecoveryService } from "./decision-effect-recovery.js";
import { decisionExpiryService } from "./decision-expiry.js";
import { notifyDecisionCreated, writeDecisionChange } from "./decision-work-notifications.js";
@@ -128,13 +129,13 @@ function boardCanActDirectly(actor: AuthorizationActor, companyId: string) {
export function decisionService(db: Db, options: DecisionServiceOptions) {
const continuations = decisionContinuationService(db, options.wakeOriginAgent);
const expiry = decisionExpiryService(db, continuations.deliver);
const targets = decisionTargetService(db, expiry.expire);
const effectRecovery = decisionEffectRecoveryService(db, recoverDecisionEffects);
const scope = databaseWorkScopeKey(db);
let flights = effectFlights.get(scope);
if (!flights) effectFlights.set(scope, flights = new Map());
const executing = flights;
const authz = authorizationService(db);
let targetSweepCursor: string | null = null;
type CreateInput = { companyId: string; actor: AuthorizationActor; agentId: string; runId: string; bundleId?: string | null;
ruleKey?: string | null; title: string; body: string; options: DecisionOption[]; inputs?: DecisionInput[] | null; expiresAt?: Date | null;
idempotencyKey?: string | null; continuationPolicy?: "none" | "wake_origin_agent"; metadata?: Record<string, unknown> };
@@ -239,6 +240,7 @@ export function decisionService(db: Db, options: DecisionServiceOptions) {
};
const id = randomUUID();
await notifyDecisionCreated(dbOrTx);
await notifyDecisionTargetWork(dbOrTx, `decision:${id}`);
const [created] = await dbOrTx.insert(decisions).values({ id, companyId: input.companyId, bundleId: input.bundleId ?? null,
originAgentId: input.agentId, originIssueId: provenance.issueId, originRunId: input.runId, ruleKey: input.ruleKey ?? null,
title: input.title, body: input.body, options: input.options, inputs: input.inputs ?? null, expiresAt,
@@ -760,40 +762,8 @@ export function decisionService(db: Db, options: DecisionServiceOptions) {
return true;
}
async function sweepExpired(now = new Date(), options: { includeContinuations?: boolean; includeExpiry?: boolean; includeEffects?: boolean } = {}) {
const batchSize = decisionSweepBatchSize();
const { resumed } = options.includeEffects === false ? { resumed: 0 } : await effectRecovery.sweep(now, batchSize);
if (options.includeContinuations !== false) await continuations.sweep(batchSize);
const ttlRows = options.includeExpiry === false ? [] : await expiry.due(now, batchSize);
const remaining = batchSize - ttlRows.length;
const targetRows = remaining > 0
? await db.select().from(decisions)
.where(and(eq(decisions.status, "open"), ...(targetSweepCursor ? [gt(decisions.id, targetSweepCursor)] : [])))
.orderBy(asc(decisions.id)).limit(remaining)
: [];
targetSweepCursor = targetRows.length === remaining && targetRows.length > 0 ? targetRows[targetRows.length - 1]!.id : null;
const rows = [...new Map([...ttlRows, ...targetRows].map((row) => [row.id, row])).values()]; let expired = 0;
for (const decision of rows) { const strictTargetIds = new Set(decision.options.flatMap((option) => option.effects.filter((effect) => effect.staleness === "strict").flatMap(decisionEffectTargetIssueIds)));
const targets = strictTargetIds.size > 0
? await db.select({ id: issues.id, status: issues.status }).from(issues).where(and(eq(issues.companyId, decision.companyId), inArray(issues.id, [...strictTargetIds])))
: [];
const targetGone = targets.length !== strictTargetIds.size || targets.some((target) => target.status === "cancelled");
// A decision whose strict targets all completed after it was proposed is moot:
// the strict guard would skip its effects anyway, so retire it instead of
// leaving it pending until TTL. Targets that were already done at proposal
// time don't count — those decisions intentionally act on a done issue.
const snapshots = decision.targetSnapshots as Record<string, Snapshot>;
const targetsCompleted = !targetGone && strictTargetIds.size > 0 &&
targets.every((target) => target.status === "done") &&
targets.some((target) => snapshots[target.id]?.status !== "done");
if (!targetGone && !targetsCompleted && (options.includeExpiry === false || decision.expiresAt > now)) continue;
const reason = targetGone ? "target_gone" : targetsCompleted ? "target_completed" : "ttl";
expired += await expiry.expire(decision, reason, now);
}
return { expired, resumed };
}
return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepExpired, sweepContinuations: continuations.sweep,
return { create, createBundle, get, list, stats, outcome, decide, cancel, dismiss, sweepContinuations: continuations.sweep,
sweepTtlExpiry: expiry.sweep, nextExpiryAt: expiry.next,
recoverEffects: effectRecovery.sweep, nextEffectRecoveryAt: effectRecovery.next };
recoverEffects: effectRecovery.sweep, nextEffectRecoveryAt: effectRecovery.next,
sweepInvalidTargets: targets.sweep, decisionTargetsChanged: targets.changed, hasDecisionTargetWork: targets.hasPending };
}
@@ -9,6 +9,7 @@ export const DELIVERY_QUEUES = {
executionStatus: "execution-status",
executionDisposition: "execution-disposition",
environmentCleanup: "environment-cleanup",
decisionTargets: "decision-targets",
decisionEffectRecovery: "decision-effect-recovery",
decisionExpiry: "decision-expiry",
decisionContinuation: "decision-continuation",
+6
View File
@@ -1,4 +1,6 @@
import { notifyDecisionTargetWork, notifyDecisionTargetIssueRemoval } from "./decision-targets.js";
import { assertOnboardingFirstTaskCreationAllowed } from "./onboarding-first-task-admission.js";
import { resolveConnectionIntentContinuationSource, resolveItemVerdictContinuationWake } from "./chat-interaction-continuation.js";
import { insertIssueComment } from "./issue-comment-writes.js";
import { measureRunTransaction } from "./execution-transaction-timing.js";
@@ -11384,6 +11386,9 @@ export function issueService(db: Db) {
(patch.status !== undefined || patch.assigneeAgentId !== undefined || patch.assigneeUserId !== undefined))) {
await notifyDeliveryWork(tx, DELIVERY_QUEUES.issueMonitors);
}
if (patch.status === "done" || patch.status === "cancelled") {
await notifyDecisionTargetWork(tx, `issue:${id}`);
}
const updated = await tx
.update(issues)
.set(patch)
@@ -11791,6 +11796,7 @@ export function issueService(db: Db) {
try {
await notifyTerminalWorkspaceWork(tx, `issue:${id}`);
await requestRestorePointDeletion(tx, { sourceIssueId: id });
await notifyDecisionTargetIssueRemoval(tx, id);
removedIssue = await tx
.delete(issues)
.where(eq(issues.id, id))
+2
View File
@@ -1,3 +1,4 @@
import { notifyDecisionTargetIssueRemoval } from "./decision-targets.js";
import { DELIVERY_QUEUES, notifyDeliveryWork } from "./delivery-work-notifications.js";
import { verifyAppWebhook } from "./app-webhook.js";
import crypto from "node:crypto";
@@ -1964,6 +1965,7 @@ export function routineService(
return updated ?? createdRun;
} catch (error) {
if (createdIssue) {
await notifyDecisionTargetIssueRemoval(txDb, createdIssue.id);
await txDb.delete(issues).where(eq(issues.id, createdIssue.id));
}
const failureReason = error instanceof Error ? error.message : String(error);