Commit Graph
4 Commits
Author SHA1 Message Date
ce332ecec0 fix(execution-admission): keep admitted project sets after settlement (#18871)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - #18746 added execution-admission storage: one admission row per
attempt and one charge row per admitted project.
> - Settlement deletes every charge row. The admission row keeps only
the primary project and an opaque context revision.
> - Thus, after settlement, the complete admitted project set is lost.
> - A retained provider session still carries the references of its
previous admission. Example: a task mentions A and B, the admission
settles, and then a user removes B from the task text. The next
admission must still count B, or start a fresh context. The primary
project or the context token cannot supply B.
> - This pull request keeps the complete admitted set on the admission
row after settlement. Charge rows stay the only active capacity.
> - The benefit is that later retained-session resolution can read the
previous set. Settled history does not hold capacity or block project
deletion.

## Linked Issues or Issue Description

Refs #18746 (corrects its storage). Related: #2831, #14333.

**Subsystem affected**

Server execution admission (`server/src/modules/execution-admission/`)
and the `packages/db` schema.

**Problem or motivation**

`settleAdmission` deletes all charges of an admission. No other storage
records the complete project set. After settlement, the set cannot be
recovered. Current task text is not a replacement, because it can change
after admission.

**Proposed solution**

Add `execution_admissions.admitted_project_ids uuid[]`. It holds the
sorted, distinct set. It is written once at admission and a trigger
blocks changes. It has no project foreign key. `{}` is a known empty
set. Null means unknown. Only settled rows can have null. Migration
`0428` fills the set of unsettled `0426` rows from their charges,
because those charges are still complete. Settled `0426` rows stay
unknown.

**Alternatives considered**

- Keep charge rows after settlement with a `released_at` column and a
partial active index. The project foreign key would then block project
deletion permanently, or the key must be removed from active charges.
One table would also hold two different facts.
- A separate history table with one row per project. This is more rows,
joins and constraints for data that is only read as a whole set by
admission ID.
- Encode the set in `context_revision`. That token is opaque and owned
by the context owner. The database cannot validate it.
- Use `heartbeat_runs.context_snapshot`. It is mutable and is per run,
not per admission attempt.

## What Changed

- Schema and migration `0428_retained_admitted_project_sets`: add
`admitted_project_ids`, a check (null only when settled, no null
elements, at most 100 entries) and an immutability trigger. The backfill
fills unsettled rows from their charges and leaves settled rows null.
The migration is safe to run again. Migration `0426` is not changed.
- `recordAdmission` writes the set. A replay compares against the stored
set. `settleAdmission` still deletes the whole charge set in one
transaction and does not change the stored set.
- Add `readAdmittedProjects` (company-scoped, by admission ID). It
returns `known` with the set, or `unknown`.
- Add a pure `retainedSessionProjects` rule: a retained session uses
previous ∪ current. An unknown previous set returns
`fresh_context_required`. No consumer uses it yet; C1b integrates
retained sessions.
- Update `doc/DATABASE.md` and `server/src/modules/README.md`.

Not changed: no public cap writes, no claim or dispatch integration, no
session integration, no provider activation, no `serialize` → cap 1
migration. Active counts still use only the indexed charge rows.

## Verification

Red, on unchanged `master` code, with the new tests:

- `server`: `npx vitest run
src/modules/execution-admission/adapters/postgres.test.ts`. The new
tests fail. With a temporary reader that uses the only storage `master`
has (charges), the settled A+B admission reads `[]` instead of `[A, B]`.
- `packages/db`: `npx vitest run
src/execution-admission-migration.test.ts` fails, because no `0428`
migration exists.

Green, with this change:

- `server`: `npx vitest run src/modules/execution-admission`: 27 passed.
The new PostgreSQL test covers: A+B admission → settlement → set still
readable, charges and counts zero, settlement replay keeps the set, a
second attempt with a different set, primaryless and empty sets, company
isolation, rollback, immutability, the check constraint, and deletion of
a project that has only settled history (its ID stays in the set). It
also checks that a project with an active charge still cannot be
deleted.
- `packages/db`: `npx vitest run
src/execution-admission-migration.test.ts`: 2 passed. The upgrade test
starts from the shipped `0426` schema with an unsettled A+B row, an
unsettled empty row and a settled row. After the upgrade the sets are
`[A, B]`, `{}` and null. A second run of `0428` changes nothing.
- `pnpm --filter @paperclipai/db typecheck` and `check:migrations`
passed. `node scripts/check-module-boundaries.mjs` passed.
- Not run locally: the server layered typecheck. The local process was
killed for memory (a 5.8 GiB limit shared with other jobs). CI runs it.

This PR does not prove C2 claim races or C3 lifecycle authority.

## Risks

- Migration: one `ADD COLUMN`, an `UPDATE` that only touches unsettled
rows (none in production, because nothing writes admissions yet), a
check and a trigger. It does not delete or rewrite data.
- Settled rows created before `0428` stay unknown. A future consumer
must start a fresh context for them. It must not treat them as empty or
primary-only.
- The stored set has no project foreign key. Company scope is checked at
admission time, under the same project locks and the charge foreign
keys.

Size: about 70 production lines (schema 8, migration SQL 22, adapter 28,
domain 15), 105 test lines and 24 doc lines. Generated metadata
(`0428_snapshot.json`, `_journal.json`, removal of `0422_snapshot.json`
by the standard pruning) is separate.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, embedded PostgreSQL tests). Default context
window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:31:10 -08:00
fef86e32e9 feat(execution-admission): add project concurrency policy and admission storage (#18746)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Projects group tasks. Agents run tasks as heartbeat runs.
> - Today, workspace mode and shared-workspace flags decide whether
tasks run at the same time. A project cannot set its own limit on
concurrent task executions.
> - A correct project limit must count every project that a task uses,
including @-mentioned projects. It must also keep the count when a
project is archived or deleted, and when a run is admitted more than one
time.
> - This change cannot go into one pull request with claims and
settlement. Thus this pull request adds only the policy semantics and
the durable accounting storage that later claims will use.
> - This pull request adds a new `execution-admission` module, three
tables and the deletion guards. No route writes a cap and no claim path
enforces it.
> - The benefit is a tested storage contract (constraints, idempotency,
rollback, indexed counts, lifecycle preservation) before any behavior
changes.

## Linked Issues or Issue Description

Refs #2831, Refs #14333 (related agent-level and adapter-level limits;
not duplicates).

**Subsystem affected**

Server execution admission (new module
`server/src/modules/execution-admission/`) and the `packages/db` schema.

**Problem or motivation**

A project cannot limit how many of its task executions are active. The
current shared-workspace flags mix resource safety with capacity. A
future project limit needs storage that records each admission attempt
and every project that the attempt charges. Run IDs cannot be the
identity, because a run can be released before dispatch and admitted
again.

**Proposed solution**

Add a separate policy table that keeps "never configured" apart from
"explicit no limit". Add admission epochs with one charge row per
distinct project. Settlement deletes the charge set in one transaction.
Later pull requests integrate claims, settlement, waits and the public
control.

**Alternatives considered**

Columns on `projects` would leak an unenforced cap through every project
reader. A project column or a `uuid[]` on `heartbeat_runs` cannot
enforce per-project uniqueness or company scope. `budget_reservations`
has no project set. A counter would be a second source of truth.

## What Changed

- Added `project_task_execution_policies`. No row means never
configured. A null maximum means an explicit "no limit". A check accepts
only positive maximums. A composite foreign key keeps the project and
company together. The row is deleted with its project.
- Added `execution_admissions`. Each row is one admission attempt of a
run: `(run_id, attempt)` is unique, and a partial unique index allows
one unsettled attempt per run. It keeps the context revision, the
nullable primary project (provenance, no foreign key) and the settlement
identity. A check sets `settlement_id` and `settled_at` together.
- Added `execution_admission_project_charges`. It has one row per
distinct admitted project, unlimited projects included. Composite
foreign keys keep the admission and the project in one company. The
project foreign key has no cascade, so a charge cannot disappear with
its project. `(company_id, project_id)` is indexed for active counts.
- Added migration `0426_absent_guardian.sql` (renumbered from `0423`,
then `0425`, as master added `0423`–`0425`). The generated SQL was
reordered so the unique indexes exist before the composite foreign keys
that need them. Snapshots `0419` and `0420` were pruned to keep five
snapshots.
- Added the `execution-admission` module (domain → application →
adapters):
- Domain: policy patch parsing and application (omitted keeps, null
clears, partial updates keep other fields, malformed values are
rejected), canonical project-set preparation (primary ∪ references,
deduplicated, ascending, bounded with an explicit `too_many_projects`
result), and the settlement replay decision.
- Application: internal `readTaskExecutionPolicy` and
`updateTaskExecutionPolicy` commands. No route, tool, CLI or import
calls them.
- Adapter: the project-set lock (`FOR NO KEY UPDATE`, ascending, one
statement), the policy store, and the storage transitions
`recordAdmission`, `settleAdmission` and `countActiveProjectCharges`.
These transitions are not exported from the module entry point. Claims
(later) must add the capacity check and the lifecycle evidence.
- Project deletion returns `409 project_execution_admission_pending`
while the project has a charge. Agent deletion returns `409
agent_execution_admission_pending` while its runs have an unsettled
admission, and removes settled history. Company deletion calls a new
module participant before it deletes runs and projects.
- Documented the tables, the lifecycle rules and the proposed global
lock order in `doc/DATABASE.md`. Added module ownership to
`server/src/modules/README.md`.

## Verification

- `cd server && npx vitest run src/modules/execution-admission` → 2
files, 26 tests pass (the Greptile finding on an empty primary ID added
an assertion to an existing test). The adapter tests use embedded
PostgreSQL and cover:
  - policy semantics, the maximum check and the company foreign key;
- charges for every distinct project, a primaryless set and an empty set
(no rows);
- replay of one admission ID, conflict on a changed record, refusal of a
second live attempt and the partial unique index;
- settlement of the whole set once, replay, a different settlement, then
attempt 2 for the same run;
- company isolation of projects, runs and admission IDs, the
duplicate-charge key, and rollback of an aborted transaction;
- active counts per project, and an `EXPLAIN` that uses
`execution_admission_project_charges_project_idx`;
- archive keeps charges, `projectService.remove` returns 409, a direct
project delete fails on the foreign key, and delete succeeds after
settlement;
- `deleteAgentDependencies` returns 409 while unsettled, and
`deleteCompany` removes only that company's admissions.
- `cd packages/db && npx vitest run
src/execution-admission-migration.test.ts` → pass. It removes the new
tables and receipt, seeds a legacy project and run, applies pending
migrations, and compares the existing rows.
- `pnpm --filter @paperclipai/db check:migrations` → pass. Five
snapshots remain (`0421`, `0422`, `0424`, `0425`, `0426`; `0423` is
SQL-only on master).
- `node scripts/check-module-boundaries.mjs` → pass.
- `packages/db` `tsc --noEmit` → pass. A scoped `tsc` of the module
entry points and domain tests → pass.
- Existing tests for the changed deletion services → 6 files, 161 tests
pass (`resource-lifecycle-events`, `prompt-injection-prevention`,
`agent-lifecycle/lifecycle`, `case-deletion`,
`project-workspace-root-provenance`, `companies-service`).
- Not run locally: the full `server` typecheck. `tsc` and `tsgo` were
stopped by the out-of-memory killer on the 7 GB machine (exit 137, also
single-threaded). The full `pnpm -r typecheck`, `pnpm test:run` and
`pnpm build` were not run locally; CI ran them on head `520b69d553`: all
gates pass, including Typecheck, Build and the 12 server test shards.
After merging master (`1c9ca5829f`) into head `c088729361`, the
migration check, migration upgrade test, module tests, module-boundary
check and `packages/db` typecheck were rerun locally and pass; CI is
rerunning the full suite.

## Risks

- Migration risk is low. It creates three empty tables and indexes. It
does not change existing tables or rows.
- Behavior changes only when admission rows exist. Nothing writes them
yet, so project, agent and company deletion behave as before.
- Later claim work must take the project-set lock in the documented
order. A claim that takes project locks after issue locks could deadlock
with project deletion.
- Settlement deletes charge rows, so a settled admission keeps its
primary project and context revision but not its full project set.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, embedded PostgreSQL tests). Default context
window and reasoning mode.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: nickyleach <331803+nickyleach@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:35:28 -08:00
Nicky LeachandPaperclip 2dd9811e8b Own agent hiring and termination in a lifecycle module (#15631)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Agent creation, approval, pause, and termination change whether an
agent can work.
> - These changes currently use separate service and database paths.
> - Some agents need setup after their record exists and before their
harness can run.
> - This PR adds one module that owns those state changes and their
transactions.
> - The module records progress and retries incomplete work after a
server restart.

## Linked Issues or Issue Description

**Problem or motivation**

A hire can become active before its execution environment is ready.
Direct agent state writes also bypass one common lifecycle boundary.
Termination does not have a durable step for resource cleanup.

**Proposed solution**

Use agent lifecycle commands for hiring, approval, pause, resume, and
termination. Keep proposed hires pending until approval. Prepare each
approved hire, test its saved harness configuration, and then permit
work. Wait for required plugin work before completing a step.

**Alternatives considered**

A general database commit hook would expose transaction details to
callers. A timer alone would delay each new hire. This module owns each
command transaction and starts its worker after commit. A periodic scan
recovers work after a process stops.

**Roadmap alignment**

This change adds the lifecycle boundary for the existing agent and
plugin systems. It does not add a user interface or a provider
implementation. It replaces the lifecycle delivery approach in #15391.
Related proposals include #11475 and #13099; this change does not add
their external executor protocols.

## What Changed

- Add durable approval, preparation, verification, pause, resume, stop,
and cleanup states.
- Keep agent creation, lifecycle state, configuration writes, and agent
record deletion behind `modules/agent-lifecycle`. Keep validation,
credential rules, reads, onboarding, invitations, general approvals, and
company data deletion in their services.
- Commit a hire before the remaining onboarding or invitation work.
Serialize requests with the existing dedicated database connection
helper. Reuse the hire after a failed attempt.
- Keep accepted work in the run queue during setup. Require execution
and cleanup evidence before completing pause or termination.
- Retain the original credential owner when approving migrated hires.
- Start lifecycle work after each command commits. Use database leases
and revision checks for retries and late results.
- Add a generic required plugin method and capability. Store the
required plugin IDs for each agent. Track host and plugin completion
separately.
- Keep harness verification and credential rules in services. Supply
their results to lifecycle through explicit ports. Reject failed logins
and incomplete probes.
- Remove reverse imports from lifecycle adapters into services. Share
record queries and validation below both layers.
- Use one hire decision path for agent IDs and approval IDs. Resolve the
approval and agent change in one transaction. Expose policy
reconciliation on the command object.
- Save configuration and verification invalidation through one module
operation. Ordinary commands own their transaction. Restrict the shared
transaction integration to its listed callers. Reject identity, company,
accounting, timestamp, and unknown fields.
- Require completed termination before deleting agent records. The
company deletion service calls the module through a separate deletion
interface.
- Save an approved secret before applying its agent binding. A failed
binding leaves the secret approved. A retry reuses it.
- Remove old permission and metadata export files. Move hire
configuration rules out of lifecycle persistence.
- Report failed worker steps and scans with operation context and known
error codes. Omit provider messages, stacks, and configuration values.
Document ordinary application use and extension rules.
- Recover expired native owners only after execution stops. Retry
cleanup of failed retained leases. Wait for setup in the live smoke
script.
- Keep manual, budget, and company pause reasons separate. Guard
execution status writes with the ready state.
- Add a migration, boundary checks, regression tests, and a lifecycle
guide. Keep the current UI.
- Limit periodic policy checks to 100 eligible agents per minute.
Continue from the last agent ID. Exclude pending hires and agents in
termination or a final state. Keep explicit company and budget updates
immediate.

## Verification

- Latest head `ccc3c86b0` aligns the shared status update schema with
the PATCH route. All 15 shared tests, 104 server route/OpenAPI tests, 2
CLI tests, and shared/server/CLI TypeScript checks pass. The Greptile
comment is fixed. CI is running; no new Apex review was requested for
this small fix.

- Rebase `75708ac94` rebases onto master at `2eb5187e4`. The backup test
uses the shared 90-second timeout from master; the older timeout commit
was removed. The affected backup test passes.
- Agent budget edits now pass the target agent ID through enforcement.
All 50 affected tests, server TypeScript, and 11 boundary tests pass.
Both configuration update paths have regression coverage. CI is running
on the latest head. No new Apex review was requested for these small
fixes.

- Rebase `9b568666a` preserves the run-control extraction and
delivery-work startup changes from master. All 559 affected tests,
server TypeScript, and 11 boundary tests pass. CI and Apex review are
running on the final head.
- The policy scan fix passes 56 lifecycle and diagnostic tests, 24
policy and caller tests, and 11 boundary tests. A regression test checks
the page limit, interval, excluded states, and cursor wraparound.

- Rebase `fd4698750` passes all 611 tests in eight affected lifecycle,
route, and heartbeat suites. Server TypeScript and all 11 module
boundary tests pass. The company deletion rollback and concurrent hire
tests are included.

- Follow-up `00aac7646` fixes the failed decision deletion test and the
public update schema. All 126 affected server tests, 8 shared tests, 2
CLI tests, and 11 boundary tests pass. Shared, server, CLI, and UI
TypeScript checks pass.

- Server TypeScript passes: `pnpm --filter @paperclipai/server exec tsc
--noEmit`.
- All 11 boundary tests pass: `node --test
scripts/check-module-boundaries.test.mjs`. The repository boundary scan
also passes.
- All 68 lifecycle, failure diagnostic, and harness verification tests
pass. Tests reject forbidden fields through the root command, combined
transition command, and restricted transaction integration. They also
check invalid fields returned by configuration preparation.
- Failure tests cover host work, plugin work, claims, renewals,
deferrals, and scans. Log tests check known error codes and exclusion of
private error contents.
- All 355 tests in 10 existing caller suites pass. These tests cover
secret approval, pairing, pending hires, credentials, company import,
built-in agents, managed agents, instructions, and skills.
- The branch is rebased onto `origin/master` at `2eb5187e4`. The queue
guards and cancellation lock fix use the extracted heartbeat queue.
Agent cancellation uses the extracted run-control file. The source diff
scan found no secrets or private URLs.
- Earlier checks cover approval retries, rollback, late verification
results, policy holds, deletion rollback, concurrent company deletion,
and queue admission during setup.
- Before the rebase, CI passed on `00aac7646` and Greptile Apex rated it
5/5 with no open comments. CI and a new Apex review are pending for the
rebased commit. The branch has no merge conflicts.
- The full local build and native Dot suite were not run. The host has
no Cargo. No live provider test was run.

## Risks

- Background policy recovery takes more than one minute when more than
100 agents are eligible. Explicit company and budget updates still
reconcile immediately.

- The restricted configuration integration keeps an agent reference and
its dependent records in one transaction. It cannot change a budget or
invoke a lifecycle transition. A periodic scan starts pending setup
after that transaction commits.
- Secret approval and binding use separate transactions. A failed
binding can leave an approved secret with no agent binding.
- New approved hires remain unavailable until setup and verification
complete. Existing clients see `paused` during these steps. This PR does
not change the UI.
- Migration `0320_daily_onslaught.sql` maps existing statuses and pause
reasons. It does not create external resources or claim that an existing
harness passed verification.
- Use a coordinated server update for this migration. Older server
versions can write legacy status without the new lifecycle checks.
- A failed onboarding or invitation request can leave a committed hire.
A retry reuses that hire and completes the remaining records.
- Deletion requires completed termination. Company deletion can require
another request while cleanup is pending.
- Required plugins must make repeated calls safe and reject older
revisions. A disabled or removed required plugin blocks completion until
it is restored.
- A new hire can run a second harness test after the existing UI test.
Some adapters can charge for this test.
- No live provider test was run. Provider implementation and resource
backfill remain separate work.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code tools, and local test
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in this PR with the feature request
field labels
- [x] I have not referenced internal or instance-local issues or links
- [x] My branch name describes the change
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] Targeted local tests pass
- [x] All Paperclip CI gates are green on the new commit
- [x] Greptile Apex is 5/5 with no open comments on the new commit
- [x] I will address review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-09 09:05:12 -07:00
Nicky LeachandPaperclip 3ed5b7c5c8 refactor(server): extract the active-run output watchdog into a feature module (#12853)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server recovery service monitors active runs and applies
watchdog decisions
> - The watchdog rules and database operations lived in one large
recovery service
> - This structure made the rules harder to test and made company
scoping harder to inspect
> - This pull request moves the watchdog into domain, application, and
adapter layers
> - The benefit is a smaller recovery service, pure policy tests, and
clear company-scoped ports

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The active-run output watchdog that detects silence, suppression, and
terminal evidence.

**Current behavior**

The recovery service contains the watchdog policy, use cases, database
operations, and process control in one file.

**Proposed behavior**

A feature module separates pure policy, use cases and ports, and
Postgres and process adapters. The recovery service delegates its public
watchdog methods to this module.

**Reason and benefit**

The separation makes policy decisions easy to test. Company identifiers
on every reader and writer port make tenant scope clear. Smaller service
methods reduce change risk.

**Breaking changes**

None. The recovery service keeps its public methods and call sites.

Related public watchdog work includes
[#7043](https://github.com/paperclipai/paperclip/pull/7043) and
[#7770](https://github.com/paperclipai/paperclip/pull/7770).

## What Changed

- Add the `server/src/modules/active-run-watchdog/` feature module with
domain, application, and adapter layers.
- Move watchdog policy, use cases, Postgres access, and local process
control into the module.
- Keep the recovery service public methods and delegate them to the
module.
- Add 53 pure module test cases and retain 8 Postgres integration cases.
- Add company scoping and transaction rollback coverage.

## Verification

- Run `vitest run --config vitest.config.ts src/modules` and confirm 3
files and 53 cases pass.
- Run `vitest run --config vitest.config.ts
src/__tests__/heartbeat-active-run-output-watchdog.test.ts` and confirm
1 file and 8 cases pass.
- Run the full server suite in pull request CI.
- Compare the type-check result with a fresh baseline on the same
checkout.

## Risks

The main risk is a behavior change in recovery decisions during the move
across layers. The pure policy tests cover the moved rules. The
integration tests cover database behavior, company scope, and
transaction rollback. Pull request CI runs the full server suite.

## Model Used

OpenAI Codex, GPT-5, runtime-managed context window, tool use, code
execution, and repository review support.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-09-05 09:15:59 -07:00