Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-12 14:19:00 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,063 Commits 1,880 Branches 2,440 Tags
a8bbdf237c2dc1e99e5652e0af34e2e146b08a71
Commit Graph
4 Commits
Author SHA1 Message Date
DottaPaperclipCodex
 		839d41aabb fix(server): retire owned plugin listeners before idle sleep (#19358) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Plugins receive durable agent lifecycle events through a registered
host.
> - The host uses a PostgreSQL LISTEN connection to wake delivery.
> - An idle report could return `none` while that host-owned connection
stayed open.
> - This change gives that listener an exclusive owner and closes it
during the current idle hold.
> - Release or expiry restores listening before journal replay.
> - Failed close and pending work continue to block sleep.

## Linked Issues or Issue Description

Refs #18988, which is now merged. This focused ten-file change now
targets master. Current integrated candidate:
`0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`, with actual master
`644ee66f5d4ae8365fe8b9eca57df2be7405b5ab` preserved through normal
merges. No history was rewritten.

**What happened?**

A normal local App with an installed lifecycle plugin could return an
idle `none` report while its lifecycle LISTEN backend and socket stayed
open.

**Expected behavior**

The host must close its own listener before it reports idle safety. It
must retain other subscriptions, durable events, company scope, retry
deadlines and useful reactivation.

**Steps to reproduce**

Start the normal server with a lifecycle plugin. Create an agent and
wait for its durable acknowledgment. Stop the server, create another
agent while it is stopped, and restart with the same files. Request an
owned idle hold after ordinary replay. Check the exact listener backend
and matched Node socket before shutdown. Let the valid hold expire
naturally, then make an ordinary authorized agent update and check its
durable plugin acknowledgment.

Live executing or persistent preview services remain active. The
separate hosted preview dependency
[paperclip-cloud#478](https://github.com/paperclipai/paperclip-cloud/pull/478)
retains its own App and operator prerequisites. This PR does not claim
deployed preview or suspension acceptance.

## What Changed

- Add a private, exclusive database listener owner that inherits the
normal client policy.
- Prepare plugin lifecycle retirement after native retirement under the
same current idle hold.
- Use one delivery-eligibility helper for dispatch and idle journal
inspection. It preserves the same manifest, capability, company-disable
query, and skip rules.
- Close only the owned listener. Keep failed close ownership and
inspection failures visible.
- Resume listening on normal release or expiry, before journal replay.
- Preserve disabled-feature retry history and the original deadline on
re-enable.
- Await lifecycle disposal during normal shutdown and consumer teardown.
- Add owner regressions and document the lifecycle contract.

## Verification

**Current integrated source:**
`0f83005122ef1bdb1f1e08f8f58424c9a840ebb1`. Fresh normal Runner
TypeScript passed in 0.785 seconds. The complete normal App layered
semantic check passed in 28.284 seconds. All tracked source, the
lockfile and the 18 separate private witness inputs stayed unchanged.
Known compiler owners closed and the isolated temporary home was
removed.

**Current product qualification:** at measured integration
`896cc9b62993e6ed10418d7a7417e1ddbf8dc89c`, all three whole files passed
with no failures or skips: lifecycle delivery 40, runtime hooks 20, and
runtime assignments 24. The actual physical LISTEN and natural-expiry
controls passed. All owned clients and captured processes closed.
Independent final session checks were zero before normal deletion of
each successful test database.

**Normal full-App physical retirement:** the original case passed in
47.574 seconds. It exercised real startup, readiness, SDK callbacks,
durable ACK, foreign-company exclusion, physical stop, offline input and
cold replay. The initial unknown hold and its ordinary release remain in
the evidence. After the original 30.5-second cadence, a new valid hold
returned quiescent. The exact current cold-App LISTEN backend and its
matched Node socket closed before server stop.

**Normal full-App natural expiry:** the separate original case passed in
77.391 seconds under its original 90-second body cap. The final valid
30-second hold expired naturally. A new listener backend and matched
current App socket appeared before fresh input. An ordinary authorized
agent-name PATCH produced journal event 20, the actual SDK callback and
durable ACK. Foreign exclusion and readiness assertions passed. The
final measured hold had no manual release, replacement hold or shortened
TTL. The original preparatory unknown hold, ordinary release and
30.5-second cadence remain explicit.

Both App cases used the normal current server helpers and actual SDK
bundling/worker entry. A private compatibility change supplied the same
owned HOME at the exact existing child-fork boundary. It did not change
the product helpers, business assertions or timing limits. These
separate acceptance fixtures remain outside the focused product diff.

**Source carry to the current candidate:** all nine other owned files,
listener mechanics, actual producer paths, SDK paths and measured hold
behavior are unchanged. The sole owned intersection from newer master is
an independent environment-delete cleanup callback in `app.ts`. The
plugin witnesses do not call that route. Normal merges preserve every
unrelated master blob and file mode. The fresh current semantic checks
above qualify the integrated source. New master Native crash-recovery
and saved-copyback Retry capabilities are not claimed by these plugin
cases.

**Actual master merge target:** newer master contains retained plugin
Runner release/resume and other Native recovery fixes. The projected
combined target passed fresh normal Runner TypeScript in 0.799 seconds
and the full App semantic check in 29.715 seconds. All ten plugin-owned
blobs and modes remain exact to the published head; every unrelated
master change is preserved. The measured plugin cases create no Native
provider turn or warm Native session. The unchanged
native-first/plugin-second ordering still checks the native result
before listener retirement. Later sparse-checkpoint and Native
response-publication changes do not enter those paths. This is a scoped
source carry, not new hosted Runner acceptance.

**Failures retained:** an earlier reactivation fixture's real timer
returned one millisecond before its original wall-clock expiry. Its
private fixture now waits any positive remainder and retains the same
expiry assertion and limits. The earlier partial-composition `unknown`
result remains distinct from normal-startup proof. Current qualification
also exposed missing normal workspace dependencies and a private guard
that assumed loose Git refs. Normal owned dependency setup and an exact
loose-or-packed reader fixed those setup failures. No business
assertion, timeout, permission boundary or product sleep gate was
relaxed.

Repeat focused product coverage with a fresh external fixture database
and the normal server Vitest command, one worker:

```sh
pnpm --filter @paperclipai/server exec vitest run src/__tests__/plugin-lifecycle-delivery.test.ts --maxWorkers=1 --no-file-parallelism
pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-hooks.test.ts --maxWorkers=1 --no-file-parallelism
pnpm --filter @paperclipai/server exec vitest run src/__tests__/agent-runtime-assignments.test.ts --maxWorkers=1 --no-file-parallelism
```

Exact current head `0f83005122ef1bdb1f1e08f8f58424c9a840ebb1` has 60
successful checks, three skips and one neutral result, with no pending
or failing contexts. Greptile completed its full review at 5/5 with no
actionable issue. All root and nested review pages were read; the
earlier eligibility finding is resolved. Ponytail's latest turn stopped
before a verdict. A normal retry was requested; no current Ponytail
verdict or score is claimed. Previous `3ca86ec` CI and scores are
historical. A full local repository typecheck/test/build or browser run
is not claimed. Controlled local App/SDK/PostgreSQL proof does not
establish live provider, deployed Railway/Neon suspension, signed Cloud
alarms or global exactly-once delivery.

## Risks

- A listener close failure or uncertain inspection must continue to
block sleep.
- Release, expiry, configuration changes, and journal arrival can race
with retirement. Epoch and current-hold checks retain conservative
results.
- Delivery remains at least once across uncertain ACK boundaries. This
change does not add global exactly-once delivery.
- The database helper closes only its exclusive listener client. Other
owners and subscriptions keep their resources.

## Model Used

OpenAI `gpt-6.1-sol`, reasoning and code/tool use. The managing Codex
agent implemented shared database/composition seams and reviewed the
change; its exact model ID was not exposed. The runtime does not expose
a verified context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
 2026-11-06 22:56:47 -06:00
Nicky LeachandClaude Opus 5.5 11ac90e31f fix(build): typecheck server/src/__tests__ and fix every type error (#18739) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server has about 1,100 test and helper files in
`server/src/__tests__`
> - `server/tsconfig.json` excludes that directory, Vitest does not
check types, and no script runs the tsconfig in that directory
> - So a test can call a function with the wrong arguments and still
pass, and no tool reports it
> - This gap hid 2,168 real type errors, and some of them hid tests that
did not test what they claim
> - This pull request fixes all of these errors and adds the directory
to the server layers from paperclipai/paperclip#18737
> - The benefit is that a type error in a server test now fails the
typecheck, with no baseline and no backlog

## Linked Issues or Issue Description

Refs paperclipai/paperclip#18737, which is merged and added the server
layers. Refs paperclipai/paperclip#18738, which is merged and checks the
test projects outside `pnpm -r`, so it is not a duplicate. Refs
paperclipai/paperclip#18137, which records that this directory has no
check.

**What happened?**

No tool checked the types of 1,074 of the 1,101 files in
`server/src/__tests__`. I searched `package.json`, the workflows, and
all tsconfig files. 27 files are in the program of another tsconfig, and
these 27 have 0 errors. 11 of the 27 are in
`helpers/maintenance/tsconfig.json`. No script ran that project when I
measured this. paperclipai/paperclip#18738 now runs it.

**Expected behavior**

The server typecheck checks these files, and a type error fails the
check.

**Steps to reproduce**

1. On the base commit, run `tsc -p server/src/__tests__/tsconfig.json
--noEmit`. It reports 6,536 errors and needs 12.3 GiB.
2. Add a type error to any file in `server/src/__tests__`.
3. Run `pnpm --filter @paperclipai/server typecheck`. It passes.

**Paperclip version or commit**

I measured the error breakdown below on commit `8dfd632452`, with
TypeScript 7.0.2. The branch now includes master up to
paperclipai/paperclip#18865.

**Deployment mode**

Not applicable. This issue applies to development and CI. No production
source file in the server changes.

### Error breakdown

Each row adds one change to the row above.

| Step | Total errors | Change | Cause |
| --- | --- | --- | --- |
| Config as committed | 6,536 | | |
| Add `src/types/*.d.ts` | 4,447 | -2,089 | Config. The Express request
types were missing. |
| Set `rootDir` to the repository root | 2,436 | -2,011 | Config. |
| Set `allowImportingTsExtensions` | 2,181 | -255 | Config. Vitest
accepts `.ts` imports. |
| Add `ES2024.Promise` to `lib` | 2,138 | -43 | Config. Node 24 has
`Promise.withResolvers`. |
| Set `allowJs` for `.mjs` imports | 2,148 | +10 | Config. It removes 8
`TS7016` errors and shows new real errors. |
| Set `declaration` to false | 2,142 | -6 | Config. Tests emit nothing.
|
| Count only files in `server/src` | 2,058 | -84 | Other packages: 53
Daytona, 29 Runner test, 2 UI. |

So 4,478 of the 6,536 errors (69%) came from config or scope, and 2,058
are real type errors in 314 files. The 29 Runner test errors are also
real, which gives 2,087 errors. Tests that master added after this
measurement had 81 more errors, so this pull request fixes 2,168 errors.

This table classifies the first 2,058 errors.

| Kind of real error | Errors |
| --- | --- |
| Wrong argument or assignment type | 1,023 |
| Stale or partial fixture objects | 525 |
| Property that the type does not have | 317 |
| Missing narrowing of null or unknown | 101 |
| Implicit any | 60 |
| Missing name or module | 16 |
| Other | 16 |

### What the type errors hid

Most errors were stale fixtures and loose mocks. These errors hid real
test problems, and this pull request fixes them:

- **A test that never ran.** `instrumentation.test.ts` required
`sdk-trace-base`, which is not installed. So 3 span tests were skipped.
They now run and pass.
- **Assertions that could not fail.**
- `issue-thread-interactions-service` used a company ID as an issue ID.
- `status-card-evaluation-work` compared `[]` with `[]` in a rollback
check.
- `slack-conversation-lifecycle` read `item.issueId`, which does not
exist.
  - `email-channels` used a permission key that does not exist.
- `heartbeat-project-env` imported `LOW_TRUST_REVIEW_PRESET`, which is
not exported. The value was undefined.
- **Postgres processes that did not stop.** 11 test files called `stop`
on the test database helper. That method does not exist, and the helper
method is `cleanup`.
- **Arguments that the service ignored.**
- `methodKey` is now `connectionMethodKey` in the Daytona, Unikraft,
1Password, and Railway tests.
- 108 calls passed `"owner-user"` to a parameter that expects an options
object.
- 9 chat wizard fixtures passed `managementContext` as `source`, so they
ran `existing_app` and not `manifest`.
  - 2 tests passed a `{ userId }` actor that the function ignores.
- Inserts into `environments` passed a `companyId` column that the table
does not have.
- **Problems in tests that master added during this work.**
- `sandbox-creation-operations` read `database.connectionString` when an
external test database was set. In that mode `database` is undefined, so
the fixture would stop with a TypeError. It now uses the external URL.
- `idle-completed-history` set a Dot operation to `dispatched`. That is
a Muse status, and the Dot type does not have it. The fixture now uses
`admitted`, which is a real in-flight Dot status.

### Possible product problems, not changed

These may be real problems in production code or types. I did not change
them, because this pull request does not change production behavior:

- `decisions-service` `service.create` does not validate the lenient
`cancel_issue_tree` input.
- `createConnection` ignores `input.connectionPurpose`.
- The `createConnection` route schema default stores `{}` for
`transportConfig`.
- `prepareManagedAiRuntime` accepted the invalid value `mode:
"connection"`.
- `createBinding` accepts invalid `projectionClass` strings outside
class 3.
- `ensureManagedProjectWorkspace` forwards an untyped `repoRef` by
spread.
- `routineVariableSchema.label` is optional, but `RoutineVariable.label`
is required.
- `ToolRunContext.projectId` may be too strict.
- `withdrawInteraction` reads `issue.status`, which its type omits.
- `issueService.list` and `reconcileNativeFinalizations` omit fields
that they return at runtime.
- `createGitHubExternalObjectProvider` handles `tokenProvider: null`,
but its type forbids this value.

## What Changed

- Fixed the 2,168 type errors. The diff changes 322 test and helper
files.
- Corrected `server/src/__tests__/tsconfig.json`. Editors and the
typecheck now read the same options and report the same errors.
- Added three `test-tree` layers to
`server/scripts/typecheck-layers.mjs`. They read the server layers as
declarations, and any error fails them. A hash of the path selects the
shard, so a new file does not move other files.
- Added `server/scripts/typecheck-stubs/daytona-sdk.d.ts`. Server tests
reach Daytona plugin source, and the root install does not have
`@daytonaio/sdk`.
- Changed two imports in `ui/src/lib/issue-execution-policy.ts` to use
`.js` extensions, because server tests import this file.
- Made `declarationsConfig` and `--declarations-only` skip the
`test-tree` layers, because these layers emit nothing. Other projects
read only the source layers. Added a test for this.
- Updated `doc/DEVELOPING.md`.

## Verification

Run these commands:

```sh
pnpm --filter @paperclipai/server typecheck
pnpm exec vitest run server/scripts/typecheck-layers.test.mjs
```

Results on macOS (arm64), Node 24, 16 cores. Each check ran after the
merge of paperclipai/paperclip#18738 unless the row says otherwise:

| Check | Commit | Result |
| --- | --- | --- |
| Server typecheck, `PAPERCLIP_TYPECHECK_LOW_MEMORY=1`, all 7 layers |
`6e45a08573` | Exit 0, 0 errors, 62 seconds, peak 3.52 GiB |
| `node scripts/typecheck-test-projects.mjs --low-memory` | `a3306bd13e`
| Exit 0, 0 errors, 42 seconds, peak 3.25 GiB |
| The same script with `--reuse-declarations` | `1b878aabc0` | Exit 0, 0
errors |
| `typecheck-layers.test.mjs` and `live-recovery-work.test.ts` |
`01db5634e4` | 90 passed. These include real compiler runs for a pass, a
type error, and an invalid tsconfig. |
| `tsconfig.declarations.json` after a full server typecheck |
`0fb1e4fb0a` | It references only `below`, `cycle`, and `above`. No
generated config names a `tsconfig.layers.json` file. |
| Owned runtime test in `idle-completed-history` | `e8f2f448ac` |
Passed, with the fix that master later replaced |
| `idle-incoming-wake`, `plugin-idle-manager`, and the changed
`chat-channels` test | `6e45a08573` | 30 passed |
| Runner `native-execution.test.ts` | `eab4dcc1f4` | 114 passed |
| The 19 server test files that changed since `eab4dcc1f4`, and
`typecheck-layers.test.mjs` | `259737b700` | 4,275 of 4,277 passed. The
2 failures are in `chat-channels` and are in the list of 7 below. |

The peak is the largest single process, from `/usr/bin/time -l`. Peaks
for the server typecheck were between 3.42 GiB and 3.56 GiB across 8
runs. Peaks for the test projects were 3.25 GiB and 3.37 GiB across 2
runs. I did not measure a cgroup.

**Runtime check of the fixes.** Types do not prove that a test still
works. So I ran all 309 changed server test files, 11,929 tests, before
the rebase onto paperclipai/paperclip#18737:

| Result | Tests | Detail |
| --- | --- | --- |
| Passed | 11,913 | Includes the 2 tests and the 54 `dot-runner` tests
that passed in a later run, as described below. |
| Failed | 7 | The same 7 tests also fail on the commit before the
fixes. |
| Skipped | 9 | These tests have their own skip conditions: `sentry` 5,
and 1 each in `bundled-plugin-registration`, `customer-success-cloud`,
`onepassword-connection`, and `workspace-runtime-https-live-exercise`. |

The 7 tests that fail with and without this change, on my machine:

- 3 tests in `chat-channels` and `email-channels` stop with `ENOENT` on
a `SKILL.md` file. The path is in the parent checkout, above the git
worktree that I used, so I expect that the worktree is the cause. I did
not prove this.
- `workspace-instance-cleanup` compares `/private/var/...` with
`/var/...`. This is a macOS symlink difference.
- 2 tests in `git-reproducible-managed-checkouts` get
`unknown_base_git_files` and expect `unpushed_managed_checkout`. I did
not find the cause.
- 1 more test in `chat-channels`, `materializes direct external-chat
finals once`, gets `undefined`. I did not find the cause.

CI is the check for these 7 tests. `ci / verify` passed on `eab4dcc1f4`.

Failures during this work:

- My fix to `agent-runtime-connector-access` changed an invalid
permission key in the setup, but not in the assertion. The runtime check
found this. I corrected the assertion, and the file passes.
- `heartbeat-plugin-environment` failed one wait in the full run. It
passed three times alone, so I think that load caused the failure. It
can fail again under load.
- `dot-runner` skipped all 54 tests in the full run because `cargo` was
not on my `PATH`. It then passed 54 of 54 with the toolchain on the
`PATH`.
- My first `lib` value removed the default DOM types, and the error
count went up by 375. I added the DOM libraries.
- My Daytona stub with bare `any` left 48 errors, because generic
helpers then inferred `unknown`. I added shapes that return promises.
- One fix to `agent-secrets-routes` used `class_3_static_lease`, and the
test failed. That class allows only specific paths. I reverted to
`unclassified`.
- `ci / General tests (server (8/12))` failed on `259737b700` because of
the flaky port in `idle-completed-history`. paperclipai/paperclip#18853
fixed this test on master, so this pull request keeps that fix.

**Audit of the diff.** I counted these patterns in added and removed
lines, to find fixes that only hide an error:

| Pattern | Added | Removed |
| --- | --- | --- |
| `@ts-ignore` | 0 | 0 |
| `@ts-expect-error` | 0 | 1 |
| `.skip` | 0 | 0 |
| `as any` | 9 | 9 |
| `: any` | 11 | 2 |
| `as never` | 8 | 2 |
| `as unknown as` | 35 | 4 |
| `Promise<unknown>` | 16 | 5 |
| `expect(` | 128 | 113 |

No hunk removes more `expect(` calls than it adds. Three removals of
test content were intentional, and I checked each one:

- `muse-control` removed `signalToken`. Line 1012 already asserts that
it is absent.
- `merged-pr-confirmation-sweep` removed legacy aliases. Other tests
cover them.
- `heartbeat-workspace-branch-containment` removed a `finalize` branch
that no test row can reach.

Some partial fixtures stay unchecked through `as unknown as` and
`Promise<unknown>`, mostly in `issue-agent-mutation-ownership-routes`
and `issue-comment-reopen-routes`.

**Accuracy of the check.** One program over the tree and the three
shards reported the same errors at the same positions. Memory, as the
peak of the largest process on one thread, was 3.93 GiB for one program
and 2.43 GiB for the largest shard. I measured these two values before
the fixes.

Not run or not proven:

- I did not run the full `pnpm test:run`, or a root `pnpm build`. I
expect CI to run the full suites.
- I did not run the 4 GiB container check, or the full
`scripts/typecheck-low-memory.mjs`. The peaks above come from macOS, so
they are not a Linux cgroup measurement.
- I did not run the UI typecheck after the merges.
- `workspace-runtime-https-live-exercise` is skipped, so its `cleanup`
fix did not run.

## Risks

- **A large diff in test files.** 328 files change. A pull request that
edits the same tests can get merge conflicts. New tests on master can
also add type errors before this merges. Any such error fails this check
and needs a fix here.
- **A higher cost for changes to source types.** A change to a type can
add errors in test files that the author did not touch. The author must
then fix them. This is the purpose of the check, but it is new work.
- **Changed test behavior.** Some tests now run paths that they did not
run before, such as the 9 chat wizard fixtures and the 3 span tests.
They pass here, but they can be more sensitive to timing or to the
environment.
- **The Daytona stub can go stale.** The stub is written by hand. If the
plugin uses a new SDK member, the typecheck reports an error until
someone updates the stub.
- **The release path.** `release-verify.yml` and `scripts/release.sh`
run `pnpm -r typecheck`, so they now run this check.
- **The tsconfig change.** The test transform reads
`server/src/__tests__/tsconfig.json`. The options that I changed are for
type checks, and the runs above passed.
- **A gap that remains.** Tsconfigs in `packages/paperclip-runner` and
in most sandbox provider plugins exclude `*.test.ts`. This check does
not read most of those test files, so they can still drift. That work is
separate from this pull request.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`) through Claude Code, with
reasoning, tool use, code execution, and subagents. The runtime does not
show the context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass (7 tests fail locally, with
and without this change. See Verification.)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-03 15:27:38 -08:00
Nicky LeachandPaperclip 194fc11406 feat(plugins): invalidate runtimes when tool definitions change (#16604) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Lifecycle plugins must refresh runtime preparation when tool
definitions change.
> - Standing assignments alone do not cover catalogs, profile entries,
or policies.
> - New journal hooks must preserve the order used by setup
finalization.
> - This pull request adds tool-definition invalidation and its required
writer guards.
> - It leaves credentials and provider access changes for the next
independent capability.

## Linked Issues or Issue Description

**What happened?**

Catalog, profile-entry, policy, gateway, and connection-configuration
changes do not request agent runtime synchronization. A new
profile-entry trigger also exposes a deletion/finalization deadlock if
the child row is changed before its parent is locked.

**Expected behavior**

Committed tool-definition changes advance runtime revisions without
polling. Bookkeeping does not. Entry changes and profile replacement use
a consistent parent/reference lock order and commit atomically.

**Steps to reproduce**

1. Change a catalog descriptor, profile entry, policy, or connection
configuration.
2. Expect committed runtime events for the affected agents.
3. Run profile-entry deletion concurrently with setup finalization while
both wait for an agent row.
4. Expect both operations to complete without PostgreSQL deadlock 40P01.

**Additional context**

Extracted from #16503. Catalog publication #16602 and standing
assignments #16603 are merged. This PR is rebased onto master and is the
active review. It addresses the original Apex profile-entry deletion
finding. Credential/access hooks #16605 remain a draft without review
requests.

## What Changed

- Add migration 0355 for application/connection configuration, catalog,
profile-entry, policy, and MCP gateway invalidation.
- Preserve transaction coalescing and ignore health bookkeeping and
trust-rule usage counters.
- Keep credential-only fields outside this migration's coverage.
- Preserve merged recipient scoping for bindings and installations.
Profile-entry edits resolve old/new standing profile bindings. Policy
edits resolve old/new agent selectors with their intersection/list
semantics.
- Lock old and new profile-entry references before parent profiles and
child mutations. Recheck the reference set under the parent lock, retry
changed snapshots up to three times, and merge unspecified fields from
the locked profile.
- Make entry add/edit/delete and profile replacement transactional,
including validation and profile reads on the reserved connection.
- Lock policy rows in ID order and change priorities in one statement so
scoped agent locks are ordered across the full set.
- Lock the managed profile before connection removal deletes its entries
in the same transaction.
- Update assignment tests whose old absence assertions are superseded by
expanded configuration coverage.
- Document the supported capability and the remaining credential/access
boundary.

## Verification

- Final head: `809fcb11794bfb12c545af0a2f96ab9bc2b570a2`. It is rebased
onto master with merged #16623. GitHub reports no merge conflicts. [CI
run
39293925136](https://github.com/paperclipai/paperclip/actions/runs/39293925136)
has 55 successful checks and two optional Storybook skips.
- Standard Greptile returned an independent 5/5. The final Apex review
returned 5/5 on this exact head at 17:47 UTC on 2026-10-19. Its summary
explicitly clears this PR for merge independently of draft #16605. The
security scan passed. All review threads are resolved.
- Two real-PostgreSQL negative controls reproduce `40P01` without the
old-reference guard or the concurrent-snapshot check. Both regression
tests pass with the final correction. Server typecheck, migration safety
and numbering, five-snapshot retention, journal history, feature maps,
and diff checks pass. The actual PR diff was scanned for secrets and
personal data before push.
- The final 15-suite local run passed 821/822 tests. The one failure was
in the unchanged runtime-delivery test: a startup callback changed its
global callback count during an independent transaction. The test-only
correction is isolated in #16871. That PR passes all 17 runtime-hook
tests, server types, hosted CI, and standard Greptile 5/5. Its
notification-disabled control fails at the independent event-delivery
assertion. No changes from #16871 are included here; this feature head
passes hosted CI independently.
- Earlier affected-suite runs passed 814 tests after rebase and 820
tests after recipient and whitespace corrections. Four affected suites
passed 514 tests before the second reference-contention regression was
added. Single-connection tests cover entry add, edit, delete, and
replacement. Validation and profile reads use the reserved transaction
connection.
- Review fixes have negative controls: the old code hangs with a
one-connection pool; parent-lock omissions deadlock entry deletion or
app removal against setup finalization; per-row policy reorder deadlocks
under contention; unscoped triggers journal unrelated peers; default SQL
trimming misses JavaScript whitespace. Corrected cases pass. Selector
tests cover all 25 ECMAScript whitespace characters. Recipient tests
cover repeated edits, old/new targets, ignored bindings, rollback,
coalescing, and company scope. The definition suite now has 28 cases.
- Earlier standard scores that depended on #16605 were not accepted as
independent clearance. The first Apex review returned 4/5 for
replacement versus application deletion. The final correction locks old
and new references, checks the reference set again under the parent
lock, and retries before any mutation. The final independent reviews
clear that correction.
- Preserve prior failures for acceptance: draft run 38975432017 had
failures whose causes were not re-investigated. Run 38990400765 exposed
the unrelated durable-resume watchdog race, fixed in merged #16623. In
run 39287657148, two jobs reported cancellation without a failed-test
summary; the cause remains unknown. Both passed on retry without source
changes. Initial local fixtures and SQL bindings also needed correction:
an internal helper name was replaced with public `archiveConnection`,
unchanged priorities were changed to exercise reorder, a driver
timestamp was bound as ISO text, and a superseded company-fanout
assertion was updated. These recovered checks do not establish a cause
for the unexplained CI failures.
- Local checks target the affected services and real database
boundaries. A full local build and repository-wide test run were not
performed. Hosted CI supplies the broader checks. Provider responses in
tests are controlled; live provider behavior was not qualified.

## Risks

Migration 0355 adds statement triggers without changing table schema.
Apply it through the normal migration flow. A tool-definition event can
advance a runtime revision without changing the bounded assignment
snapshot. Consumers still need authorized readers for definitions.
Credential-only changes, grants, secrets, and endpoints remain for the
access-hook migration. Profile replacement now rolls back as a unit if
any entry fails. Binding and install notifications retain scoped
recipients. Profile-entry changes use standing bindings, and policies
use old/new agent selectors. Unscoped policies, company-bound profiles,
and other conservative definition changes retain synchronous
transaction-atomic company-wide fanout: O(eligible company agents) locks
and journal storage, without a fanout ceiling or mutation quota. This
retains the accepted tradeoff from #16603 for company-wide changes.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run local checks; results and the isolated test failure are
recorded above
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-19 10:56:08 -07:00
Nicky LeachandPaperclip 5f5602bba7 feat(plugins): synchronize standing connection assignments (#16603) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Lifecycle plugins prepare agent runtimes outside the control plane.
> - They need committed connection assignment changes without polling.
> - A reader and its revision hooks must cover the same fields.
> - This pull request adds a bounded standing-assignment snapshot and
its matching hooks.
> - Plugins can react to assignment changes while execution keeps its
existing access checks.

## Linked Issues or Issue Description

**What happened?**

The agent runtime reader does not expose standing connection
assignments. Changes to connection installs, active profiles, and
connector feature switches do not request synchronization. Adding
company-wide invalidation also exposes lock inversions in assignment
writers and AI adoption.

**Expected behavior**

The owning lifecycle plugin can read a bounded assignment snapshot.
Every committed change to its stored fields advances the runtime
revision. Rollback removes both the change and its event. Assignment
writers finish without a lock-order cycle.

**Steps to reproduce**

1. Install a connection or bind an active profile to an agent.
2. Read its runtime inputs through the owning lifecycle plugin.
3. Rename or disable the connection, revoke the assignment, or switch
connector features.
4. Expect a committed runtime event and an updated snapshot without
periodic polling.

**Additional context**

Extracted from #16503. Catalog publication #16602 and the separate test
maintenance PRs #16601 and #16610 are merged. This PR now targets master
and is the active review. Tool-definition hooks #16604 and
access/credential hooks #16605 remain drafts without review requests.

## What Changed

- Add SDK types and an owning-plugin runtime snapshot for installed
connections, active profile IDs, and effective connector feature
switches.
- Reject more than 512 connections or 128 active profiles instead of
returning partial data.
- Add migration 0354 for the exact stored fields used by the snapshot,
with transaction-coalesced invalidation. Agent-targeted bindings and
installs notify only their old/new target agents;
project/routine/issue/gateway bindings are outside the snapshot and
produce no notification.
- Use ordered agent locks that are compatible with foreign-key
references and exclude runtime readers. Prelock complete old/new target
sets before replacing assignments or restoring managed profiles.
- Record one company baseline at delivery startup for process-managed
feature overlays.
- Lock assignment inputs before install saves, task completion, and
email assignment. Setup finalization reads the locked connection state
and prelocks company agents before activating, re-enabling, or reviving
a profile.
- Make profile binding and reassignment writes transactional and lock
parent profiles first. Keep target validation and profile/detail reads
on the reserved transaction connection, including with a one-connection
pool.
- Lock company agents in order before validated AI adoption. Keep
adoption and shared configuration writes at `FOR NO KEY UPDATE`,
avoiding an upgrade that can deadlock foreign-key inserts during gateway
creation. Configuration fields cannot change agent identity; lifecycle
transitions and deletion retain their existing locks.
- Document supported coverage and the inputs reserved for later PRs.

## Verification

- Thirteen affected suites passed: 757 tests covering runtime
hooks/assignments/skills, concurrent skill publication, tool
access/catalog, task connection intents, lifecycle delivery, email, AI
accounts, migration histories, snapshot drift, and the agent lifecycle
module. This includes the merged catalog PR's OAuth and
overlapping-refresh regressions.
- The assignment suite passes all 24 cases, including bounded snapshots,
company/agent isolation, rollback, health-field no-ops, process-start
feature overlays, and real lock contention.
- Four one-connection PostgreSQL regressions reproduce the earlier
helper-read hangs before the correction; bind, unbind, delete, and
reassign complete with it. The earlier adoption/gateway probe exposed
both lock upgrades: changing only validation still deadlocks in shared
configuration. Current scoped coverage verifies adoption with
agent-targeted binding writes; gateway-only bindings no longer
invalidate this snapshot.
- The original ordered-adoption regression reproduces PostgreSQL `40P01`
without ordered company-agent locking. Scoped-recipient tests reproduce
unrelated-peer journaling before the trigger correction and verify
repeated bind/unbind operations leave peers untouched, ignored binding
types produce no events, both reassignment recipients are reached, and
coalescing remains intact.
- Six replacement contention cases cover installs, existing setup,
activation, re-enabling, profile revival, and profile reassignment.
Negative controls reproduce `40P01` without the corresponding
target/company prelocks; all cases pass with the guards. Activation
decisions use the connection row reread under lock.
- An initial contention fixture failed because its settings connection
lacked an MCP URL. Supplying the fixture URL and controlled lookup
corrected that test setup error.
- Server typecheck, database migration numbering/safety, five-snapshot
retention, feature-map structural validation, and `git diff --check`
passed. Migration metadata preserves historical journal entries and adds
no schema change. The complete diff was reviewed for secrets and
personal data before push.
- Earlier hosted PR CI runs 38983440861 and 38985530069 passed; the
superseded promotion run 38983440470 was cancelled. Superagent
acknowledged the recipient fix and accepted the documented company-wide
cost at `359b5b2f30a8a0248177367e458663d5379810d1`. Its two standard
Greptile summaries were conditional on a later setup correction and were
not counted as independent acceptance; the setup correction is now
included here.
- At `7da246aab46274db1a120a0546abbb404c5427c7`, hosted PR run
38986958988 passed: all 55 checks succeeded, with two optional Storybook
checks skipped. Superagent passed, all review threads are resolved, and
standard Greptile independently returned 5/5 with no blocking findings
(review completed 2026-10-17 00:40 UTC). Apex independently returned 5/5
with no actionable findings (completed 2026-10-17 00:50 UTC). CI was
rechecked after Apex and remains green; the branch is mergeable against
master. Full local build and repository-wide tests were not run.

## Risks

Migration 0354 adds database functions and triggers without changing
table schema. Apply it through the normal migration flow. Agent-targeted
bindings and installs only notify their affected agents. Explicit
company-wide assignments, connection metadata, profile
activation/removal, and connector feature switches retain synchronous,
transaction-atomic company fanout. Each such transaction takes ordered
row locks and inserts up to one connector runtime event per eligible
company agent: O(number of eligible agents) work and journal storage,
with no company-size cap or cross-transaction mutation quota. This
remains an intentional scope tradeoff for this PR; deferred fanout is
not implemented. Many of these conservative changes can request
preparation even if an individual snapshot stays unchanged. Snapshot
limits fail closed. This PR provides desired assignments, not
credentials or task authorization. Later PRs must add mutation coverage
before consumers can rely on immediate synchronization of tool
definitions or credentials.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository inspection, code
execution, and testing. The exact deployment model ID and context window
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-10-16 17:55:47 -07:00