Commit Graph
67 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 adb2bcf896 fix(connectors): activate plugin connectors with agent grants and no sign-in (#22201)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types (#22004) and serve their tools
with the `plugin` transport (#22019)
> - A plugin connector method can declare `grantKinds: ["agent"]` with
`auth: "none"` or `"api_key"`
> - Such a connector cannot reach its agent: the connect form cannot
name the agent, connect creates no agent grant, and Paperclip Runner
runs never list plugin-transport tools
> - The product rule is that a plugin connector behaves exactly like any
existing connector
> - This pull request fixes the connect form, creates the agent grant at
connect, and lets runner runs list plugin-transport tools
> - The benefit is that an operator can activate a per-agent plugin
connector, and only the chosen agent gets its tools

## Linked Issues or Issue Description

Refs #22004
Refs #22019

**Subsystem affected**
Connectors (connect wizard, tool access service), Paperclip Runner
assigned-tool gateway.

**Problem or motivation**
A plugin connector with agent grants and no sign-in cannot be activated:
1. Connect form: it says "No sign-in needed, available to all agents".
"Change" shows "No identity required" and "Which agent owns this GitHub
account?", with no agent picker. Connect sends `grantKind: "agent"`
without `subjectAgentId`, and the server rejects it with a validation
error. The "Finish setup" resume form has the same problem.
2. Grant: connect with `subjectAgentId` saves a `per_agent` connection
with zero grants. Finish adds the agent install, but there is still no
grant. Only the OAuth callback writes agent grants, and these methods
have no OAuth step. For `api_key` methods connect fails, because the
health check finds no agent grant, and the key has nowhere to go.
3. Runner: a Paperclip Runner run builds its assigned-tool gateway from
an allowlist of transports (`mcp_remote`, `local_stdio`, REST, Browser
Use, GitHub bot, voice). `plugin` is not in it, so the run sees no
plugin-transport tools, even when the install, profile and grant allow
them.

**Proposed solution**
Create the agent grant at connect for curated methods without sign-in.
Add `plugin` to the runner assignment and its digest. Show a one-agent
picker with connector-neutral copy.

**Expected behavior**
Connecting alone grants no tools. Finish with agent A gives only agent A
the tools, in gateway discovery and in a runner run. The connection has
exactly one active agent grant, for A.

## What Changed

- `server/src/services/tool-access.ts`: `connectGalleryApp` writes an
active agent grant for `subjectAgentId` when a curated method with
`auth` `none` or `api_key` uses `grantKind: "agent"`. The API key goes
on that grant, not on the connection. A resume without `subjectAgentId`
keeps the agent of the existing grant and keeps its saved key. A second
agent on the same connection is a 409 conflict. The grant write is
audited and joins the existing revive rollback.
- Finish does not create agent grants. Finish carries no credential, and
it can list many agents while a per-agent connection has one identity. A
credential-less grant would also claim a connected OAuth identity that
does not exist. The grant comes from connect (or the OAuth callback).
Finish creates only the install.
- `server/src/services/heartbeat/run-preparation.ts` and
`native-runtime/runtime-context.ts`: include the `plugin` transport in
the runner assignment and in its digest (the two lists must match).
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: for an agent
grant, show the agent picker (exactly one agent), keep the install
choice on "specific", open the "Change" panel and disable Connect until
one agent is chosen, and send `subjectAgentId`. Copy is neutral ("Which
agent uses this connection?"); the GitHub copy stays for GitHub. The
summary no longer says "available to all agents" for an agent grant.
Finish sends the chosen agent, never `all_agents`, for an agent grant.
- `doc/connections/CONNECTOR-PLAYBOOK.md`: document agent grants for
methods without sign-in.

## Verification

Each new test failed before its fix and passes after it:
- `server/src/__tests__/plugin-connector-agent-grant.test.ts` (embedded
Postgres, real routes and gateway):
- `auth: "none"` + agent grants: after connect, no agent has tools;
after finish with agent A, A's discovery lists the tools and B's lists
none; exactly one active agent grant, for A. (Red: zero grants.)
- `auth: "api_key"` + agent grants: same, the key is on A's grant, and a
tool call receives it. (Red: connect returned 422.)
  - A second agent on the same connection is a 409. (Red: 201.)
- Resume of an API-key draft without the key keeps the key on the grant.
- Paperclip Runner session: `buildPaperclipRuntimeMcpServers` +
`createAssignedMcpTools` list the installed plugin tools for A, and
search by the manifest tool name (`update_record`) finds the gateway
tool. B gets none. (Red: no tools.)
- Regression: an organization-grant method keeps one default
organization grant, and both agents get the tools with `all_agents`.
- `ui/src/pages/apps/AppsConnect.test.tsx`: for `none` and `api_key`
agent-grant plugin methods, the form shows the picker, has no GitHub or
"available to all agents" copy, blocks Connect until an agent is chosen,
and sends `subjectAgentId`. (Red: "available to all agents" shown.)
- Regression suites: `plugin-connectors-plugin-transport`,
`plugin-connectors-circleback-parity`, `heartbeat-runtime-mcp-servers`,
`run-preparation`, `tool-access-service`,
`tool-access-oauth-protocol.integration`, `generic-mcp-connection`
(OAuth, GitHub per-agent and organization flows) pass. UI:
`src/features/connections`, `src/pages/apps`, `App.deep-link-routing`
pass (66 files, 1103 tests).
- `tsc --noEmit` passes for `server` and `ui` source. `pnpm
check:module-boundaries` passes.

## Risks

- Low. The grant change applies only to curated methods with agent
grants and no OAuth. Organization, personal and OAuth flows keep their
code paths.
- The runner change makes plugin-transport tools visible to runner runs.
Access still needs an install, a profile and a grant, and every call is
re-authorized by the gateway.
- Connections created before this fix have no agent grant. Connecting
again (or "Finish setup") with the agent creates it.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:25:25 -06:00
DottaandClaude Opus 5.5 23ce68a8dc feat(connections): end Google access when a Paperclip-managed Google connection is removed (#22065)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Connectors let agents use Google Workspace through a Google sign-in
that Paperclip Cloud manages for the user
> - When a user removes a Paperclip-managed Google connection, Paperclip
deletes the tokens locally, but the grant stays live at Google
> - Google's OAuth review expects that a disconnect ends the app's
access, and users expect it too
> - Google's revoke ends the user's whole grant to the shared client, so
only the Paperclip Cloud broker can decide when it is safe to revoke
> - This pull request asks the broker to revoke each removed grant,
keeps local removal unconditional, and tells the user the outcome in
plain language
> - The benefit is that removal ends Google access when no other
Paperclip connection needs it, and the user knows what to do when it did
not

## Linked Issues or Issue Description

**Subsystem affected**
Connectors: Paperclip-managed Google Workspace and the legacy
per-service Google connections (server removal, shared contracts,
connection UI).

**Problem or motivation**
Removing a Paperclip-managed Google connection or grant was local only.
The refresh token was deleted in Paperclip, but Google still listed
Paperclip as having access to the account. The docs said so ("Removal is
local for managed grants").

**Proposed solution**
On removal, send each removed grant's refresh token to the broker's
signed `POST /v1/connector/revoke`. The broker revokes at Google only
when no other live binding could share the grant. Paperclip records the
broker's outcome code and shows one of three plain messages. Local
removal never waits on, or fails because of, that call.

**Alternatives considered**
Call Google's revoke endpoint from the app. This ends the user's whole
grant to the shared client and can break their other Google connections,
so the broker owns the decision.

**Additional context**
Older brokers answer 409 `REVOCATION_NOT_AVAILABLE`. The app treats
that, a timeout, and a network error as "could not end access at Google"
and still removes the connection.

## What Changed

- `paperclip-cloud-connector.ts`: `revoke()` takes an optional
`timeoutMs`, returns the broker's `outcome` from an allowlist, and keeps
`REVOCATION_NOT_AVAILABLE` as an allowed rejection reason.
- New `tool-access/connections/google-access-removal.ts`. For a managed
Google connection, `prepare()` reads each removed grant's refresh token
and returns a `send()` step. `send()` calls the broker for each distinct
token, one after another, under one 5-second deadline. Every result and
every error (including a Paperclip Cloud client that cannot be created)
becomes an outcome code. It never rejects and never logs a token. An
unexpected error is logged with the error name only and reported as
`unexpected_error`.
- Order for removing a whole connection: read the tokens, run the whole
local teardown (grants, installs, app profile, gateway tokens, runtime
slots, catalog, OAuth state, archive the connection), call the broker,
delete the sent refresh-token secrets whatever the outcome, then delete
the remaining secrets. A failure anywhere in the teardown makes no
broker call. Order for removing one grant: read the token, commit the
revocation, call the broker, delete the grant's refresh-token secret.
`send()` owns the call and the deletion through an injected
secret-removal function.
- Owed revocations: a revoked managed Google grant whose refresh-token
secret still exists (and that nothing live uses) is sent by every later
removal, then the secret is deleted. This covers a process stop after
the teardown and before the call, with no new column. Once the secret is
deleted, nothing is sent again.
- Shared tokens: the shared (organization) sign-in writes the same
secret refs to the organization grant and the connection. So revoking an
organization grant alone neither calls the broker nor deletes the
secret, because the connection still uses that token; removing the
connection does both. The same rule applies to any refresh token still
used by another live grant or connection.
- Audit history: once the broker call settles, a
`connection.google_access_removal` audit event records `reasonCode` =
result and `providerAccess` (plus `grantId` for a single grant). If that
write fails, the removal still succeeds and the failure is logged.
- Revoke signs with the subject that consented to the grant
(`providerTenant.externalId`, saved by the managed callback from the
claim). Grants saved without it fall back to the refresh rule. Refresh
is unchanged; see Risks.
- `removal.ts` and `grants.ts`: `providerAccess: { provider, result,
outcomes }` goes to the responses and to the `tool_connection.archived`
and `tool_connection.grant_revoked` activity entries. The tool access
audit field `providerRevocation` keeps its Vercel meaning only.
- `packages/shared`: `ManagedGoogleAccessRemoval`,
`ManagedGoogleRevocationOutcome`, `ConnectionGrantRevocationResult`.
- UI: one helper, `connectionRemovalToast`, builds the removal toast
(base sentence plus the Google notice, tone and action) for Connectors,
Connections and the app page. No codes are shown.
- API: `docs/api/tool-connections.md` documents both DELETE responses
and `providerAccess`. The OpenAPI registration describes the optional
field and its four results.
- Docs: `GOOGLE-WORKSPACE.md` (order, owed revocations, decision table),
`GMAIL.md`, the Google connector memory note, and the help page on
deleting connection data.

Decision table, judged over the distinct tokens sent in this removal:

| Outcomes | `result` | User message |
| --- | --- | --- |
| Every token `revoked` | `ended` | "Paperclip's access to this Google
account was also ended at Google." |
| Any failure-type code: `provider_revocation_failed`,
`binding_not_known` (the broker had no record, so Google was not
called), `provider_revocation_disabled`, `revocation_in_progress`,
`revocation_not_available`, `broker_unavailable`, `broker_error`,
`credential_unavailable`, `unexpected_error` | `not_ended` | "Paperclip
removed the connection but couldn't end its access at Google. You can
remove Paperclip from your Google account's third-party connections." +
"Open Google account connections" |
| Only `other_connections_remain` | `other_connections_remain` | "Other
Paperclip connections still use this Google account, so Google keeps
Paperclip's access until you remove those too." |
| At least one `revoked`, the rest `other_connections_remain` |
`ended_unconfirmed` | "Paperclip's access was ended at Google. If more
than one Google account was connected here, check your Google account's
third-party connections to be sure." + "Open Google account connections"
|

Grants record no Google account identity: the managed callback saves
`externalId` (the Paperclip subject), `name` (the provider name), and
OAuth timing and scopes, and none of the 14 scopes returns an account
identifier. So the last row is always `ended_unconfirmed`. This includes
two tokens of one user and several agent grants on one Google account.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/google-managed-connection-removal.test.ts
src/services/tool-access
src/services/paperclip-cloud-connector-revoke.test.ts
src/services/paperclip-cloud-connector.test.ts
src/__tests__/tool-connection-removal.test.ts
src/__tests__/tool-access-service.test.ts
src/__tests__/openapi-routes.test.ts` (714 passed)
- `pnpm --filter @paperclipai/ui exec vitest run
src/features/connections/googleAccessRemoval.test.ts src/pages/apps`
(passed)
- Server test typecheck (`node server/scripts/typecheck-layers.mjs`)
passes.
- Red then green: the server tests failed on `master` sources (21), then
each review round's new tests failed on the previous head and pass now.
They cover: the order on both paths, local failure before the grant
commit making no call and the retry sending, cleanup failure after the
call making no second call, a process stop between commit and call being
sent on retry with the secret deleted after, a Paperclip Cloud client
that cannot be created, the consenting subject for an organization
grant, sequential sends under one deadline, each row of the decision
table (per-agent and same-user cases give `ended_unconfirmed`, a mixed
failure gives `not_ended`), the unexpected-error log without a token, no
token in any log call, the delete-stored-data option, and a toast test
per surface.
- Added time, measured locally with a stubbed broker that answers at
once: median 22.1 ms against 20.3 ms without the broker step (12 runs
each), about 2 ms. With a real broker, one round trip per removed token,
capped at 5 seconds in total.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Not verified against a live broker or a real Google account.

## Risks

- A removal of a managed Google connection or grant can take up to 5
seconds longer when the broker is slow.
- Refresh subject is not changed in this PR. In the broker
(`recordProviderGrant`), a ledger row's subject is fixed when the row is
first written, and a refresh under another subject leaves that row
untouched. So moving refresh to a new subject could stop updating the
row's last-seen time. After 200 days the row would then count as not
live, and a later removal elsewhere could revoke at Google while this
connection still works. That risk cannot be ruled out from code for rows
written before this release, so both refresh paths keep their old rules.
The two refresh paths already use different subject orders today: the
gateway prefers `connectorSubjectAgentId` and then
`connectorSubjectUserId`, while background refresh prefers the grant's
own subject. That is pre-existing, and this PR leaves it unchanged.
Revoke uses the consenting subject, which matches the claim-time ledger
row. When the row was first written by a refresh under a different
subject, the broker answers `binding_not_known` and the user gets the
warning.
- Repeated removals can send an already-removed token again; the broker
returns its recorded outcome and changes nothing at Google.
- Against an older broker, users see the "could not end access at
Google" message, which is accurate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:52:20 -06:00
DottaandClaude Opus 5.5 f4cb69205e Let plugin workers serve connector tools (plugin transport) (#22019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types to the Connectors catalog
(#22004), but only for transports the host already runs (remote MCP,
local stdio)
> - A plugin cannot yet serve a connector's tools itself, and legacy
plugin tools (top-level manifest `tools`) still reach every agent
without a connection or install
> - A plugin should be able to ship a full connector: settings,
credentials, tools, grants, policies and audit, like a built-in one
> - This pull request adds the `plugin` transport, the SDK API to serve
it, and moves legacy plugin tools under the same activation rules
without changing today's access
> - The benefit is that any connector can live in a plugin, and agents
get plugin tools only when an operator enables them

## Linked Issues or Issue Description

Refs #22004

**Subsystem affected**
Plugins (SDK, worker protocol, manifest), Connectors (tool access,
catalog discovery), tool gateway, database.

**Problem or motivation**
A plugin connector can only describe an existing transport. There is no
way for a plugin worker to serve a connector's tools with per-connection
settings and credentials. Legacy plugin tools are visible to every agent
as soon as the plugin is enabled, which is the opposite of the connector
activation rules.

**Proposed solution**
Add a `plugin` transport. The gateway keeps every decision (grant,
policy, approval, rate limit, audit) and calls the plugin worker with
the connection, its settings and a per-call credential. Legacy top-level
tools become one implicit connector per plugin. Existing installations
keep exactly their current access through a one-time migration.

**Alternatives considered**
Move legacy tools to connection tool names (`app.<connection>:<tool>`).
Rejected: names change, so existing tool policies and agent instructions
that name `plugin:tool` would stop matching (a possible widening or
loss). Legacy tools keep their names and handlers; only their visibility
is now gated by an install of the implicit connector.

## What Changed

- `packages/db`: migration `0491` widens
`tool_connections_transport_check` with `plugin` (`NOT VALID`, annotated
`online_compatible`), and adds `plugin_connector_legacy_access`, seeded
with one row per installed legacy-tools plugin and company.
- `packages/shared`: method schema gains `settingsSchema` and `tools`
(`{ kind: "static", list: [{ name, displayName?, description?, risk,
parametersSchema }] } | { kind: "dynamic" }`), only for `transport:
"plugin"`; tool names must match `[a-z0-9_]`; plugin-served methods use
`none` or `api_key`. The connect API accepts `settings`.
- `packages/plugins/sdk`: `ToolRunContext.connection` (`id`, `appSlug`,
`methodKey`, `grantId`, `settings`, `credential?`);
`ctx.connectors.register(appSlug, toolName, fn)` and
`ctx.connectors.listTools(appSlug, fn)`; new `listConnectorTools` RPC;
`executeTool` routes connector calls by `appSlug`. A worker refuses to
start when a declared static tool (or dynamic lister) has no handler.
The test harness gains `executeConnectorTool` and `listConnectorTools`.
- `server/src/services/plugin-connector-runtime.ts`: routes one call to
the plugin's worker; a stopped worker is a
`PluginWorkerUnavailableError`.
- Connect flow (`tool-access.ts`): a `plugin` method creates a
`paperclip_plugin` application and a `plugin` connection with `config =
{ sourceTemplateKey, connectionMethodKey, settings }`, validated against
`settingsSchema`. Credentials, grants, credential policy, installs and
profiles use the existing code.
- Catalog: static tools come from the definition, dynamic tools from
`listConnectorTools`, with the existing version hashing and retirement.
Tool risk comes from the declaration.
- Gateway: plugin connections join connected-tool discovery and both
execution paths (direct and approved actions). For each call the gateway
resolves the grant and, for `api_key`, the credential for that call
only. The run context comes only from the session. A stopped worker
returns `503 plugin_connector_unavailable` with `retryable: true`. The
token broker refuses to export plugin connector credentials.
- Legacy tools: a plugin with top-level `tools` and no `apps` gets an
implicit connector (slug `default`, named after the plugin, no
authentication). Its tools keep their names, handlers and today's access
decision, and an agent sees them only when a connection of that
connector is installed for it or for the company. The connector's saved
action rules (Off, Ask first) can only restrict them.
`completeLegacyPluginToolAccess` turns each migration row into such a
connection with a company install, at boot and when a disabled plugin is
enabled again; it is single-flight across servers, resumes a draft left
by an interrupted attempt, and never overrides an operator's later
changes.
- Catalog: tools a dynamic connector stops offering are retired from
discovery; `listConnectorTools` carries the connection's `companyId`,
and uses the acting person's grant for personal connections.
- Board: the setup flow renders `settingsSchema` with the shared JSON
Schema form, sends `settings`, and restores saved settings on resume and
reconnect.
- `create-paperclip-plugin --template connector` now scaffolds a plugin
connector (`apps` + `ctx.connectors.register` + a harness test). SDK
README and `PLUGIN_SPEC.md` describe connectors; top-level `tools` stay
documented as legacy.
- `inferToolRisk` moves to `tool-risk-inference.ts` so the catalog can
reuse it.

Design notes where the spec left room:
- OAuth for `plugin`-transport methods is not supported in this PR
(`none` and `api_key` only); OAuth plugin connectors can use
`mcp_remote` today. Noted as follow-up.
- Board "Test" calls on a plugin connection pass an empty
`agentId`/`runId`, because there is no agent run.
- Companies created after the upgrade get no automatic legacy
connection; they follow the normal activation rules.

## Verification

- `server`: `npx vitest run
src/__tests__/plugin-connectors-plugin-transport.test.ts` (new):
- rules 1-3: a ready plugin and configured accounts give no tools; an
install for agent A gives only A the tools; agent B is denied on
invocation;
- the worker receives `runContext.connection` with the right id, method,
grant, settings and the call's credential; spoofed arguments do not
change it; each account calls with its own key; results do not contain
the key;
- invalid settings are rejected (422); a stopped worker is `503
plugin_connector_unavailable` (retryable); a disabled plugin hides the
tools;
- dynamic tools are listed through the worker and a tool it stops
offering is retired;
- rule 5: a new installation of a legacy-tools plugin gives no tools; an
installation recorded by the migration keeps the same tool names for
every agent, through one company-wide connection (two concurrent
carry-overs make one; an interrupted draft is finished; a retry keeps an
operator's restriction), and is not also exposed as connection tools;
Ask first and Off on the implicit connector restrict the legacy tools;
  - the token broker refuses to export a plugin connection's credential.
- Existing gateway plugin-tool tests (`tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`) now set up the
legacy grant an existing installation has, and pass unchanged otherwise.
- Regression: `plugin-connectors-circleback-parity`, `tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`,
`tool-access-service` (846 passed), plus `connector-tool-policy`,
`plugin-lifecycle-restart`, `remote-mcp-connectors`,
`ai-connection-router` in an earlier run.
- UI: `AppsConnect.test.tsx` covers sending plugin settings from the
settings schema.
- SDK: `tests/worker-rpc-host.test.ts` (start refusal without handlers,
connector routing, tool listing). Scaffold: `entrypoints.test.ts`.
- `packages/db`: `pnpm typecheck` including `check:migrations`; server
layered typecheck (sources and tests); `tsc --noEmit` for shared, SDK,
scaffold and UI.

## Risks

- Behavior change for new installations: legacy plugin tools no longer
reach agents until an operator connects the plugin's connector.
Installations that exist at upgrade time keep their tools through the
migration rows.
- The migration widens a check constraint (`NOT VALID`) and copies
plugin × company rows into a new table. Both are fast; the previous
release never writes `plugin`.
- Calls depend on the plugin worker; when it is down the agent gets a
retryable error.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:09:42 +00:00
DottaandClaude Opus 5.5 b1efe32c60 feat(connections): name and guide connections to another Paperclip instance (#22018)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each Paperclip instance serves an assistant connection (MCP) at
`/mcp/paperclip`, and any instance can connect a remote MCP server by
its address
> - So one instance's agents can file and follow tasks on another
instance, for example a colleague's
> - But the other instance's tools look like the agent's own Paperclip
tools: same product, similar names, and the connection prefix was a
generic internal link key
> - An agent can then file local work on someone else's instance, or the
reverse
> - This pull request recognizes a Paperclip address, names the
connection and tool prefix after that instance, adds agent instructions,
and refuses a connection to the same instance
> - The benefit is that agents can tell their own Paperclip from a
connected one

## Linked Issues or Issue Description

**Subsystem affected**

Connections: "Connect your own MCP server" (generic remote MCP) and the
tool gateway names that agents see.

**Problem or motivation**

When instance A connects instance B through `/mcp/paperclip`, A's agents
get B's `paperclip_create_task` and other tools. The tool prefix came
from an internal key (`app-gallery🔗<uuid>`), so native runners saw
`app_mcp_app_gallery_link_paperclip_create_task_…`. Nothing told the
agent that the tools act on a different instance. An agent could send
internal work to another organization by mistake.

**Proposed solution**

Recognize an address whose path is `/mcp/paperclip`. Use only the typed
address, never data from the remote server. Name the connection after
the host (`acme Paperclip`). Use that name as the tool prefix for these
connections only. Add default agent instructions. Refuse the instance's
own address with `paperclip_self_connection` (422).

**Alternatives considered**

Read the company name from the remote `paperclip_connection` tool. This
needs an extra authenticated call during setup, and it puts
remote-controlled text in names and instructions. The host is enough to
identify the stack. Require approval for each write. We rejected this:
actions start on, the same as other connections.

**Roadmap alignment**

Supports instance-to-instance work through existing MCP connections. It
adds no new transport.

## What Changed

- `packages/shared/src/paperclip-peer.ts`: detect a Paperclip MCP
address, derive a label and name, compare with own origins (loopback
hosts on the same port are equal), and build default agent instructions.
- `server/src/services/tool-access.ts`: default name and default agent
instructions for a new Paperclip connection. Instructions that the
person supplies are kept.
- `server/src/routes/tool-access.ts`: refuse the instance's own address
before any network request.
- `server/src/services/tool-gateway.ts`: use the prefix recorded on a
Paperclip connection (`paperclipPeerToolNamespace`, derived from its
address at creation). Renames never change it. Other connections, and
Paperclip connections made before this change, keep their current tool
names.
- `ui/src/pages/apps/generic-mcp-connect.ts`: same default name in the
setup flow.
- `doc/connections/GENERIC-REMOTE-MCP.md`: new section "Connecting
another Paperclip instance".

## Verification

- `packages/shared`: `npx vitest run src/paperclip-peer.test.ts`, 20
passed.
- `server`: `npx vitest run
src/__tests__/generic-mcp-connection.test.ts`, 98 passed. The naming,
instructions, self-connection and tool-prefix tests fail on the base
revision and pass with this change (red/green).
- `ui`: `npx vitest run src/pages/apps/generic-mcp-connect.test.ts`, 29
passed.
- `tsc --noEmit` passes for `packages/shared` and `server`.
- Live check with two local instances. Instance B was in authenticated
mode on port 3201 with assistant connections on. Instance A was on port
3202.
- A connected to `http://localhost:3201/mcp/paperclip`. The name was
`localhost-3201 Paperclip`. A registered with B by dynamic client
registration and asked for read, write, configure and offline access.
- B's consent page showed "Connect Paperclip localhost-3202 to
Paperclip". After approval, A had all 37 tools active, writes included.
- From A, `paperclip_connection` returned B's company.
`paperclip_create_task` created `ACM-1` on B.
- An agent session on A listed
`mcp.localhost-3201-paperclip-…:paperclip-create-task` (runner name
`app_mcp_localhost_3201_p_paperclip_create_task_…`). Before the gateway
change, the name was `app_mcp_app_gallery_link_paperclip_connection_…`.
The run received the connection instructions.
- Connecting A to `http://localhost:3202/mcp/paperclip` returned 422
`paperclip_self_connection`.

## Risks

- Low risk. Only connections whose address path is exactly
`/mcp/paperclip` change. Other connections keep their names,
instructions and tool names.
- A Paperclip connection created before this change keeps its original
tool names, because only connections that record
`paperclipPeerToolNamespace` at creation get the new prefix. Saved exact
tool-name rules and pending approvals for those connections keep
matching. They also get no default instructions; people can add them.
- Native runners keep only 20 characters of the namespace (`mcp_` uses
4), so long host labels are cut. The tool part stays complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser pane).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:41:13 -06:00
DottaandClaude Opus 5.5 01451fae5b Let plugins add connector types to the Connectors catalog (#22004)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use external services through connectors: a catalog of
connector types, connections (accounts) per type, and per-agent
installs, policies and audit in the tool gateway
> - Today only core can add a connector type. Every type is a JSON
definition compiled into `@paperclipai/shared`
> - Plugins can add agent tools, but those tools skip connections,
grants, per-agent installs and the Connectors page
> - This pull request lets a plugin add connector types with the same
definition schema, and makes them behave exactly like built-in
connectors
> - The benefit is that a connector can ship as a plugin, and an
operator controls it with the same Connectors page and the same access
rules

## Linked Issues or Issue Description

**Subsystem affected**
Plugins, Connectors (tool access), tool gateway

**Problem or motivation**
A plugin cannot add a connector type to the Connectors catalog. Plugin
tools (`tools` in the manifest) do not use connections, credentials per
account, agent installs, policies or the Connectors page.

**Proposed solution**
Add a manifest field `apps` (gated by a new `connectors.provide`
capability). Each entry uses the built-in app-definition schema. The
host namespaces the slug by plugin key and runs setup, OAuth, discovery,
grants, policies and audit through the existing code. This is the first
of two pull requests. The second adds a `plugin` transport so a plugin
worker can serve the tools itself.

**Activation rules (product requirements)**
1. Installing or enabling a plugin grants no tools. It only adds
connector types to the catalog.
2. Connecting an account grants no tools.
3. An agent gets the tools only from an agent install or a company
install of an active connection. Discovery and invocation both enforce
this.
4. A disabled or uninstalled plugin (or an upgrade that removes the app)
hides its connector types and connections from the Connectors APIs and
they reach no agent. Saved setup survives disable and comes back on
re-enable. Uninstall keeps rows archived for audit and revokes
credentials.


**Alternatives considered**
Keep plugin tools (`tools`) as the only plugin path. Rejected: they
bypass connections, per-account credentials, per-agent installs and the
Connectors page.

## What Changed

- `packages/shared`: new capability `connectors.provide`; manifest
`apps` validated with `appDefinitionSchema`. The validator rejects apps
without the capability, duplicate slugs, host-derived fields (`plugin`,
`aiConnectionRouter`, `experimentalFeatureKey`, `availability`), and
methods that need provider code in the host (REST adapters, chat
channels, AI runtimes).
- `packages/shared/src/plugin-apps.ts`: reversible namespaced slugs
(`plugin-<hex plugin key>-<slug>`), the catalog entry builder, and
`reviewedToolRisk`.
- `packages/shared/src/app-definitions.ts`: catalog lookups
(`getConnectableAppDefinition`, `getAppStoreDefinition`, ...) also
resolve the current plugin apps. The set is replaced on each read of
plugin state, so a built-in app is never shadowed.
- App definitions can declare `reviewedTools` (reviewed
read/write/destructive names and the risk for unlisted tools).
Circleback's reviewed risk moves from
`server/src/services/connectors/circleback/risk.ts` into
`circleback.json`, with the same classification. A copy of the
definition now classifies tools the same way.
- `validators/json-schema.ts`: `jsonSchemaSchema` moves to its own
module to break an import cycle between the plugin and app-definition
validators (still re-exported from `validators/plugin.ts`).
- `server/src/services/plugin-app-catalog.ts`: reads ready plugins from
the `plugins` table and answers "is this connection's connector
available". It does not trust in-memory lifecycle events, because
several lifecycle managers and processes can share one database.
- Gateway: connected-tool discovery and `resolveConnectionGrant` drop
connections from an unavailable plugin connector. Built-in connections
do not read the plugins table.
- Connectors APIs: the gallery lists ready plugin apps (with the same
Cloud connector profile filter as core); `listConnections` and
`listApplications` hide unavailable plugin connections; setup entry
points refresh plugin apps first.
- Plugin uninstall archives the plugin's connections through the
standard connection removal (grants revoked, owned credentials deleted,
rows kept).
- UI: the gallery response registers plugin apps for the setup screens;
`/apps/connect?source=plugin-…` loads the gallery first and shows an
unavailable state when the plugin is disabled.
- Docs: `doc/plugins/PLUGIN_SPEC.md` section 11A and the capability
list.

Design choices where the spec left room (picked to match "a plugin
connector behaves like a built-in connector"):
- Only `ready` plugins provide connectors. `error` and `upgrade_pending`
hide them, the same as the AI connection router plugins.
- `rest_api` is rejected for plugin apps in this PR. Core REST
connectors need reviewed adapter code in the host, so a definition alone
cannot work. The `plugin` transport in the next PR covers that case.
- Disabled plugin connections are hidden from list endpoints. Reading
one connection by id still works, so audit links keep resolving.

## Verification

- `packages/shared`: `npx vitest run src/plugin-apps.test.ts
src/validators/ src/app-definitions.test.ts` (416 passed). New tests:
manifest validation, slug namespacing, no shadowing of built-in apps,
reviewed risk.
- `server`: `npx vitest run
src/__tests__/plugin-connectors-circleback-parity.test.ts` (new). A
test-only plugin
(`server/src/__tests__/fixtures/plugin-connectors/circleback-parity/`)
ships an unmodified copy of `circleback.json`. The same journey runs
against built-in Circleback and the plugin copy with a recorded DCR
OAuth + MCP fixture, and must give identical catalog risk/status,
suggested defaults and per-agent tool sets. It covers each rule:
- Rule 1: the gallery adds only the namespaced type; no connections
exist; no agent tool changes.
  - Rule 2: two signed-in accounts, no agent sees a tool.
- Rule 3: an install for agent A gives only A the tools of that
connection; agent B is denied on invocation; each account calls the
provider with its own credential.
- Rule 4: disable hides the type, both connections and applications,
removes tools from discovery and invocation (404) and blocks new setup,
while built-in Circleback is unchanged; installs and grants are
byte-identical; re-enable restores tools with the same names; an upgrade
that removes the app hides it; uninstall archives both connections,
revokes grants and deletes owned secrets.
- Red/green: removing the gateway gate or the declared review makes the
test fail.
- An env-gated live check (`PAPERCLIP_CIRCLEBACK_LIVE=1`) starts the
real Circleback sign-in for both connectors and compares the requests.
It never runs in CI. I did not run it for this PR.
- Regression: `ai-connection-router`, `connector-tool-policy`,
`mcp-apps`, `plugin-lifecycle-restart`, `tool-access-service`,
`braintrust-connection`, `remote-mcp-connectors`,
`connection-intents-service`, `tool-gateway` (796 passed).
- `ui`: `app-connect-policy.test.ts`, `AppsConnect.test.tsx` (new plugin
cases) pass.
- `tsc --noEmit` for shared, server, ui (only pre-existing ungenerated
Storybook catalog errors), plugin SDK and CLI.
- `pnpm verify --explain` selects a broad plan because a doc under
`doc/plugins` changed; CI supplies that coverage.

## Risks

- No migration. Built-in connectors only gain one string check per
discovery; the plugins table is read only when a plugin connection is
present.
- Circleback's reviewed risk is now data. The behavior is the same, and
the existing policy tests and new tests cover it.
- Connection intents (agent connection requests in chat) still search
only built-in apps. Plugin connectors can be set up from the Connectors
page. This is a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:51:39 +00:00
DottaandClaude Opus 5.5 c652c813bc feat(google-workspace): per-service transport switch for REST-backed tools (off by default) (#22016)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use Google Workspace through one built-in `google-workspace`
connection and the tool gateway
> - Today every Google tool call goes to Google's hosted Workspace MCP
servers, which are a Developer Preview and cannot be offered to the
public
> - #22000 added Paperclip's own Gmail tools over Google's REST APIs in
`server/src/google-workspace-rest-tools/`, but nothing calls them yet
> - This pull request adds one per-service transport switch and connects
the gateway to the REST tools behind it. Every service stays on Google's
MCP servers, so production behavior does not change
> - The benefit is that a service moves to the REST tools with a
one-line, reviewed change once all its tools exist, and moves back the
same way

## Linked Issues or Issue Description

Refs #22000

**Subsystem affected**
Tool gateway and catalog discovery for the built-in Google Workspace
connection.

**Problem or motivation**
Google's Workspace MCP servers are a Developer Preview. Paperclip
launches on its own tools over Google's REST APIs. The REST tools exist
(#22000), but the gateway cannot call them, and there is no way to
choose them per service.

**Proposed solution**
Add a per-service transport setting (`google_mcp` or `paperclip_rest`)
in the shared Google Workspace module. Add one server adapter that is
the only importer of the REST folder. When a service is set to
`paperclip_rest`, the gateway call path and discovery use the adapter in
process. All gateway controls stay the same.

**Alternatives considered**
A loopback or hosted MCP endpoint for the REST tools. It adds a
connection, JSON-RPC framing and the protocol probe to every call, and
authenticated public deployments refuse loopback addresses. An
environment variable or instance setting. A plain constant keeps every
flip a reviewed code change.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: adds
`GOOGLE_WORKSPACE_TOOL_TRANSPORT` (every service is `google_mcp`),
`transport` on the result of `googleWorkspaceToolTarget`, and
`googleWorkspaceServiceToolNames`.
- New `server/src/services/google-workspace-rest-dispatch.ts`: the only
importer of the REST folder. It keeps only the bearer token from the
gateway's credential headers, passes the gateway's egress-guarded fetch
and cancellation signal, and returns either the MCP result or a token
problem. `googleWorkspaceRestTransportGaps()` lists allowlisted tools
that a `paperclip_rest` service does not implement. Today Calendar is
eligible (9 of 9) and Gmail is not (3 of 9); nothing is flipped.
- `server/src/services/tool-gateway.ts` (`executeRemoteHttpTool`): for a
`paperclip_rest` tool, the gateway rechecks the current grant, calls the
adapter, and returns through the same result handling as the MCP path.
Google's insufficient-scope answer maps to `oauth_insufficient_scope`. A
rejected token forces a refresh and returns
`oauth_refreshed_retry_required` without a replay. A passed deadline
returns `tool_timeout`. The audit record names the protocol `Google
Workspace REST`. The grant recheck is now one shared helper for both
paths. A timed-out Google Workspace write, on either transport, now
returns `tool_timeout` with a warning that the change may or may not
have been made; a timed-out read keeps the plain timeout message.
- `server/src/services/tool-access/catalog/discovery.ts`: for a
`paperclip_rest` service, discovery and health run one REST probe with
the caller's token, then list the REST tools under the same names and
scope filter. A 401 refreshes and probes again. A second 401 on a
managed grant marks it `needs_reauthorization`. Any other probe failure
makes only that service unavailable.
- `server/src/google-workspace-rest-tools/`: adds `probe.ts` and its
export. Each service has an ordered list of cheap reads with the scopes
that allow each (Gmail: `users.getProfile`; Calendar: one calendar list
entry, one primary-calendar event, or one minute of free/busy). The
probe uses the first read the grant allows. A grant that allows none is
listed without a probe, and an insufficient-scope answer never drops a
service. It also and lets `boundary.test.ts` allow exactly the adapter
as the importer. The README now describes the implemented seam, how to
flip a service, and the removal procedure.
- `doc/connections/GOOGLE-WORKSPACE.md`: describes the transport
setting.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

## Verification

- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts`: 15 tests. They
were red before the gateway and discovery changes (10 failed), and green
after. With Gmail set to `paperclip_rest` through a test-only change of
the map, the tests use the real managed setup, discovery, gateway,
policy and screening with a fake Google fetch. They prove: discovery
lists the same tool names filtered by scopes after one probe; a read
runs with the caller's token and is audited; a draft waits for approval
and then runs; a tool outside the grant and a narrowed grant are refused
before any Google request; Google's insufficient scope maps to
`oauth_insufficient_scope`; a 401 refreshes the token, is not replayed,
and the retry uses the new token; an injected email is blocked; a passed
deadline aborts the Google request and returns `tool_timeout`; the
experimental-feature gate stops calls and discovery; a failed Gmail
probe leaves Docs usable and Gmail comes back; a lasting 401 asks for
reconnection. A timed-out draft carries the uncertain-write warning in
the error, the invocation record and the audit entry on both transports,
and an approved REST draft whose request was lost returns the warning.
With the default map, a call goes to `gmailmcp.googleapis.com` and the
adapter is never called.
- `cd server && npx vitest run
src/services/google-workspace-rest-dispatch.test.ts
src/google-workspace-rest-tools`: eligibility (no partial flip),
bearer-only forwarding, signal and fetch pass-through, token outcome
mapping, probe, boundary.
- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts`: every service defaults to
`google_mcp`.
- Existing Google tests are unchanged and pass:
`google-workspace-injection-screening.test.ts`, the Google cases in
`tool-gateway-service.test.ts` and `tool-access-service.test.ts` (142
Workspace/Google/Gmail cases).
- `npx tsc --noEmit` for `packages/shared` and `server`: clean.
- `pnpm verify`: exit 2 (a doc under `doc/` is a global CI input), so
required CI supplies broad coverage.
- Speed, stubbed fetch, 300 calls after warm-up: the adapter adds about
15 µs per call. A full `gmail__get_message` through `executeTool` took a
median of 18.2 ms on REST and 18.3 ms on MCP (p90 21.6 ms and 22.3 ms).
The REST path adds no waits, no extra round trips and no dynamic
imports.

## Risks

- Low risk for production: every service stays on `google_mcp`, and the
default-map test proves the adapter is not called.
- The server now loads the REST folder at start. It uses only modules
the server already loads (MCP SDK, zod).
- A later flip must add a probe and the API host for each new REST
service. The `Record` type of the probe table makes typecheck fail when
a REST service has no probe.
- On `paperclip_rest`, MCP resources, MCP Tasks and elicitation do not
exist for that service.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:49:49 -06:00
DottaandClaude Opus 5.5 d1b5cfe038 feat(connections): let users delete data read through a connection (#22010)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page

## Linked Issues or Issue Description

**Subsystem affected**

Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.

**Problem or motivation**

Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:

- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.

**Proposed solution**

A durable, resumable, company-scoped deletion request per connection.
See "Design" below.

**Alternatives considered**

- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.

## Design

**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.

**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.

| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |

**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.

**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.

**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.

**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.

**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.

**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.

**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."

**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.

**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.

## What Changed

- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.

`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.

## Verification

- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.

## Risks

- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:55:35 -06:00
DottaandClaude Opus 5.5 66ac5d7aa0 fix(google): keep Paperclip's Google client inside the 14 submitted scopes (#22001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector uses Paperclip's own Google
OAuth client through the Paperclip Cloud broker, or a customer-owned
client
> - Google is reviewing that client for exactly 14 scopes (Gmail
read/compose, Drive read/file, Docs, Sheets, Slides, Calendar)
> - The code still accepted the old 21-scope union for the unified
connection, and other Paperclip-managed Google methods (Gmail archive,
Chat, People, Workspace Search) could ask the same client for scopes
outside the submission
> - A consent request for an undeclared scope through the verified
client breaks the review
> - This pull request makes the 14 an explicit constant, enforces it on
every unified-connection request, callback and refresh, and stops new
consent for managed Google profiles outside it
> - The benefit is that Paperclip's Google client cannot request a scope
that Google has not reviewed, while saved connections keep working

## Linked Issues or Issue Description

Refs the plan in `doc/plans/2026-11-17-google-workspace-code-fixes.md`
(items 2, 3 and part of 10).

**What happened?**
The unified `workspace.all` profile was the 21-scope legacy union, and
`isGoogleWorkspaceScopeGrant` accepted any subset of it. The Gmail
`paperclip-archive` method asked for `gmail.modify`. The Chat, People
and Workspace Search cards had Paperclip-managed methods with Chat,
Contacts, Directory, profile and `calendar.readonly` scopes. None of
these are in the Google submission.

**Expected behavior**
Paperclip's own Google client requests, stores and refreshes only the 14
submitted scopes. New managed setup is not offered for anything else.

**Steps to reproduce**
1. Advertise `gmail.archive` (or `chat.read`) from the broker.
2. Open Apps. The managed archive/Chat method is offered, and starting
it signs a broker session for scopes outside the 14.
3. Refresh a unified grant: the signed binding lists all 21 legacy
scopes.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: add
`GOOGLE_WORKSPACE_SUBMITTED_SCOPES` (the 14, written out by hand).
`workspace.all` now uses it. `isGoogleWorkspaceScopeGrant` checks
against it. Add `googleWorkspaceProfileWithinSubmittedScopes`.
- `server/src/services/paperclip-cloud-connector.ts`: add
`paperclipCloudConnectorConsentAllowed`. `startAuthorization` refuses a
Google profile outside the 14 before any broker call.
`appWithPaperclipCloudConnectorAvailability` does not offer those
methods for new setup, even when the broker advertises them.
- `authorization-start.ts`: a saved connection on such a profile cannot
restart consent (`paperclip_cloud_connector_scope_not_offered`, 422).
- Capping contract, the same on managed and customer-owned OAuth
(`paperclip-cloud-connector.ts` `openCredentials`, `cloud-callback.ts`,
`direct-callback.ts`, `grant-refresh.ts`): Paperclip caps Google's
reported scopes to the 14 (`googleWorkspaceSubmittedScopesIn`) and to
the saved request, then continues. A scope outside the 14 is ignored and
never enables an action. Only missing scope evidence or an empty result
marks the grant for reconnecting. Reconnecting with a new service choice
always works. What Paperclip *requests* stays fail-closed: an outgoing
unified authorization request with any scope outside the 14 is rejected.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared file.

### Investigation

**Scope flow, unified connection.** Both paths start in
`authorization-start.ts`. The requested set comes from the picker
(`googleWorkspaceSelectedScopes`), a saved draft, or the saved grant,
and must be inside the method's `scopesHint` (the 14 since the connector
was added in #15545). Managed: `cloudConnector.startAuthorization` signs
`scp` = the request; the broker validates it as a subset of its
registry; the claim is unsealed with the request as AAD;
`cloud-callback.ts` caps Google's grant to the request. Customer-owned:
`direct-callback.ts` checks Google's raw `scope`, then caps it. Refresh:
managed refresh signs `scp` = the profile definition (previously 21) and
unseals with that set as AAD; customer refresh checks the raw `scope`.

**What depends on the 21-scope set.** Only the managed refresh binding
(`scp` and envelope AAD) and tests. The broker (`paperclip-cloud`
origin/master, `src/connections/providers.ts`, `broker.ts`) accepts any
nonempty, duplicate-free subset of its `workspace.all` union for
session, claim and refresh, and seals with the signed `scp`. So
narrowing the app's `workspace.all` to 14 keeps the contract. I narrowed
the definition itself instead of adding a parallel check. A saved
unified grant can still report one of the 7 extra scopes: older app
versions signed the 21-scope `workspace.all` binding on refresh, and
Google can return scopes the same client was granted before. This PR
therefore caps those scopes instead of failing the grant.

**Managed Google methods.** See the table below. The broker's
origin/master registry has no `gmail.archive` profile, so
`paperclip-archive` could not reach the managed client today.

| App / method | Profile | Scopes | Before | After |
| --- | --- | --- | --- | --- |
| google-workspace / paperclip-workspace | workspace.all | subset of the
14 | 21 accepted | 14 only |
| gmail / paperclip-read, paperclip-draft | gmail.read, gmail.draft |
gmail.readonly, gmail.compose | offered | offered |
| drive, docs, sheets, slides, calendar / paperclip-read,
paperclip-write | per service | inside the 14 | offered | offered |
| gmail / paperclip-archive | gmail.archive | + gmail.modify | offered
if advertised | saved only |
| google-chat / paperclip-read, paperclip-write | chat.read, chat.write
| chat.* | offered if advertised | saved only |
| google-people / paperclip-read | people.read | directory, contacts,
userinfo.profile | offered if advertised | saved only |
| google-workspace-search / paperclip-read | workspace-search.read | +
calendar.readonly, chat.messages.readonly | offered if advertised |
saved only |

"Saved only": the method stays in the app definition so saved
connections resolve their method, refresh and run. New setup and new
consent are refused. Customer-owned methods are unchanged.

### Broker changes needed in paperclip-cloud (not in this PR)

1. Narrow the `workspace.all` request allowlist for `session` (and the
provider URL) to the 14. On refresh and claim, cap Google's reported
scopes to the allowlist (or ignore extras) instead of rejecting with
`provider_rejected`, and keep accepting a signed `scp` binding of the
old 21 from older app versions until they are retired.
2. Drop or disable `chat.read`, `chat.write`, `people.read`,
`workspace-search.read` for new sessions on the verified client. Keep
refresh for saved grants until the owner retires them.
3. Do not merge the `gmail-archive-oauth-profile` branch for the
Paperclip-managed client.

## Verification

- Red first, then green: `pnpm exec vitest run
packages/shared/src/google-workspace-connectors.test.ts
server/src/services/paperclip-cloud-connector.test.ts` (13 new or
changed tests failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "refuses new managed
setup|saved Paperclip-managed Gmail archive|fails closed on invalid
refreshed|pilot"` (8 failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "caps a|keeps a .*
grant on its submitted|reconnects a"` (4 capping tests failed on the
earlier fail-closed revision; both reconnect tests pass).
- `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts
-t "combined Workspace|Workspace|Gmail|managed"`: 161 passed. Full
`tool-access-service` and `connection-management.integration` files: 599
passed on the earlier revision.
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts
server/src/services/gmail-tool-governance.test.ts
server/src/services/gmail-archive-tool-policy.test.ts`: pass.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`: pass.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Rollout order is app first, then broker. Refresh now signs the 14. If
the broker narrows `workspace.all` first, refresh from app versions that
still sign 21 (including self-hosted instances on the broker) fails
until they upgrade.
- A saved unified grant whose provider-reported scopes include a scope
outside the 14 keeps working on its in-set scopes; the extra scopes no
longer enable any action. A grant with only out-of-set scopes needs
reconnecting, and a reconnect with a new service choice succeeds.
- When the broker narrows `workspace.all` to the 14, it must cap or
ignore extra scopes on refresh and claim (it re-validates them today),
not reject saved grants.
- Saved Chat, People, Workspace Search and Gmail archive managed
connections keep working, but their owners cannot reconnect them through
Paperclip's Google client. They see a 422 with a plain message. The
owner must decide whether to retire them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:41:31 -06:00
DottaandClaude Opus 5.5 84cb28f9b7 fix(connections): keep machine sign-in secrets and settings in one locked commit (#21566)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21473 added machine sign-in (MCP OAuth client credentials) for
organization connections
> - Auto-merge squashed #21473 from an earlier head, so its last review
fixes did not reach master
> - Those fixes close a P1 race and two smaller setup gaps that Greptile
found on #21473
> - This pull request applies that last commit to current master, with
its tests
> - The benefit is that a client ID and its secret always stay together,
and key switching and app selection work as intended

## Linked Issues or Issue Description

Refs #21473

**What existing behavior does this improve?**
Machine sign-in setup and rotation, added in #21473.

**Problem**
- A connection test wrote a replacement client secret before checking
whether the connection changed. If the save was then refused, the secret
stayed and could pair with another client ID, so the next token request
failed.
- Switching a client-secret connection to a Paperclip key always created
an RS256 key, even when the sign-in service accepts only ES256.
- Machine sign-in dropped the setup's app ID and created a separate app.

**Proposed improvement**
Write the secret and the settings in one locked transaction after the
change check. Record the accepted signing algorithms and use them. Pass
the app ID through.

## What Changed

- `client-credentials-service.ts`: `saveConnection` locks the row,
checks that the sign-in method and credential references are unchanged
and that the connection is not removed, then writes any new credential
and the settings in the same transaction. Test, secret rotation and key
staging all use it.
- Setup records `tokenEndpointAuthSigningAlgValuesSupported`. Switching
from a secret to a key uses `chooseAssertionAlgorithm` on it.
- UI: `MachineSignInSetup` takes `applicationId`, and the custom-server
flow passes `prefill.applicationId`.
- Tests: two new server tests and one UI expectation.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 25/25
pass. The 2 new tests fail on master's service code ("never leaves a
refused test's secret behind with another client ID", "creates the kind
of key the sign-in service accepts when switching from a secret").
- `MachineSignInSetup.test.tsx` and
`GenericMcpConnectStep.machine.test.tsx`: 8/8. The app ID expectation
fails on master's component.
- Server `tsc --noEmit` clean. UI `tsc` clean except the pre-existing
generated Storybook catalog imports.

## Risks

Low. The change is limited to machine sign-in. A concurrent change now
returns 409 `client_credentials_changed` instead of saving partly. No
migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 05:30:14 +00:00
DottaandClaude Opus 5.5 67652580ee perf(tool-access): index enabled policies so each decision checks only candidates (#21563)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool access policies decide whether an agent can call each tool, and
an MCP tools/list response decides access once for each listed tool
> - Each decision matched the raw selector of every enabled policy two
times, so one listing cost tools x policies selector checks on the
server's main thread
> - On staging, a company with 363 enabled policies kept the main thread
at 95-98% of a core: policy matching was 22% of CPU time, with single 80
ms runs inside the selector match
> - This pull request compiles the enabled policies once for each
decision cache and indexes them by the selector key that rejects most
(catalog entry, then tool name)
> - The benefit is that each decision checks only the policies that can
match its tool, with the same decisions, so a listing no longer blocks
the event loop

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Tool access decisions for MCP tool listings
(`toolAccessPolicyService.decide`, called by `decideToolsForListing` in
the tool gateway).

**Subsystem affected**
Server: tool access policy evaluation
(`server/src/services/tool-access-policy.ts`).

**Current behavior**
Each decision loops over all enabled policies two times. The first loop
matches the selector and schema-checks the conditions of every
conditioned policy. The second loop evaluates conditions and matches the
selector of every policy. The selector match reads and allocates every
selector key in a fixed order, so a catalog entry mismatch is found only
at the end. A listing of 800 tools with 365 policies does 305,600
selector evaluations. On staging this was 22% of main-thread time
(event-loop utilization 0.965, delay p99 144 ms, max 710 ms).

**Proposed behavior**
The decision cache holds the enabled policies compiled once: parsed
selectors, an index by catalog entry and tool name, and a memoized
conditions schema check. Each decision evaluates only the candidate
policies, in the original priority and creation order.

**Reason and benefit**
Listing cost becomes proportional to the policies that can match each
tool. In the benchmark, one listing goes from about 164 ms to 12 ms of
main-thread time, and the longest event-loop block goes from 84 ms to 4
ms.

**Breaking changes**
None. The policy data model, API, and listing contract do not change.

## What Changed

- Add `server/src/services/tool-access/policy-selector-index.ts`. It
compiles a policy selector into a matcher with the same semantics as the
raw `selectorMatches`, and it indexes policies by `catalogEntryId`, then
by `toolName` (matched against the tool name and the upstream tool name)
for policies without a catalog entry selector. Policies with neither key
are candidates for every call. Candidates keep their original order.
- `decideInternal` caches the compiled index under the same
`enabled-policies:${companyId}` key. The new `matchEnabledToolPolicies`
keeps the two-step semantics: the first selected policy with unsupported
conditions denies the call before any match applies, then the selected
policies with matching conditions go to the unchanged decision loop
(block, rate limit, trust rule, require approval, allow, explanations).
- Decisions without a cache, and decisions that consume a rate limit
(which read fresh rows), compile the rows they read in the same way.
- The conditions schema check runs once per policy and cache, not once
per decision.
- Conditions are now evaluated only for policies whose selector matches.
Before, a malformed legacy condition on an unrelated policy could throw
in every decision. A selected policy with such conditions still gets the
unsupported-conditions deny first.
- Add `server/scripts/benchmark-tool-policy-matching.ts` and a section
in `doc/mcp-discovery-performance.md`.

## Verification

- Red/green regression:
`server/src/__tests__/tool-access-policy-index.test.ts` "reads policy
selectors once per decision batch, however many tools it decides". It
decides 40 tools with one cache against a staging-shaped set of 365
policies and counts reads of the raw policy selectors. On
`origin/master` it fails: 308,110 reads for 40 tools against 7,711 for
the first tool. With this change the count stays at the first-tool
count. The test also checks that cached and uncached decisions are
equal.
- Equivalence: the same file compares the compiled selector with the raw
`selectorMatches` for 404 selectors x 300 contexts (including empty,
whitespace, non-string, string-plural, duplicate and contradictory
values). It compares the indexed policy selection with the old unindexed
two-step selection (copied into the test) for 5 random sets of 400
policies x 400 contexts, and for the staging-shaped set over 800 tools.
The sets include catalog-entry-only, catalog-entry+connection,
multi-key, tool-name, unindexed, conditioned, unsupported-condition,
block, rate_limit, trust_rule, allow and legacy policy types with mixed
priorities.
- Index edge cases: an earlier tool-name policy (matched through the
upstream name), an unindexed policy and an empty catalog-entry selector
apply before a catalog-entry block; an unsupported-condition policy in
another bucket still denies. A DB-backed test checks the same through
`decide`, with and without a cache.
- Focused suites pass locally: `tool-access-policy-index`,
`tool-access-policy-service`, `tool-gateway-listing-memory`,
`connector-tool-policy`, `tool-gateway`, `tool-gateway-service` (6
files, 406 tests). `node server/scripts/typecheck-layers.mjs` passes.
- `pnpm verify` exits 2 (the new benchmark script is a broad input), so
CI supplies the broad coverage.
- Benchmark, local embedded PostgreSQL, macOS arm64, Node 24. One
listing of 800 tools, 365 enabled policies (staging shape plus the 2
fixture policies), 16 concurrent decisions with one cache as in the
gateway. Three runs of 9 alternating rounds after 2 warm-up rounds. Run
3 uses the event-loop delay histogram that also covers the end of the
listing. Both versions return identical decisions.

| Per listing | `origin/master` median (min-max) | This PR median
(min-max) |
  | --- | --- | --- |
  | Elapsed, run 1 | 157.5 ms (146.1-192.8) | 11.4 ms (10.3-15.3) |
  | Elapsed, run 2 | 165.3 ms (161.7-185.4) | 11.9 ms (10.1-14.0) |
  | Process CPU, run 1 | 160.5 ms (149.7-188.9) | 13.7 ms (11.1-21.0) |
  | Process CPU, run 2 | 162.8 ms (157.2-192.7) | 12.3 ms (9.6-20.5) |
  | Elapsed, run 3 | 163.9 ms (151.6-185.3) | 12.4 ms (10.4-18.5) |
  | Process CPU, run 3 | 165.4 ms (155.7-214.9) | 14.5 ms (10.6-20.2) |
| Longest event-loop block, run 3 | 84.1 ms (76.9-95.1) | 3.7 ms
(3.0-5.0) |
  | Selector evaluations | 305,600 | 1,164 |
  | Raw selector property reads | 6,155,830 | 9,490 (compile only) |

  ```sh
cd server && pnpm exec tsx scripts/benchmark-tool-policy-matching.ts
--baseline-ref origin/master --tools 800 --rounds 9
  ```

- Not verified: the hosted staging profile after deploy.

## Risks

- Low risk. The decision loop is unchanged; only the set of policies
that reach it is narrowed, and every left-out policy has a selector that
rejects the call. The equivalence tests compare against the old
selection.
- The index lives only in the per-listing decision cache (or for one
uncached decision). There is no cross-request cache, so policy edits
apply on the next listing as before.
- The cached index memoizes the conditions schema check on shared
entries. The check is pure, so concurrent decisions in one listing get
the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 22:51:43 -06:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
Devin FoleyandPaperclip dc018f9bc0 Keep agent-owned connections healthy during catalog refresh (#21539)
Use dedicated agent grants for catalog refresh, GitHub health checks, and reconnect status. Preserve shared connection health when a different agent lacks authorization.

Add PostgreSQL and HTTP regressions for credential selection, stale health recovery, caller isolation, and actual provider rejection. Correct the MCP demo approval fixture to submit a valid saved-card option through normal answer validation.

Validation: 629 focused tests, server typecheck, all CI checks, and Greptile 5/5 with no unresolved review comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 18:45:20 -08:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
DottaandPaperclip 5859e7cfa6 perf(apps): avoid repeated writes during permission saves (#21033)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps give agents access to tools through saved profiles and approval
rules.
> - A save replaces the full set of action entries and access bindings.
> - It also updates each Ask first rule and each retired rule in a
separate query.
> - Adding one agent can therefore cause hundreds of database round
trips.
> - This pull request keeps unchanged rows and batches the required
changes.
> - The benefit is faster saves with the same action permissions.

## Linked Issues or Issue Description

**What happened?**

An app permission save repeats database work for actions and agents that
did not change. With 100 actions and 500 retired rules, one agent
addition makes 636 queries. Of these, 599 update approval policies.
Related boundary coverage: Refs #19973. This change keeps that issue's
Off and agent-selection protections.

**Expected behavior**

Adding an agent applies the saved Allowed, Ask first, and Off choices.
The save changes only the rows that need a change. Independent approval
rules remain in force.

**Steps to reproduce**

1. Create an app with 100 active actions and 99 Ask first rules.
2. Add 500 disabled app approval rules.
3. Save access for three agents. Then add a fourth agent.
4. Count database queries during the save.

**Paperclip version or commit**

Baseline: `51d77648bc62`.

**Deployment mode**

Built from source. The reproduction uses disposable local PostgreSQL.

## What Changed

- Keep action entries and access bindings when their saved values match.
- Insert new bindings in one query. Remove only bindings and entries
that no longer apply.
- Read managed approval rules for the selected app. Skip unchanged rules
and rules that are already disabled.
- Batch new or changed Ask first rules and disable stale enabled rules
in one query.
- Keep existing policy conditions, priority, author, and creation time
when a rule changes.
- Add real PostgreSQL regression tests and a repeatable benchmark.
Document independent approval rules.

## Verification

- Red: the original regression made 2,995 policy writes across five
saves and failed the zero-write check.
- Green: 636 focused service and runtime tests passed after the rebase.
CI supplies the broad checks required by `pnpm verify` (exit 2).
- Benchmark: `node --import ./server/node_modules/tsx/dist/loader.mjs
scripts/benchmark-app-permission-save.ts --baseline-ref=51d77648bc62
--output=/tmp/app-permission-save.json`.
- Local benchmark: two warmups and six samples in alternating order.
Restore three agents outside each timed sample, then add the fourth
agent. Median save time fell from 458.51 ms to 38.05 ms (12.0 times
faster). Query count fell from 636 to 32. Unchanged policy writes fell
from 599 to zero.
- Compare all saved profile, entry, binding, and policy values after
mapping fixture IDs and excluding generated row IDs and write times.
Check Allowed and Ask first decisions for the added agent. Tests also
cover Off, an unassigned agent, additive access, removal, independent
policies, company boundaries, and re-enabled rules.
- CI passes on `069e2c76252a19ec3c11a061de0506eccb0463d1`: [PR
workflow](https://github.com/paperclipai/paperclip/actions/runs/43162600023).
This includes typecheck, build, server tests, and all eight browser
shards.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/21033#issuecomment-6815608829)
gives the final head 5/5. Its one finding is resolved. The separate
Paperclip Review check is still queued. Its review task stopped after a
Runner failure.
- Hosted request timing and the specific reported approval prompt are
not verified. No live permissions changed.

## Risks

- The save now retains unchanged row IDs and timestamps. Consumers must
use permission values to determine access.
- Existing transaction locks and permission checks remain in place. The
runtime concurrency tests exercise these paths.
- The benchmark measures a local service with synthetic data. It does
not establish hosted endpoint latency.
- No migration or startup change is required.

## Model Used

OpenAI Codex, GPT-6. Exact provider model ID is not exposed in this
session. Used reasoning, tool calls, and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 12:57:03 -06:00
DottaandClaude Opus 5.5 b0e99dfcfa fix(connections): show the setup card unless the person asked to set up the connection (#21400)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents find and request app connections with `connections_search`
and `connection_request`
> - For one unconnected match, the search told the agent "The user asked
to connect" and said to configure the connection with `call_api`
> - That text was given for every search, also when the person only
asked a question or asked for app data
> - The agent then created a draft connection itself and sent an
absolute `http://127.0.0.1:<port>` link, and the person got no setup
card to accept or decline
> - This pull request gives the configure instruction only when the
person's own message asks to set the connection up, and makes the
agent's sign-in handoff link a relative app route
> - The benefit is that a question or a data request shows the setup
card, and setup links open on the host the person uses

## Linked Issues or Issue Description

**What happened?**
In chat eval cohort 12 (OpenCode with DeepSeek V4.1 Flash), the person
wrote "Read my Notion and tell me the latest on the launch".
`connections_search("Notion")` returned one result with the instruction
"The user asked to connect. If method.management.allowed, use
search_api/call_api ... to configure it". The agent called `POST
tools/apps/connect` twice and replied with
`http://127.0.0.1:<port>/COM/apps/connect?resume=...`. No setup card was
shown, so the person could not decline. The same happened for "What is
the easiest way to connect Notion here?".

**Expected behavior**
When the person asks about an app or needs its data, the agent shows the
setup card (`connection_request`) and answers in plain words. The agent
configures a connection itself only when the person asks for that setup.
Links the agent gives are relative app routes.

**Steps to reproduce**
1. Have no Notion connection.
2. In chat, write "Read my Notion and tell me the latest on the launch".
3. See that the agent creates a draft connection and links an absolute
URL, with no setup card.

## What Changed

- `server/src/services/connection-intents.ts`: for a single unconnected
tool match, read the latest human message on the task. Give the
configure instruction only when its first clause opens with a setup verb
and names the matched app ("Connect Notion, then ...", "Can you set up
Notion?"). "Connect Slack, then read my Notion" does not count for
Notion. Otherwise tell the agent not to create a connection or draft.
For a setup question, the agent answers in plain words and links the
catalog setup page `/<prefix>/apps/connect?source=<app>`, because
`connection_request` ends the turn before any answer text is shown. For
a data request, the agent calls `connection_request` to show the setup
card. A saved custom connection always uses the card. The check is
conservative: a miss never configures anything.
- `server/src/services/tool-access/oauth/authorization-start.ts`: the
agent handoff link (`authorizationUrl`, also
`requiredUserAction.setupPath`) is now a relative app route, not built
from the server's redirect origin.
- Tests: a red/green service test for the three messages, and the
handoff test now requires a relative link.

## Verification

- `cd server && npx vitest run
src/__tests__/connection-intents-service.test.ts -t "configure a single
match"`: passes for a data request, a setup question, a mixed-app
request and an explicit setup request. With the old
`connection-intents.ts` it fails (`expected 'The user asked to connect.
...' to contain 'connection_request with service notion'`).
- `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t "generic app
authentication"`: passes.
- A narrow paid eval run of `connection-setup.notion-declined` and
`connection-setup.choose-provider` follows on this branch.

## Risks

- A person who asks for setup with other words ("I'd like Notion hooked
in") now gets the setup card instead of agent-side configuration. The
card still completes the setup.
- One small indexed read of the latest human message per single-result
search. There is no change to agent boot or time to first token.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:02:36 -06:00
DottaandClaude Opus 5.5 8af889f612 feat(mcp-apps): view messages with consent, app context, full screen and provider tool icons (#21266)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - MCP Apps (#21119) show interactive widgets from MCP servers, such as
Circleback meeting cards, with the agent's reply in the task chat.
> - The first version deferred three parts of the MCP Apps spec. A
widget could not propose a chat message (`ui/message`). It could not
give the agent context (`ui/update-model-context`). It could not go full
screen.
> - Paperclip also ignored the `icons` that MCP servers declare on tools
(SEP-973).
> - The security review of #21119 showed what these features need from
day one: explicit consent, untrusted-data labels and strict icon
sanitizing.
> - This pull request adds the three host features and provider tool
icons, with those protections built in.
> - The benefit is that widgets can now work with the person and the
agent, not only display data.

## Linked Issues or Issue Description

Refs #21119

**Subsystem affected**
MCP Apps host (task chat widgets), tool catalog, run wake prompt.

**Problem or motivation**
The MCP Apps host refused `ui/message` (-32601), dropped
`ui/update-model-context` and kept every view inline. MCP tool `icons`
were not parsed or shown. So widgets could not ask the agent a
follow-up, the agent could not see what the person selected in a widget,
large widgets were cramped, and tool rows used only generic icons.

**Proposed solution**
Add consent-gated view messages, per-view app context delivered to the
task's next run as untrusted data, a full-screen dialog, and sanitized
provider icons. Details are under "What Changed".

**Alternatives considered**
- Send view messages with a new server route that creates the comment
itself. Rejected: comment creation has many rules (agent chat ownership,
paused conversations, queued delivery, wakeups, receipts). The widget's
Send button uses the same send path as the composer. A small route then
records which view the comment came from.
- Keep the iframe in place for full screen with CSS. Rejected: chat
containers can create containing blocks that break `position: fixed`. A
moved iframe always reloads, so the dialog hosts a new frame instead.

**Roadmap alignment**
Continues the MCP feature work after #21119, #21140 and #21219. It does
not touch the MCP Tasks work (gateway `tools/call` polling and
elicitation).

## What Changed

- **`ui/message`.** The host shows "<App> wants to send: …" with
**Send** and **Dismiss** under the widget (and inside the full-screen
dialog). Nothing is sent before the click. Send posts the text as the
viewer's own comment through the chat's normal send path
(`handleThreadAdd`). `POST
/api/issues/:issueId/mcp-app-views/:viewId/messages` then logs
`issue.mcp_app_message_sent`. It accepts only the caller's own comment
on the same task, matched by `clientRequestId`, from the last 10
minutes. One proposal per view can wait at a time. The view gets `{}`
after the send, or -32000 "Message sending denied" after Dismiss. Only
`user` text up to 8,000 characters is accepted.
- **`ui/update-model-context`.** `PUT
/api/issues/:issueId/mcp-app-views/:viewId/model-context` stores the
latest context per view in `tool_mcp_app_views.model_context` (text
blocks and `structuredContent`, 16 KB at most). Updates are applied in
order and deduped in the browser and on the server. A change logs
`issue.mcp_app_context_updated`, and an empty update clears the context.
The task's next run gets the three most recently updated contexts as an
`## App context (untrusted)` wake-prompt section. The section quotes the
data as JSON and says it is third-party data, not instructions. The read
runs in parallel with the other run preparation and never fails the run.
CLI adapters and the native runner render it through the same
`renderPaperclipWakePrompt`.
- **Who may do this.** Messages and context need a signed-in person with
`issue:comment` on the task. Agents and API keys get 403. Other tasks
and companies get 404. `message` and `updateModelContext` are advertised
in `hostCapabilities` only where the viewer can send messages.
- **Full screen.** The host advertises `availableDisplayModes:
["inline","fullscreen"]` and handles `ui/request-display-mode`. It
honors the view's declared modes and answers with the resulting mode. It
sends `host-context-changed` with `displayMode` and the container (fixed
`height`/`width` in full screen), and again when that container resizes.
A new **Open full screen** control sits next to "Widget from …". Full
screen is a Paperclip dialog that closes with Exit, Escape or the view's
own request.
- **Provider tool icons.** `icons` on MCP tool descriptors are sanitized
into the new `tool_catalog_entries.icons` column, outside the version
hash, so a new icon never quarantines a reviewed tool. Only `https:`
URLs and `data:image/*` URIs are kept, and SVG only as `data:`. A
`data:` icon is at most 32 KB, a URL at most 2,048 characters, with at
most 4 icons per tool. `GET /api/companies/:companyId/tool-icons` lists
them keyed by gateway tool name, in one cached request per chat. Task
chat tool rows and the app's actions list show them through
`ProviderToolIcon`. It renders `<img referrerPolicy="no-referrer">` at a
fixed size and never inlines markup. It checks the source again before
rendering and falls back to the taxonomy icon.
- **Schema.** Migration `0477` adds `tool_catalog_entries.icons`,
`tool_mcp_app_views.model_context` and `model_context_updated_at`, all
nullable.
- **Docs and stories.** `doc/connections/MCP-APPS.md` documents the new
behavior, and the finished items are removed from "Limits and
follow-ups". `Tasks/MCP App views` has new stories: "View asks to send a
message", "Full screen" and "Tool rows with provider icons". The fixture
view now uses `ui/message`, `ui/request-display-mode` and
`ui/update-model-context`.

## Verification

- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts
server/src/__tests__/tool-catalog-publication.test.ts
server/src/__tests__/openapi-routes.test.ts
packages/shared/src/mcp-icons.test.ts
packages/shared/src/mcp-apps.test.ts`: pass (embedded Postgres). These
tests cover:
- Context: stored, deduped, logged once, 413 when too large, cleared,
scoped to the task and company, refused for agents (403), and rendered
into the wake prompt as untrusted data.
- Messages: attributed only to the viewer's own recent comment on the
same task. A teammate's comment, an unknown request id or an old comment
gets 404. An agent gets 403.
- Icons: stored sanitized, and the version hash and status do not
change. Another company sees nothing.
- `cd ui && pnpm exec vitest run src/components/mcp-app/
src/components/ProviderToolIcon.test.tsx src/pages/apps/app-detail/`: 69
pass. These tests cover:
- The host protocol for each new method: capabilities, consent wait, one
proposal at a time, invalid input, ordered and deduped context, the size
limit, and the display-mode rules.
  - The consent card (nothing sent without Send).
  - The full-screen dialog (one frame at a time).
  - Icon rendering (no-referrer `img`, no inline SVG, fallback).
- `cd ui && pnpm typecheck` (including Storybook) and `tsc --noEmit` for
the server: clean.
- Red/green: the new route and host tests fail on `master`, where the
routes do not exist and the host answers -32601 or drops updates. The
icon descriptor test failed when the publication write was removed.
- Not run locally: the full suite, and
`packages/adapter-utils/src/server-utils.test.ts › runChildProcess ›
does not crash when the child closes stdin before a delayed write`. That
test times out on this Mac with or without this change, so it is a local
platform issue. CI covers both.

## Risks

- Migration risk is low: three nullable columns, no backfill.
- Every run on a task now does one extra indexed read, in parallel with
the other preparation. Errors are logged and the run continues without
app context.
- If the attribution request fails after the comment is sent, the
comment stays without the `mcp_app_message_sent` activity row. This is
documented.
- Switching between inline and full screen restarts the view, so state
inside the widget resets. This is documented as a limit.
- Icon names shortened with a hash by the gateway (very long names) do
not match, and those rows keep the taxonomy icon.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution. One subagent (the same model) built the provider-icon
part.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 07:32:48 -06:00
DottaandClaude Opus 5.5 2d1c2db64a feat(mcp): host the MCP Skills extension for connected apps (#21219)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use connected apps through MCP. Paperclip already hosts MCP
Apps views and speaks stateless MCP 2026-07-28.
> - The MCP Skills extension (SEP-2640, Final) lets an app publish
skills (instructions and files) next to its tools. Hugging Face already
serves 25.
> - Paperclip did not read them, so agents missed each app's own
workflow guidance.
> - Skills are a prompt-injection surface. The spec makes manifest
binding, verification, origin tagging and approval revocation host
requirements.
> - This pull request makes Paperclip a Skills host: discover, review,
verify, and deliver app skills to the agents that use the app.
> - The benefit is that connected apps can teach agents their workflows,
under board control.

## Linked Issues or Issue Description

**Problem**
Connected MCP servers can publish skills with the
`io.modelcontextprotocol/skills` extension. Paperclip ignored them. A
board could not see, review or approve them, and agents never received
them.

**Proposed solution**
Implement the host side of SEP-2640. List skills during catalog refresh,
record them per connection and company, bind board approval to the full
file manifest, and load approved skills into agent runs with verified,
lazily fetched content.

**Why it matters**
Apps ship workflow guidance with their tools. Without the extension,
each team must copy that guidance into company skills by hand, and it
goes stale.

Refs #21119, Refs #21140 (the MCP Apps and stateless MCP client work
this builds on).

## What Changed

- **Client capability.** Every stateless request declares
`io.modelcontextprotocol/skills`. Legacy handshake sessions do not. The
capabilities that `server/discover` advertises are remembered beside the
protocol era.
- **Discovery.** Catalog refresh calls `skills/list` (paged) only when
the server declared the extension. A listing inside its `ttlMs` is
reused. A skills failure never fails the tool refresh. Nothing runs
while a run starts.
- **Schema.** New `tool_mcp_skills` (company, connection, skill URI;
frontmatter, sorted manifest, manifest digest, review state) and
`tool_mcp_skill_files` (verified bytes keyed by connection, URI and
digest). Both cascade with the connection.
- **Governance.** New skills wait for review. Approval binds to the
order-free digest of every `{uri, digest, size}`; a request for any
other digest gets 409. A changed manifest or frontmatter revokes
approval and logs `tool_mcp_skill.approval_revoked`. Dynamic, oversized
and invalid skills are declined with a reason. Approve and decline use
the company skill policy (`skills.install`, `external_package`,
`mcp-connection:<id>`).
- **Run time.** Approved skills of connections the agent can use load in
parallel with other skill preparation, inside a 3 s budget. `SKILL.md`
is read with the agent's grant and checked for size, SHA-256 and
frontmatter. A failure calls `skills/get` and takes the skill out of
use. Each skill becomes a per-run bundle on the connector skill path,
named in its app's namespace, tagged with its origin. Paperclip writes
the frontmatter: `allowed-tools` only if the board approved it.
- **Supporting files.** `GET
/api/agents/me/mcp-skills/:skillId/files?path=` serves manifest-listed
files to an agent in an active run, through the skill's own connection.
The sandbox bridge allows the route.
- **Board API and UI.** `GET/POST /api/companies/:companyId/mcp-skills…`
(list, read instructions, approve, decline). The Skills page shows
"Skills from connected apps" with app, state, size, requested tools and
plain-language notes. It renders nothing when no app publishes skills.
- **Docs.** `doc/connections/MCP-SKILLS.md`, linked from
`doc/connections/README.md`.
- **Shared frontmatter parser.** It now decodes YAML single-quoted
scalars. Before, 5 of Hugging Face's 25 skills kept their quotes and
would fail the frontmatter check; now all 25 match. The generated runner
copy is regenerated.

File size notes: `server/src/services/tool-gateway.ts` grows by about
200 lines. The run-time reads must use its grant resolution and
`callAssignedConnectionProtocol`, like MCP Apps view reads. Splitting
the gateway is a separate refactor. `CompanySkills.tsx` and
`CompanySkills.production.tsx` gain 2–4 lines each; the section lives in
its own component.

## Verification

- `server/src/__tests__/mcp-skills.test.ts` (15 tests) uses a fake
stateless skills server and embedded Postgres. It covers: discovery only
after the server declares the extension; the capability in `_meta`;
`ttlMs` reuse; lazy fetch (no reads at listing or approval; `SKILL.md`
at load; supporting files on read); a stale-digest approval refused; a
reorder that keeps approval; a changed or added file that revokes it;
dynamic skills declined; a same-size tampered `SKILL.md` rejected,
refreshed with `skills/get`, and taken out of use; mismatched
frontmatter rejected; reads outside the manifest refused; the same URI
and name from two servers kept apart; company scoping; a slow app
skipped within the budget and cached later; HTTP authorization (board
only, agent needs an active run, run header must match) and activity
logging.
- Red/green: with the digest check disabled, the tamper test fails; with
the manifest comparison disabled, the revocation test fails. Both pass
with the code.
- `server/src/__tests__/mcp-protocol-versions.test.ts`,
`mcp-apps.test.ts`, `connection-instructions.test.ts`,
`tool-gateway-discovery-http.test.ts`: 47 pass (two capability
assertions updated for the new extension).
- `packages/shared/src/frontmatter.test.ts` adds a single-quoted scalar
case; the runner contract test passes (23 tests).
- `ui/src/components/skills/ConnectedAppSkills.test.tsx` (3) and
`ui/src/pages/CompanySkills.test.tsx` (44) pass.
`packages/adapter-utils/src/sandbox-callback-bridge.test.ts` (89)
passes.
- `pnpm --filter @paperclipai/server typecheck` passes. UI `tsc` passes
except the known missing generated Storybook catalog files.
- `pnpm verify` returned exit 2 (doc change is a global CI input);
required CI supplies broad coverage.
- Live check against `https://huggingface.co/mcp` without credentials,
through Paperclip's own services and the real egress guard: 25 skills
discovered as pending; three approved; cold load 135 ms, cached load 13
ms; a supporting file read through the manifest; the bundle tagged with
its origin.
- Storybook: `Agents and reusable capabilities/Skills and Skill
Studio/Connected app skills` (needs review, all states, tool request,
name collision, reading instructions).

## Risks

- **Migration:** two new tables only. No changes to existing tables.
Safe with the previous release running.
- **Run start:** companies without an approved app skill pay one indexed
read. With approved skills the wait is capped at 3 s; late or failing
skills are left out with a `skills.mcp_unavailable` run warning and
never fail the run.
- **Agent harness enforcement:** Paperclip cannot gate each
code-execution tool call an agent makes while following an app's skill.
Approval is explicit and per skill, the approval note says agents may
run commands the skill describes, and `allowed-tools` is never granted
without a separate tick.
- **Shared-home adapters** receive app skills like company skills. A
revoked skill leaves on the next skill sync.
- Not in this PR: per-connection opt-in for dynamic skills; serving
Paperclip's own skills from its public MCP endpoint;
`resources/directory/read`; app skills on an agent's skills tab.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, local test runs, live HTTP checks).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 19:18:06 -06:00
DottaandClaude Opus 5.5 267983356c feat(mcp): speak stateless MCP 2026-07-28 to provider servers, keep older servers working (#21140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through Paperclip's MCP client. The client calls
each provider's remote (Streamable HTTP) or local (stdio) MCP server.
> - The client spoke only protocol 2025-06-18. It always did the
`initialize` handshake and it sent client capabilities only in that
handshake.
> - MCP 2026-07-28 removes the handshake and sessions. Every request
carries its version, capabilities and client identity in `_meta`.
Servers that only speak 2026-07-28 reject the old requests, and they
never see that Paperclip renders MCP Apps views.
> - Public servers already speak it: DeepWiki and Context7 answer
`server/discover` with 2026-07-28 today.
> - This pull request makes the client negotiate the version per server.
It speaks stateless MCP to 2026-07-28 servers and keeps the handshake
for 2025-11-25 and older servers.
> - The benefit is that new and old providers both work, and a tool call
to a stateless server is one HTTP request with no handshake.

## Linked Issues or Issue Description

Refs #21119 (MCP Apps host). This PR sends its MCP Apps capability to
stateless servers too.

**Subsystem affected**
Server: the MCP client in `server/src/services/mcp-http.ts`, catalog
discovery, the tool gateway (`tools/call`, context reads, stdio), and
the remote MCP OAuth flow.

**Problem or motivation**
The client used protocol 2025-06-18 only. MCP 2026-07-28 makes MCP
stateless: no `initialize`, no `Mcp-Session-Id`, per-request `_meta`,
`server/discover`, `resultType` with multi round-trip requests (MRTR),
cacheable lists, required `Mcp-Method`/`Mcp-Name` headers, and RFC 9207
`iss` validation. Servers that speak only 2026-07-28 cannot be used.
Paperclip advertised MCP Apps support only in `initialize`, so stateless
servers never saw it.

**Proposed solution**
Probe each server once with `server/discover`, pick the newest mutually
supported version, and save it on the connection. Send stateless
requests to modern servers and keep the handshake for legacy servers.
Tool calls use the saved version and never probe. The saved version is
checked again only when a request built from it is refused.

**Alternatives considered**
Inline negotiation (send each request in modern shape first and fall
back on `400`) adds a failed request to every call to a legacy session
server. Probing on every tool call adds a round trip before each tool
runs. Saving the negotiated version keeps tool calls at one request.

**Roadmap alignment**
MCP protocol currency for the Apps and connections substrate. It follows
the MCP Apps host work in #21119.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/changelog. Out
of scope (separate work): the Tasks extension, Skills over MCP,
`subscriptions/listen`, progress and cancellation, icons,
enterprise-managed authorization.

## What Changed

- `mcp-http.ts`: `negotiateMcpProtocolVersion` probes with
`server/discover`. A `DiscoverResult` or a modern error
(`-32020`/`-32021`/`-32022`) means a modern server. Anything else means
a legacy server, which gets `initialize`. A `401`/`403` keeps the
challenge for OAuth discovery. The era is remembered per connection and
endpoint.
- `mcpHttpRequestInit` builds every request. It always sends
`Mcp-Method` and `Mcp-Name` (Base64 sentinel when unsafe). For modern
versions it adds `_meta` (`protocolVersion`, `clientCapabilities` with
`extensions["io.modelcontextprotocol/ui"]` and URL elicitation,
`clientInfo`) and `Mcp-Param-*` headers from `x-mcp-header`.
- Legacy `initialize` now requests 2025-11-25 and advertises the MCP
Apps extension.
- Catalog refresh saves `config.mcpProtocolVersion`. It drops tools with
invalid `x-mcp-header` annotations. It reuses a fresh `tools/list`
result (`ttlMs`, `cacheScope`). It probes again when the saved version
is refused, and it negotiates again after a credential refresh.
- `tools/call`: a saved modern version is called directly (no probe, no
handshake). A modern rejection of a legacy-shaped call, or `-32022`,
gets one retry with a negotiated version. The server rejected the first
request before it ran, so the retry cannot repeat a side effect. A
remembered legacy server that refuses `initialize` is probed again.
- MRTR: `input_required` with URL requests becomes the existing provider
link flow (`provider_interaction_required`). The next identical call
sends `requestState` and `inputResponses`. Form, sampling and roots
requests stop with `provider_input_required` and a plain message. A
missing `resultType` means `complete`.
- Context reads (`resources/*`, `prompts/*`) use the saved version,
cache fresh results, and stop on `input_required`. The error code is
kept, so resource not found (`-32602` or `-32002`) is visible.
- stdio: probe with `server/discover` (2 s limit). Restart a server that
exits on the probe. Send `_meta` to modern servers.
- OAuth: refuse a callback without `iss` when the server advertises
`authorization_response_iss_parameter_supported`. Check `iss` before
acting on a provider error. Send `application_type: "native"` for
localhost callbacks and `web` otherwise. Issuer-bound clients and the
CIMD-first order already existed.
- Docs: `doc/connections/MCP-PROTOCOL-VERSIONS.md`, linked from
`doc/connections/README.md`.

## Verification

- New suite `server/src/__tests__/mcp-protocol-versions.test.ts` (19
tests) uses a fake 2026-07-28 server and a fake legacy session server.
It covers discovery, `_meta`, headers, version fallback, `-32022`,
`resultType`, MRTR link and retry, form stop, cache TTL, legacy flow,
and recovery when a saved legacy server moves to 2026-07-28. Embedded
Postgres runs the end-to-end cases.
- New OAuth tests in `generic-mcp-connection.test.ts`: `iss` required
when advertised, `iss` checked on provider errors, `native` for
localhost, and a stateless server listed again after a revoked token.
- Red/green: with the master service code, all 21 new protocol and OAuth
tests fail. With this change they pass. The token-refresh regression
test fails without its fix and passes with it.
- Existing suites: 12 touched files, 1088 tests pass. `tsc --noEmit`
passes for `server`. Updated expectations count the one-time
`server/discover` probe, and the Calendly DCR test now expects `native`
for its localhost callback.
- `pnpm verify` exits 2 (a doc is a global CI input), so CI supplies the
broad coverage.
- Live check with the changed client, without credentials:
- DeepWiki (`mcp.deepwiki.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (3 tools).
- Context7 (`mcp.context7.com/mcp`): discover 200 → 2026-07-28, no
session, `tools/list` 200 (2 tools).
- GitMCP (`gitmcp.io/docs`): discover 400 → `initialize` (2025-03-26) →
session → `tools/list` 200 (5 tools).
- Circleback (`circleback.ai/api/mcp`): discover 401 with its
`WWW-Authenticate` challenge kept for sign-in.

## Risks

- A legacy server that is not saved as session-required gets one extra
request (`server/discover`) the first time it is refreshed. Then the
version is saved. Saved session-required connections are not probed.
- DCR from a localhost callback now registers a `native` client. This is
what the spec asks for. Existing registrations are reused.
- An authorization server that advertises `iss` support but does not
send `iss` now fails sign-in with a clear message. The spec requires
this.
- MRTR answers are kept in memory for 30 minutes. After a restart the
server asks again, and the user sees the link again.
- No schema change. `config.mcpProtocolVersion` is a new key in the
existing JSON config.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits, web fetch of the MCP spec), extended
reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 14:34:26 -06:00
DottaandClaude Opus 5.5 8ff45c0fa9 feat(connections): render MCP Apps views in the task chat, starting with Circleback (#21119)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents reach external apps through MCP connections. Paperclip
records each call and shows it as a tool row in the task chat.
> - MCP Apps (SEP-1865, extension `io.modelcontextprotocol/ui`, spec
2026-01-26) lets an MCP server attach an interactive `ui://` view to a
tool. Claude, Claude Code and VS Code render these views in chat.
Circleback uses them to show meeting cards.
> - Paperclip dropped the tool's `_meta.ui` at discovery and the
result's `_meta` at call time. Its gateway reported MCP App UI as
"unsupported". Circleback was reachable only through Composio, whose
toolkit has no `Display*` tools.
> - This pull request makes Paperclip an MCP Apps host. It also adds
Circleback as a native catalog connection.
> - The benefit is that a Circleback recap in a task shows the same
meeting cards as in Claude, and any other MCP server with views works
the same way.

## Linked Issues or Issue Description

**Subsystem affected**
Connections and apps (remote MCP gateway) and the task chat.

**Problem or motivation**
An agent that uses Circleback in Paperclip can only return text. The
same tools in Claude Code show interactive meeting cards
(`DisplayMeetings`). Paperclip discards the MCP Apps metadata, so it
cannot show them.

**Proposed solution**
Paperclip implements the host side of MCP Apps end to end. It keeps the
tool's view declaration and records each call that has a view. It
renders the view under the tool row in a sandboxed iframe that uses the
MCP Apps postMessage protocol. Circleback becomes a native catalog
connection, so its `Display*` tools are available.

**Alternatives considered**
Composio's `circleback_mcp` toolkit has 11 read tools and no `Display*`
tools, and its meta-tools return plain JSON, so it cannot carry views.
The spec's double-iframe sandbox proxy on a separate origin needs a
second origin that Paperclip does not have. A single opaque-origin
`srcdoc` frame with an injected CSP gives the same isolation.

**Roadmap alignment**
This extends the existing Apps catalog and MCP gateway. It adds no new
integration substrate (decision D4 in doc/connections/README.md).

**Additional context**
The standard, the design and the testing plan are in
`doc/connections/MCP-APPS.md`.

## What Changed

- **Shared:** `packages/shared/src/mcp-apps.ts` adds MCP Apps types,
`_meta.ui` and resource parsing (current and legacy keys), and a CSP
builder that keeps only well-formed `https` origins.
- **DB:** migration `0471` (`online_compatible`) adds
`tool_catalog_entries.ui_meta` and a new `tool_mcp_app_views` table.
- **Server, catalog:** discovery keeps each tool's `_meta.ui`. It is not
part of the version hash, so a view never quarantines a reviewed tool.
- **Server, gateway:**
- Paperclip advertises the MCP Apps client capability when it
initializes a session.
  - App-only tools (`visibility: ["app"]`) are hidden from agents.
- The result `_meta` is kept for the view and is not sent to the agent.
- Each successful call to a tool with a view records a view row.
Recording is best effort.
- The gateway's `initialize` now reports `paperclip/mcp-app-ui:
host_rendered`.
- The context protocol (`resources/read`) now uses the injected test
transport, like the other remote calls.
- **Server, routes:** `server/src/routes/mcp-apps.ts`
- `GET /api/issues/:issueId/mcp-app-views` lists a task's views. It
needs task read access.
- `GET /api/issues/:issueId/mcp-app-views/:viewId` returns the view
document, read live through the agent's connection.
- `POST /api/issues/:issueId/mcp-app-views/:viewId/tool-calls` runs the
view's `tools/call`. It needs `tools:use` and reaches only tools of the
same connection that allow `app`. It uses the connection test-call path,
so policy, Ask first and audit apply.
- **UI:** `ui/src/components/mcp-app/`
- The host protocol: `ui/initialize` with theme, style variables and
size limits; tool-input; tool-result; size-changed; `tools/call`;
`ui/open-link` (http and https only); teardown.
- An opaque-origin iframe (`allow-scripts`, never `allow-same-origin`)
with the CSP placed before the view's markup.
- The views of a run render with that run's reply, in the live turn and
in settled history. They are keyed by the reply's run, so history needs
no transcript.
- **Circleback:** a native catalog app (`mcp-oauth`, DCR, `user
user:write`) with:
  - the official logo;
- a reviewed risk classifier (Share* tools are writes, Delete* tools are
destructive, Display* tools are reads);
  - warnings in the sign-in step;
  - `doc/connections/CIRCLEBACK.md`.
  
Search now prefers a native app over an aggregator listing named "X
MCP".
- **Storybook:** `Tasks/MCP App views` shows the tool row with its view,
the ready state (Refresh calls a tool), loading, view unavailable, and a
tool call that needs approval.

## Verification

- `pnpm exec vitest run packages/shared/src/mcp-apps.test.ts
ui/src/components/mcp-app/`: 19 passed.
- `pnpm exec vitest run server/src/__tests__/mcp-apps.test.ts`: 3
passed, on embedded Postgres. The test connects Circleback through the
catalog with DCR OAuth against a fake Circleback MCP server. It checks:
  - `ui_meta` is stored;
  - an app-only tool is hidden from the agent;
- an agent call records one view with arguments, structured content and
`_meta`;
  - a call without a view records nothing;
  - the routes serve the view's HTML and CSP;
  - the view's tool call works and a model-only tool is refused;
  - another task, another company and an agent actor are refused.
- Circleback connector: 13 focused files, 645 tests passed. These are
the app definitions, search, aggregator fallback, connection intents,
tool policy, `AppsConnect` and `Browse`.
- Typecheck passed for `@paperclipai/shared`, `@paperclipai/server` and
`@paperclipai/ui`. `node scripts/storybook-catalog.mjs` and the
migration compatibility check also passed.
- Storybook check in the browser:
- In both themes, the view runs the handshake, renders the meeting cards
from `structuredContent`, picks up the theme variables, and resizes to
its content.
  - Refresh sends `tools/call` and re-renders the cards.
- **Live Circleback end to end** on a local instance, with a real
Circleback account:
- Connected through the catalog entry with Circleback's dynamic client
registration and OAuth.
  - Discovery stored the real view declarations:
    - `DisplayMeetings` uses `ui://circleback/meetings.v2.html`.
    - `DisplayActionItems` and `DisplayCalendarEvents` also have views.
- `ListAssigneeOptions` and `SetCalendarEventRecording` are app-only
helpers with no view. They are hidden from agents.
  - A `claude_local` agent called `DisplayMeetings` on two tasks.
- The real 3.5 MB Circleback view rendered the meeting cards with the
agent's reply.
- Clicking a card opened that meeting in Circleback through
`ui/open-link`.
- The live run found two bugs, both fixed in this PR, each with a
regression test:
    - The 1 MB tool-result limit blocked the view document.
    - Views were hidden inside the folded tool history.

## Risks

- **New migration:** a new table and a nullable column. It is
`online_compatible`.
- **Existing connections:** tools get `ui_meta` on the next catalog
refresh. Until then, no view shows and the tool row is unchanged.
- **Untrusted provider HTML:** the view runs as provider HTML in an
opaque-origin sandbox. It cannot reach Paperclip's session, DOM or API.
Its network access is limited to the origins its server declared. A
view's tool calls use the viewer's permissions and the normal policy and
Ask-first path.
- **Agent tool lists:** app-only tools are removed from them. No current
provider marks tools app-only, so no listed tool disappears.
- **Search:** the change that prefers a native app over an "X MCP"
aggregator listing affects every app.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
subagents (one subagent authored the Circleback catalog connector).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:28:05 -06:00
DottaandClaude Opus 5.5 b033cc124d fix(evals): repair private human steps for webhook, device and key entry; name failed private steps (#21012)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat evaluation catalog checks agent replies and then the human
step that follows, such as a person who reveals a key or approves a
device
> - Some human steps run in a separate private browser process, because
they handle credentials. CI keeps the output of that process private on
purpose
> - Four cases failed only in that private step, so CI showed one fixed
message and no cause
> - I ran each private step locally against a dev server, where its
output is visible, and found the cause of three failures
> - This pull request fixes the harness and two product defects, and
adds a closed failure record so that the next CI failure names its step
> - The benefit is that these human steps work, and a private-step
failure is diagnosable without any credential in logs

## Linked Issues or Issue Description

Bug report (following `.github/ISSUE_TEMPLATE/bug_report.yml`).

**What happened?**
- `routines.webhook-trigger`: the private person opens the routine
overview. The trigger cards are only in the Triggers section, so the
step waits for a missing button. An unhandled response wait then stops
the process.
- `connection-setup.secure-api-key`: a key-protected MCP server answers
401 without a challenge header. Paperclip shows "Remote app returned
HTTP 401" and does not ask for a key. The person cannot enter the key in
the normal flow.
- `assistant-connections.device-login`: an expired device code page
shows "This item doesn't exist or was moved." The server sends "This
code is expired, already decided, or invalid." in the OAuth
`error_description` field, and the UI drops it.
- `onboarding.new-company`: the private step reads back all saved state
and then fails one proof check. CI does not say which check.

**Expected behavior**
- The person reveals the webhook key in the Triggers section and sends
authenticated events.
- A pasted MCP link that answers 401 shows the key field.
- An expired device code says that the code is expired.
- A private-step failure names its fixed step and error class, with no
value.

**Steps to reproduce**
1. Start a local dev server.
2. Create a routine with a bearer webhook, and run the private webhook
worker. It times out on the "Edit webhook" button.
3. Paste the address of an MCP server that answers a plain 401 into
Connectors > Add by link. No key field appears.
4. Open `/mcp-device?user_code=<expired code>`. The page shows generic
not-found text.

**Paperclip version or commit**
`master` (reproduced on origin/master before this change)

**Deployment mode**
Local dev server (embedded PGlite) and the CI chat-eval runner

**Installation method**
Source checkout (`pnpm dev`)

## What Changed

- `integrator-webhook-private-worker.ts`: open `<routine>/triggers`
(also for the setup-resume link), and attach a handler to the early
response wait so a failed click reports its real step.
- `server/.../catalog/discovery.ts`: a 401 with no `WWW-Authenticate`
header, to a connection that sends no credential, is handled like a
challenge without discoverable sign-in (`oauth_challenge`). The existing
UI guidance then asks for a key. A 401 to a connection that already
sends a credential keeps the old error.
- `ui/src/api/errors.ts`: read the OAuth `error_description` field as a
readable server message.
- New `private-child-failure.ts`: a private child records only a fixed
step name and an error class from a fixed list. The parent echoes a step
only when it is in the caller's closed list. Error text, stdout and
stderr stay private.
- The webhook, private browser and identity children use it. Each has a
closed step list file.
- `identity-new-company-native-proof.ts`: each proof check has a fixed
reason name. The private child reports `new-company-proof-<reason>` when
a check fails. The accepted conditions do not change.
- Tests: server red/green test for the bare 401, UI red/green test for
`error_description`, unit tests for the failure record (no text leaks,
closed list only, extra fields dropped).

## Verification

- Local dev server (embedded PostgreSQL), private workers run directly
with the real child process:
- Webhook, ready and setup-pending triggers: before, timeout on "Edit
webhook". After, receipt with invalid status 401, valid status 202 and a
linked task.
- Secure key entry with the synthetic notes provider: before, timeout on
the key field. After, an active connection with health `ok` and 2
authorized `tools/list` calls.
- Expired device page: before, `at step device-expired-page
(TimeoutError)`. After, `expiredPageVisible: true`.
- `server`: `vitest run src/__tests__/tool-access-service.test.ts -t
"pasted link answers"` (fails before the discovery change with 502,
passes after).
- `ui`: `vitest run src/api/errors.test.ts` (fails before, passes
after).
- `pnpm test:e2e:runner:typecheck`, `pnpm test:e2e:runner:unit`, server
`tsc --noEmit`.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD sha>
--check`: unchanged.
- One paid run of the four cases on this branch. Results are in a PR
comment.

## Risks

- A plain 401 from an MCP link now returns 422 `oauth_challenge` instead
of 502. The draft connection is removed, as for other challenges. Agents
that call the connect API see the "needs sign-in" answer instead of a
raw status. This only applies when the connection sends no credential.
- Some API errors that carry `error_description` now show that text
instead of generic copy. That text is written for people.
- The failure record is opt-in per child and contains only constants.
The parent ignores a record with any other field.
- `onboarding.new-company` is not fixed here. Its native hello probe
cannot start locally on macOS, so the failing check is not yet known.
This change makes the next run name it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file edits, local browser automation through Playwright). Extended
reasoning was on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 22:06:29 -06:00
DottaandPaperclip 5bad074baa feat: add individual agent interaction permissions (#20411)
Add eleven sparse incoming and outgoing restrictions, human CLI/API administration, shared mutation enforcement, and pending-delivery checks. Preserve existing defaults, agent settings, and existing authorization boundaries.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-09 20:34:04 -06:00
DottaandPaperclip c67909cb0f fix: keep tool approvals valid through OAuth refresh and delayed review (#20340)
## Thinking Path

> - Paperclip lets people manage agent work and approve app actions.
> - An approval must bind the action that the person reviewed.
> - OAuth access tokens can expire while that person is away.
> - The old target snapshot treated each token refresh as a new
authorization. It also imposed a one-hour review deadline.
> - This pull request separates token renewal from authorization
identity and keeps the human decision durable.
> - Current access, the reviewed arguments, meaningful target changes,
task closure, and execution receipts still control dispatch.
> - People can approve later without repeating a decision after routine
connection maintenance.

## Linked Issues or Issue Description

**What happened?**

A pending app action returned `approved_tool_target_changed` after its
OAuth tokens refreshed. The tool and connection settings were unchanged.
The same card also expired after one hour. A sign-in problem could
discard an accepted decision before provider dispatch.

**Expected behavior**

The original card remains reviewable after routine OAuth refresh and a
server restart. A later approval runs the exact reviewed action once. A
connection repair preserves the decision while Paperclip verifies the
same authorization. Meaningful changes require a new review.

**Steps to reproduce**

1. Connect an OAuth MCP app and configure an action to require approval.
2. Request the action, then refresh its access and refresh tokens before
approving.
3. Approve the original card. Before this change, it returns HTTP 409
even when the authorization is unchanged.
4. Wait more than one hour. Before this change, the card expires without
a task-specific deadline.

**Paperclip version or commit**

Reproduced from master at `76254ead2229`. The final branch is rebased
onto current master.

## What Changed

- Add versioned approval target comparison. New snapshots bind
credential identity epochs, consent, permissions, ownership, connection
configuration, tool contracts, and credential references. Authenticated
OAuth refresh preserves identity. Manual credential replacement does
not.
- Remove the implicit one-hour deadline from ordinary app action
reviews. Keep explicit deadlines and terminal-task expiry. Synchronize
the nullable interaction deadline in shared contracts and UI.
- Keep accepted, undispatched actions when sign-in is required. Show a
saved-approval repair state. Resume on committed connection changes
after target and access revalidation, without an idle retry loop.
- Preserve the existing execution claim, result receipt, and
uncertain-outcome no-replay rules.
- Explain meaningful target changes without inviting the person to retry
the stale card.
- Add a production task-thread Storybook example and document
compatibility and recovery behavior.

## Verification

- Red: production-refresher invalid-grant handling and formal-approval
withdrawal during a sign-in wait also reproduced as failing regressions
before their fixes.
- Red: before implementation, the local OAuth HTTP fixture failed for
both public and confidential clients with `approved_tool_target_changed`
after token refresh.
- Green: the extended fixture passes for public and confidential clients
with personal and shared grants. It covers actual local HTTP token
exchanges, multiple refreshes, a rebuilt gateway, an ended originating
run, approval after 25 hours, duplicate approval, preserved sharing
arguments, sign-in waiting, and event-driven repair. The gateways use
the production OAuth refresher; an invalid refresh preserves the
accepted decision before provider dispatch. Omitted-scope responses
cover equivalent scope-reporting changes. The provider is an isolated
local fixture, not a deployed Notebook service.
- Negative controls refuse changed consent, scopes, tool definition,
destination, and manual credential replacement before provider
execution.
- Existing gateway service, connected-tool acceptance, secret owner,
secret service, unsigned-review queue, and connection-health suites
passed. Explicit-deadline fixtures still prove expiry behavior,
including deadlines on accepted sign-in waits. A withdrawn formal
approval cancels a waiting action before connection recovery.
- Approval card, task-chat, error-copy and target-comparison tests
passed. Server and UI typechecks passed. Token gates and Storybook
catalog checks passed.
- Inspected the new saved-approval repair card in local Storybook using
the production task thread. It shows a reconnect link and no second
approval button or false running state.
- `pnpm verify --explain` returned exit 2 and requested broad CI
coverage. [Full
CI](https://github.com/paperclipai/paperclip/actions/runs/42677855507)
passed on `eb4f1c7caa9ee6e802c314ffc5eadec475a5de0c`, including build,
typecheck/release registry, general and serialized tests, and all eight
browser shards.
- Greptile reviewed the same final head at 5/5. Both findings from the
first review are fixed and their threads resolved; no open follow-ups
remain.

## Risks

- No database migration is needed. The new comparison reuses the vault
identity epoch. Only authenticated OAuth refresh code can preserve that
epoch; HTTP credential-write schemas cannot select that behavior.
- Earlier signed requests keep their original exact-version checks and
saved deadlines. Already-expired cards are not revived. A historical
card whose credentials already changed needs a fresh review.
- A new consent, account, permission set, destination, or manually
replaced credential must not inherit the old authorization. Reconnection
continues automatically only when revalidation proves the same saved
authorization.
- The interaction enrichment now permits a null deadline. Existing
string deadlines remain valid. The web client and shared validator
change together.
- This PR does not deploy to staging or perform a live Notebook
publication.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
browser inspection. The runtime does not expose the exact model ID,
context-window limit, or reasoning-mode label; those values are not
asserted.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:46:28 -06:00
DottaandClaude Opus 5.5 267f54ef72 fix(chat): make task-not-found, unreachable-MCP and board denials final for agents (#20208)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in Agent Chat, and the agents act
through the Paperclip API
> - A board-only denial now tells the agent that it is final and where
the person acts (#20151, #20182). Other refusals and errors still give
no sign that they are final
> - So the agent keeps trying until the reply deadline, and the person
gets no answer. In chat evaluations this happened on a task lookup that
returned 404, on an MCP server that Paperclip could not reach, on a
model question, and after a final budget denial
> - This pull request makes those answers say that they are final and
what the person must do, and it tightens the chat guidance after a
board-only denial
> - The benefit is a prompt, correct reply instead of a long search or
forbidden workarounds

## Linked Issues or Issue Description

Refs #20151, #20182 (same remedy for board-only denials).

**What happened**
In four Agent Chat evaluation cases on a slower model, the agent did not
reply before its deadline:
- Asked to open a task from an old link, the agent got `404
{"error":"Issue not found"}`. It quoted the private-task guidance, then
searched archived projects, activity and the old host for 3 more
minutes.
- Asked to connect an MCP server at an unreachable URL, the agent got
`502 {"error":"fetch failed","code":"runtime_error"}` from the health
check. It retried, then used `curl` on the host and gateway IP addresses
to look for the server.
- Asked which coding models an agent can use, the agent got the
runtime's model list in its first minute. The catalog did not say how to
list OpenRouter models or that these lists answer the question, so it
enumerated OpenRouter with the shell until the deadline.
- Asked to raise a budget, the agent got the final board-only denial.
The chat guidance said "Complete other permitted work", so it probed
approvals and other routes and replied too late.

**Expected behavior**
The answer the agent receives says whether it is final and what the
person must do, with a link when the page is known. The agent replies at
once. It does not search other routes, retry, or contact other network
addresses.

**Steps to reproduce**
1. Open Agent Chat with a CEO agent.
2. Ask it to open a deleted task by its old link, or to connect an MCP
server at a closed local port.
3. The agent calls `GET /api/issues/{id}` (404) or `POST
/api/tool-connections/{id}/health-check` (502 "fetch failed").
4. The body says nothing about finality. The agent keeps searching or
probing and replies late or not at all.

## What Changed

- `GET /api/issues/:id`: for an agent, the 404 body adds `code:
"issue_not_visible"` and a `remediation`. It says that the lookup is
final, that the agent must not search other routes or open a similarly
named task, and that it should reply now and offer sharing or pasting. A
missing task, a private task and another company's task get the same
body, so the body reveals nothing. People and other routes keep the
plain body.
- MCP health check and catalog refresh: when Paperclip cannot reach the
server (connection refused, DNS, network, reset, timeout or TLS), an
agent's error adds a `remediation`. It says the person must correct the
URL or start the server, that the agent must not contact other hosts,
ports or network addresses, and it links Connectors (`/<prefix>/apps`).
Status, `error` and `code` do not change.
- `AGENT_CHAT_DIRECTIVE`: after a board-only denial, the agent replies
now instead of making more calls. "Complete other permitted work" is
removed.
- API catalog: `POST /api/companies/{companyId}/tools/connections` says
it saves without a check and points MCP-by-URL setup at
`tools/apps/connect`, which checks the server first and saves nothing
for a fresh setup that cannot reach it. `GET
/api/companies/{companyId}/adapters/{type}/models` says which lists
answer a model question (the runtime's list with `provider`, `acpxAgent`
for ACPX agents and `environmentId`, and `opencode_local` with
`provider=openrouter` for OpenRouter keys), that the OpenRouter list is
not a qualification, and that with those inputs these calls answer the
question. `openrouter` is added to the documented `provider` values.
- API catalog authorization: budget policy upsert, budget incident
resolution, company and agent budget updates, and approval
approve/reject are now listed as board-only. Their routes already call
`assertBoard`, but the catalog said `board_or_agent`, so an agent
explored them as its own action before it got the 403.
- Tests: new `agent-refusal-finality.test.ts` (including the catalog
actor for the budget and approval routes); the private-task access test
now checks that a private and a missing task give an agent the same 404
body; the guidance test checks the new wording.

## Verification

- `cd server && npx vitest run
src/__tests__/agent-refusal-finality.test.ts
src/__tests__/board-access-denial.test.ts
src/__tests__/issue-access-grants-routes.test.ts
src/__tests__/authz-existence-oracle-guard.test.ts`: all pass (embedded
Postgres included).
- `cd server && npx vitest run src/__tests__/error-handler.test.ts
src/__tests__/generic-mcp-connection.test.ts
src/__tests__/openapi-routes.test.ts
src/__tests__/remote-http-rebinding.test.ts
src/__tests__/cloud-portfolio-error.test.ts
src/__tests__/database-backup-catalog-guidance.test.ts`: 183 tests pass.
- `npx tsc --noEmit -p server`: passes.
- Real chat journeys on the chat evaluation branch (runner-opencode):
the results will follow in a comment.

## Risks

- Low risk. HTTP status, `error` and `code` stay the same for each
changed error. Agents get extra top-level fields (`code` on the task
404, `remediation`). A client that compares a whole agent 404 body for
`GET /api/issues/:id` exactly will see a change.
- The task 404 body is the same for missing, private and other-company
tasks, so it adds no existence oracle. Other issue routes keep their
plain body.
- The guidance change can make an agent stop after a board-only denial
in a request that had other, permitted parts. The agent still reports
what it finished.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 04:30:19 -06:00
Devin FoleyandPaperclip 0e49576fd9 fix(github): default repository owner to the only installed organization (#20141)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use a managed GitHub connection to create repositories and to
fork repositories
> - The Paperclip GitHub App can be installed only on an organization,
and not on the user's personal account
> - Then `create_repository` and `fork_repository` fail when
`organization` is empty, because GitHub sends them to the personal
account
> - Agents, approval cards, and connection tests all fail on their first
attempt with `403 Resource not accessible by integration`
> - This pull request fills an empty `organization` with the only
organization where the App is installed
> - The benefit is that repository creation works on the first attempt
for these connections

## Linked Issues or Issue Description

No public issue exists. Refs #19655 (GitHub repair guidance for the same
error).

**What happened?**
A user connected GitHub through the managed connection. The Paperclip
GitHub App was installed on one organization and not on the user's
personal account. An agent called `create_repository` without
`organization`. GitHub returned `POST https://api.github.com/user/repos:
403 Resource not accessible by integration`. The test button on the
connection page gave the same error. The agent then stopped and asked
the user to create the repository manually.

**Expected behavior**
When the App can create repositories only in one organization, an empty
`organization` uses that organization. The approval card and the test
result show that owner.

**Steps to reproduce**
1. Install the Paperclip GitHub App on one organization only. Do not
install it on the personal account.
2. Connect GitHub with the managed connection.
3. Run `create_repository` with only `name`, from an agent or from the
connection test dialog.
4. GitHub returns 403.

## What Changed

- Added `server/src/services/tool-access/github-default-owner.ts`. It
finds the default organization from the grant's access snapshot. It
returns an organization only when there is exactly one installation, and
the snapshot marks that owner's repositories as organization
repositories. Older snapshots without account types get no default until
access is refreshed.
- `governedToolArguments` in `tool-gateway.ts` fills an empty
`organization` for `create_repository` and `fork_repository` on managed
GitHub connections. The change occurs before policy, approval signing,
previews, and audit. Thus agent calls, approval cards, and connection
tests use the same owner.
- The grant lookup runs only for these two tools, and only when
`organization` is empty. If the credential is unavailable, the call does
not change and fails with its own error at dispatch.
- An explicit owner does not change. A fork into its own owner does not
change. Token (PAT) connections, personal installations, and multiple
organizations keep GitHub's behavior.
- `projectedConnectionToolInputSchema` describes the default on the
`organization` argument. Agents and the connection test form see where
the repository goes.
- Updated `doc/connections/GITHUB.md`.

## Verification

- `pnpm --filter @paperclipai/server typecheck` passes.
- `npx vitest run
src/services/tool-access/github-default-owner.test.ts`: new unit tests
for the owner rules (single organization, personal installation, several
organizations, another person's account, older snapshots, blank values,
forks into the same owner, schema text).
- New gateway test in `src/__tests__/tool-gateway.test.ts`: "defaults
GitHub repository creation to the only installed organization for agents
and connection tests". It covers an agent call, an explicit owner, a
connection test call, and a personal installation. It fails without the
gateway change.
- Related suites pass, 393 tests in 9 files: `tool-gateway.test.ts`,
`tool-gateway-service.test.ts`, `connection-instructions.test.ts`,
`github-access-errors.test.ts`, `github-default-owner.test.ts`,
`tool-gateway-discovery-http.test.ts`,
`github-operation-credentials.test.ts`,
`project-git-authorization.test.ts`, `prompt-injection-gateway.test.ts`.
- Not tested against a live GitHub App installation.

## Risks

- Low risk. The default applies only when the grant snapshot shows one
organization installation and no personal installation. Without the
change, these calls fail with 403.
- A stale access snapshot can choose the organization after a user
installs the App on the personal account. Installation events and
**Refresh access** update the snapshot.
- An approval that was created before this change, without
`organization`, no longer matches the projected arguments. That approval
cannot run and must be requested again. Without the change, it fails
with 403.
- `gh repo create` and `gh repo fork` in agent shells do not use the
gateway, so they keep GitHub's personal default.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code reading, shell commands, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-08 17:08:13 -08:00
DottaandPaperclip b94963f91e fix: retry temporary repository catalog failures before dispatch (#19690)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - A task must check repository access before it starts agent work.
> - Repository discovery can fail while the user's existing grant
remains valid.
> - Discovery previously replaced a known temporary provider refusal
with an empty catalog.
> - The task then reported lost access and did not schedule recovery.
> - This change keeps temporary discovery failures distinct and uses the
existing bounded retry owner.
> - Each retry must pass a fresh access check before it can use the
saved repository.

## Linked Issues or Issue Description

**What happened?**

A task with a saved catalog repository stops with an access denial when
GitHub returns an exhausted-quota response or HTTP 503. The task does
not start provider work, but the discovery layer discards the temporary
failure and returns an empty catalog.

**Expected behavior**

Known temporary failures before provider dispatch use the existing
bounded infrastructure retry. The successor checks the same responsible
user's current grants and reads the saved files only after fresh
discovery succeeds. A completed denial remains terminal.

**Steps to reproduce**

1. Bind a task to an authorized catalog repository with saved local
changes.
2. Make its authorized GitHub discovery return HTTP 403 with zero
remaining quota, or HTTP 503.
3. Let discovery recover before the existing retry deadline.
4. Observe one successful successor using the original repository bytes.

**Paperclip version or commit**

Reproduced on master at `b8d94b3ad1`. A search found no existing PR for
this temporary catalog failure.

## What Changed

- Keep a safe typed failure for known HTTP availability and quota
responses. Honor the later provider deadline and bound it to one day.
- Propagate that failure only when fresh discovery cannot find a
required repository. Preserve successful partial catalogs and the
existing connection setup error contract.
- Classify the failure only before provider dispatch. Reuse existing
bounded retry, interaction ownership, and bootstrap settlement rules.
- Keep current grant checks, saved repository selection, normal Stop,
and provider-started failure policy.
- Keep provider deadlines when accepted interactions use their own retry
owner. Recognize secondary HTTP 403 rate limits with valid retry advice
while keeping ordinary permission denials terminal. This matches [GitHub
rate-limit
guidance](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api).
- Keep direct repository request responses outside the automatic
bootstrap retry contract. Test successful partial discovery and
missing-required-repository recovery with two independent eligible
connections.
- Document task repository discovery recovery.

## Verification

- Behavioral RED: real heartbeat and PostgreSQL fixtures reached actual
HTTP 403 and503 discovery responses, started no adapter, and found zero
successors instead of one. Both cases failed before the production
change.
- Behavioral GREEN: both cases waited for the real retry deadline, used
the normal queue worker, ran the controlled adapter once after fresh
discovery, and read the original saved bytes. Two cases passed in 74.73
seconds. No external model or provider request was made.
- Deadline RED/GREEN: a combined Retry-After and quota-reset response
initially selected the earlier deadline. The regression now selects the
later deadline.
- Initial repair at `d8d4714a18` (same production source as
`018e8c52d345d79972d7907468080dc9908c299f`): seven affected files
passed, 69 cases in 185.16 seconds. That test-only cleanup then passed
all five real-database acceptance cases in 134.76 seconds.
- Commands: `pnpm exec vitest run` with
`heartbeat-repository-catalog-recovery.test.ts`,
`github-catalog-timing.test.ts`, `github-grant-metadata.test.ts`,
`project-repositories-persistence.test.ts`,
`task-workspace-repositories-persistence.test.ts`,
`github-repository-catalog-failure.test.ts`, and
`services/heartbeat/retries.test.ts` from the server source tree. The
acceptance file also ran fresh at `018e8c52d3`. A fresh five-case run
after the P1 repairs passed at `f490ca6b78` in 122.06 seconds. The final
eight-case file passed fresh at
`e5b2efab11919467850f55fd6345317132b9b672` in 124.62 seconds, using real
owned embedded PostgreSQL. It covers automatic recovery with saved
bytes, genuine denial, provider-started classification, normal Stop, the
direct HTTP route, and both partial-catalog outcomes.
- Review repairs at `f490ca6b787cf77df96d3d6edfb2ab9115eb19a6`: both
reported P1s reproduced RED. The real database interaction scheduler
chose a retry after 24 seconds before the provider deadline of one hour.
A secondary HTTP 403 with valid Retry-After lost its retry
classification. The narrow fixes then passed 44 cases across scheduler,
header, and metadata-contract files in 37.66 seconds.
- A real direct request through the task workspace router and normal
error middleware reproduced a 500 response instead of the existing 403.
Required-ID retry mode now applies only to admission and its rechecks.
The route and both two-connection partial-catalog cases then passed in
29.09 seconds. Both partial cases use actual owned grants and credential
bindings.
- All seven server type layers passed again at final head
`e5b2efab11919467850f55fd6345317132b9b672`. The affected test type layer
passed again after the test-only cleanup. Environment: macOS arm64, Node
26.4.0, pinned pnpm 9.15.4, Vitest 5.0.3, owned disposable PostgreSQL 18
on loopback. The current checkout's shared packages, plugin SDK, and
Runner were built first.
- One local acceptance run under concurrent owned test/typecheck load
hit the unchanged 15-second Stop-case timeout. Its four other cases
passed. The unchanged complete file then passed serially. The timeout
receipt remains recorded; no assertion or timeout was weakened.
- A provider-started failure retains the existing generic adapter retry
policy. It does not receive the catalog no-provider certificate or a
provider retry deadline. Fixture cleanup cancels its scheduled retry
through the normal owner.
- Final head `e5b2efab11919467850f55fd6345317132b9b672` has 59
successful checks, 3 allowed skips, no pending or failed check, and
Greptile 5/5 with no open findings. [Completed CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/42322202977).
- Additional local diagnostic for the late recovery-owner advisory: on
unchanged `e5b2efab`, the dedicated scheduler returned `retry_exhausted`
with attempt 3/max 2. The normal stranded sweeper then escalated once,
created no extra run, and started no adapter, even after catalog access
recovered. One case passed in 17.32 seconds. The existing
failed-predecessor path delegates to that bounded scheduler and returns;
it does not enter a new generic retry budget. No production registry
change was made. This diagnostic was not a CI case. Its local harness is
preserved separately at `74e3854eea3f7c4121fdcd9c8a8ad522cd8c7477` on
`codex/catalog-exhaustion-acceptance`; its production source is
identical to `e5b2efab`.
- Module boundary and whitespace checks passed.

## Risks

- A temporary failure from another eligible connection can delay a truly
missing repository for the existing limit of two attempts. It cannot
grant access.
- Unknown network errors and genuine access denials retain their
existing behavior. This change handles known HTTP failures only.
- Historical provider quota causality is unproven. This repair has a
separate local producer regression. It does not rewrite past failures or
queued user intents.
- Acceptance uses real local database and heartbeat owners with
controlled HTTP and adapter fixtures. It does not claim a live
native-provider recovery.

## Model Used

OpenAI GPT-6.1 Codex (`gpt-6.1-sol`), with high reasoning, repository
tools, code execution, and controlled local PostgreSQL tests. The host
did not expose a context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 19:29:40 -06:00
Devin FoleyandPaperclip 2a4a001d37 Preserve GitHub access repair guidance for MCP callers (#19655)
Return GitHub permission denials as failed MCP tool results with trusted,
connection-specific recovery guidance. Explain the personal-account and
organization App installation boundary without replaying writes or bypassing
provider-content checks. Align the repair steps with the Permissions UI.

Cover enabled model checks and malicious provider failures across direct
calls, connection tests and stored approvals. Repair the CI-exposed synthetic
onboarding environment fixture without weakening cancellation assertions.

Verified 311 targeted tests, six onboarding Playwright tests, server
typecheck, module boundaries, full CI and Apex 5/5 with all threads resolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 16:32:27 -08:00
2e35730637 feat(chat): receive sealed Slack events through the current SDK owner (#19019)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Slack connectors turn authenticated messages into company work.
> - Direct callbacks need an awake tenant.
> - Cloud can retain authenticated asynchronous events until the tenant
wakes.
> - Replay must preserve the original callback proof and current
endpoint authority.
> - Configuration changes must retain registration responsibility
through failures.
> - This PR enrolls verified Slack identities and drains sealed events
through the normal SDK owner.

## Linked Issues or Issue Description

**Subsystem affected**

Server chat connectors and signed Cloud connector delivery.

**Problem or motivation**

Direct Slack Events API delivery depends on an awake tenant. Broker
outages, lost responses and configuration changes must preserve delivery
and registration responsibility. Enrollment failures must also leave
other connector events able to drain.

**Proposed solution**

Stage nonsecret enrollment or retirement intent inside the local
configuration transaction. Let the existing connection-event owner
perform broker requests and retries. Replay sealed events through the
current Slack SDK with their original authentication evidence.

**Alternatives considered**

An early acknowledgment for every callback changes command and modal
semantics. A second delivery ledger duplicates existing ownership.
Reusing a new stack identity to retire old registrations crosses an
authorization boundary.

A companion Cloud broker change is required. Related setup fixes: Refs
#18837, Refs #18878. Kickoff searches found no competing async
enrollment consumer. The ROADMAP's connector setup and access controls
stay in place.

## What Changed

- Persist enrollment and credential-free retirement intent before local
configuration commits. Broker failure does not make a completed pause or
removal look unsuccessful.
- Bind company, endpoint, connection, public ID, generation and
authorized actor. Preserve lost-response responsibility and recover only
the supported same-instance configuration outage.
- Permit an explicit move to a new authorized stack only after confirmed
old retirement. Keep the old identity until the new broker receipt
matches the current intent and generation.
- Show permanent signing-credential failures as attention. Preserve
transient retries and reject uncertain inbound authentication before SDK
admission.
- Drain events and acknowledgments before enrollment reconciliation.
Persist a bounded retry on the exact busy intent so other due endpoints
can enroll or retire. Preserve current custody under concurrent changes.
- Stage retirement in the Apps removal transaction too. Archive the
endpoint and connection, retain the original subject and instance, then
revoke credentials. Retirement does not need the deleted Slack
credential.
- Acknowledge a malformed sealed Slack receipt as invalid once. Keep
actual broker outages on the retry path.
- Run the real configuration, SDK ingress, generation fences and public
Apps removal paths in the default SQL suite. Keep Vitest transport
helpers outside production services.
- Preserve original callback bytes, signature, time and public URL
evidence. Use the existing transactional chat-delivery ledger for
redelivery.
- Provide a checked-in prewire guard and launcher using the current Node
executable. Add broker, worker, SDK, SQL and controlled-process
acceptance.
- Stabilize AI account summaries by ID and grant ID. This fixes a CI
migration-repeat equality failure without changing SQL, authorization or
reconnect behavior.
- Open the four configured database connections before the enrollment
test freezes its clock. Keep all 35 original assertions and all
timeouts.
- Measure status-card dispatch entry before its mock does database work.
Keep the original strict 500 ms bound and check any still-armed earlier
deadline. Bound that fixture's pool to four connections.

## Verification

- Candidate: `54fa701d41b4ced62f4ff338d39b946015670ae3`. Normal merges
preserve master `47027e7` and all owned changes. A read-only merge
assessment against fetched master `cb5d7e44` is conflict-free. The only
overlapping file has separate AI test additions. The branch has not
merged that later master; fresh remote checks must validate the current
PR test tree.
- At this head, normal server production typechecking and declaration
emission passed in 35.24s. The affected actual CI test shard passed in
6.27s. Other unchanged CI layers, 92 affected pure controls and 12 guard
controls retain their source-matched results.
- All **15 default real-PostgreSQL cases passed in 34.66s**, with no
skips or filtering. They cover normal configure and SDK ingress,
rollback, stale generations, malformed receipts, busy-lease fairness,
concurrent intent custody, unknown errors, and public Apps removal.
Removal tests check actual credential revocation, retained retirement
through outage and repeat removal, one ordinary retry and confirmed
retirement, transaction rollback and foreign-company refusal.
- The public removal counterfactual failed before the fix: archival and
credential revocation passed, but the inactive intent was absent. The
same test source passed after the staging call. Separate counterfactuals
also reproduced busy-endpoint starvation and malformed-receipt retries.
Original failures remain retained.
- The paired broker/SDK/process journey passed in 31.98s at ancestor
`0324c011`. It preserves callback proof, lost-ACK redelivery, exactly
one comment, one delivery and one successful controlled process. Its
test, producer, SDK, guard and runtime bytes are unchanged. The new
public Apps archive branch is not called by that journey; its evidence
carries on that checked basis.
- The new SQL cases need no companion Cloud checkout. Normal CI runs
them with the repository fixture. For an existing dedicated PostgreSQL
fixture, set `PAPERCLIP_SLACK_OWNER_TEST_DATABASE_URL` and
`PAPERCLIP_TEST_POSTGRES_ADMIN_URL`, then run `node
server/scripts/test-slack-cloud-owner.mjs` with Node24. The optional
paired launcher still requires the normally built companion Cloud
checkout.
- Every pool was bounded to four or fewer connections. The final 15-case
run sampled three runtime clients plus one observer. The independent
final audit found zero clients and all five recorded processes absent.
Earlier closing-socket observations are kept separate from final
cleanup.
- Earlier unrelated CI failures are documented: [enrollment
timeout](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483754)
and [status-card observation
bound](https://github.com/paperclipai/paperclip/actions/runs/41944373937/job/126973483843).
The enrollment test retains all 35 assertions and its original timeouts.
A controlled connection probe established the frozen-clock mechanism;
the original remote access history remains unknown. The status-card
fixture measures dispatch entry with the original strict 500ms bound. A
real delayed-dispatch control fails that bound.
- Full local workspace typecheck, build and test inventory was not run
because its embedded-cluster and wider resource paths exceed this local
lease. Exact-head remote CI now passes: 54 successful checks, three
skips and Ponytail's neutral check, with aggregate verification green
and the legacy security context independently confirmed successful. Both
complete bot reviews are 5/5, with all 13 paginated review threads
resolved. Seven execution jobs originally reported runner shutdown; one
diagnostic rerun recovered them and the dependent aggregate check.
Original logs remain retained, and the underlying shutdown cause is
unknown. This is recovery evidence, not a product-fix claim.
- Provider responses are controlled. This proves local SQL, broker, SDK
and process behavior. It does not prove live Slack, agent tools,
synchronous callbacks or deployed suspension.

## Risks

The companion broker is required. Outage or conflicting authority can
leave explicit pending or attention responsibility. A move between
stacks requires confirmation from the old authorized owner first.
Synchronous commands, actions, modal validation, options and challenges
retain their deadlines and endpoint blocker. This slice grants no full
Slack endpoint sleep exemption. No deployed suspension or savings claim
is made.

## Model Used

OpenAI `gpt-6.1-sol` workers through Codex, with reasoning, repository
tools and code execution. The served backend revision and context limit
are not exposed. A GPT-6-family coordinator performed technical review
and integration.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-11-06 15:22:51 -06:00
Devin FoleyandPaperclip 8f79a32898 fix(connections): keep agent access consistent through setup and permission changes (#19352)
Task-hosted tool setup kept requester-only defaults, and audience edits could leave runtime installations restricted after permission bindings were widened. Large provider catalogs could also fail before creating an access card.

Use the normal agent audience defaults for new tool setup, preserve explicit selections through verified OAuth and draft resumption, and synchronize permission bindings and runtime installations atomically. Enforce agent-edit authority and preserve existing access during additive account reuse.

Validation: 915 focused regression tests, typechecks, and builds passed. All 54 CI checks passed and Greptile Apex scored the reviewed head 5/5 with no unresolved threads. Broad local test failures, recoveries, and incomplete heartbeat coverage are documented in the PR.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-05 17:38:32 -08:00
b929ad43de fix: keep dependency waits neutral and explain GitHub access failures (#19315)
## Thinking Path

> - Paperclip manages agent work and shows users what can move it
forward.
> - Task status and tool approval cards describe different parts of that
work.
> - A queued dependency wait has no provider attempt to retry.
> - GitHub can reject an approved action while other connection tools
still work.
> - These cases showed a false Retry banner and an unhelpful provider
error.
> - This pull request keeps dependency waits neutral and gives GitHub
access failures a direct repair path.

## Linked Issues or Issue Description

Related: #19268 preserves pending Retry preparation and current failure
causes.

**What happened?**

A queued wake cancelled by the dependency gate appeared as a failed
attempt with Retry. A GitHub MCP permission denial appeared as raw HTTP
error text after an approved repository action.

**Expected behavior**

Keep the task's dependency state. Show GitHub access failures with a
clear explanation and a link to the selected connection's Permissions
page. Preserve actual failures, provider details, and approval gates.

**Steps to reproduce**

1. Queue a wake, then leave its task dependent on unfinished work before
dispatch.
2. Open the task after the dependency gate cancels that wake.
3. Separately approve a GitHub action whose MCP result contains `403
Resource not accessible by integration`.
4. Reopen the saved tool approval.

## What Changed

- Recognize both queues' unstarted dependency-wait receipts in
cancellation provenance and task projections. No dispatch or recovery
gate changes.
- Classify GitHub's specific permission error after MCP result
normalization. Keep result validation, the raw diagnostic, and working
tools available.
- Apply the same display code to both historical GitHub error codes.
Identify the provider from company-scoped connection metadata, including
renamed apps, without rewriting stored decisions.
- Show a token-based inline access card with **Manage GitHub access**
and collapsed **Error details**.
- Give agents trusted recovery steps after an approved GitHub action
fails. Keep provider text in the untrusted result and prevent automatic
replay.
- Add regressions, a production-component Storybook case, and
execution/connector documentation.

## Verification

- Red before the fix: cancellation provenance, dependency-wait
projection, the real queue cancellation, approved MCP failure
classification, saved interaction projection, and the error card
assertions failed at the expected behavior.
- Green focused tests: queue cancellation, one ready-task executor,
approval-before-dispatch, one failed GitHub request, connection reuse,
retained provider diagnostics, saved interaction reads, task list/detail
projection and redaction.
- UI: 106 component tests passed. Token gates and Storybook catalog
checks passed.
- `node scripts/typecheck-low-memory.mjs`: passed.
- Local source instance: verified a saved dependency wait stays neutral
after reload; a saved GitHub error shows repair guidance, expands its
original error, and opens the correct connection Permissions page.
PostgreSQL and HTTP APIs were real. The GitHub denial was a controlled
MCP fixture; no account or provider permission changed.
- `pnpm build`: passed for all workspace packages, the Runner binary,
server, CLI and UI. The server build and all server typecheck layers
also passed after the review repair.
- Additional safety checks passed for realistic repository arguments,
active recovery, failed Retry preparation, and bounded provider
diagnostics.
- Review repair: 45 pure tests and 7 PostgreSQL cases passed. These
cover both historical error codes, renamed apps, and
company/provider/application identity boundaries. A second local browser
check verified the renamed app, retained error detail, and correct
Permissions destination.
- Final review regressions: the dispatcher and metadata cases failed
before the repair. Afterward, 25 focused cases and 59
projection/identity cases passed, including connection-key precedence.
The native wait and app-only GitHub source also passed a local browser
check.
- The historical read adds at most one connection query per interaction
read batch. It selects IDs and the provider key only and runs only for
candidate permission failures. It adds no provider requests.
- Full local `pnpm test:run` stopped at the chat suite's `afterAll`
cleanup timeout. All 268 assertions in that group and the three prior
chat groups passed. The failure is retained here; it is not counted as a
full-suite pass. CI passed on `e8a692e1e2` after one workspace-startup
rerun. Fresh CI is pending for the combined head. Its first workspace-a
attempt timed out in the CLI company import/export test's child-server
startup wait; 4,968 UI tests and 287 CLI tests passed in that job. The
same shard passed on the preceding commit. The workspace shard passed on
its one rerun. All other test, build and browser jobs passed. The [first
failure](https://github.com/paperclipai/paperclip/actions/runs/42093823928/job/127478869941)
is retained in the Actions history. Greptile reviewed the preceding head
at 5/5. Fresh review is pending for the combined head.

## Risks

The GitHub classifier matches the documented permission-denial phrase,
not every 403. Rate limits, missing resources and other providers retain
their existing error behavior. Historical normalization affects display
only. A dependency cancellation needs an unstarted run and a supported
queue receipt; current user gates, Retry preparation and recovery remain
authoritative. No schema migration or credential changes.

## Model Used

OpenAI GPT-6, through the Codex agent. The exact runtime model ID and
context-window size are not exposed in this session. Used reasoning,
repository tools, TypeScript, PostgreSQL, MCP fixtures and browser
verification. The parallel agent-recovery commit records Anthropic
Claude Opus 5.5 as co-author.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with version and capability
details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the bug template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change
- [x] I have run the focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open findings
- [x] I will address reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 20:55:34 +00:00
DottaandPaperclip f187de99aa Preserve domain deadlines across owned idle sleep (#18656)
Reviewed and verified at 9b4acc456d.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 15:40:56 -06:00
DottaandPaperclip 8c2502503d fix(github): preserve repository access during PR webhooks (#18805)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Agents need current repository access before they can start work.
> - Managed GitHub access checks run while webhook delivery can update
the same grant.
> - Pull request events update webhook health but do not change
repository permissions.
> - The refresh guard treated those health updates as access changes and
could exhaust its retry.
> - This change keeps repository preparation available during normal
pull request activity.

## Linked Issues or Issue Description

**What happened?**

A pull request webhook can make a managed GitHub repository refresh fail
with `github_access_changed`. Repository discovery then omits the failed
connection. Workspace admission can report that repository access is no
longer available, even after the checkout succeeds.

**Expected behavior**

Pull request activity must preserve valid repository access.
Installation changes and newer access snapshots must still stop a stale
refresh.

**Steps to reproduce**

1. Start a managed GitHub access refresh.
2. Process a pull request edit while the provider request is in flight.
3. Repeat on the bounded retry. The old guard rejects both checks
because `lastWebhookAt` changes.

## What Changed

- Use `accessRevision` and `lastAccessRefreshAt` as the existing access
fences. Stop using webhook delivery time as an access fence.
- Allow the existing commit merge to retain the latest webhook timestamp
and health after a pull request event.
- Exercise real webhook processing and PostgreSQL during refresh. Cover
a pull request event, an access conflict followed by that event, and
installation suspension.
- Document the distinction between access changes and delivery health.

## Verification

- Red: both pull request race cases failed on the old code with
`github_access_changed`.
- Green: all seven focused refresh cases pass. These include the
existing concurrent access-change checks and the new suspension case.
- Command: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t 'binds a managed
GitHub identity'`.
- Full affected group: 526 tests pass across the tool-access,
repository, and webhook suites.
- `pnpm -r typecheck` and `pnpm build`: pass.
- Current-head CI: 54 checks pass. The duplicate Paperclip review is
neutral. Two Storybook checks are skipped. Greptile and the completed
Paperclip review both rate this commit 5/5 with no findings. There are
no unresolved review threads.
- The duplicate local `pnpm test:run` did not complete. Its first four
chat shards passed. It was stopped after CI completed on this commit;
the full CI test matrix passed. The complete affected local group passed
separately.
- Retained earlier failure: the first affected-group attempt passed 525
tests and had one socket hang-up in the profile attention feed test.
That test passed alone without changes. The unchanged base tool-access
suite also passed all 498 tests, and the complete fixed affected group
then passed. The cause of the socket closure remains unproven.
- Environment: source checkout on macOS arm64, Node 26.4.0, pnpm 9.15.4,
disposable PostgreSQL. Provider responses and Cloud transport use
controlled fixtures. The webhook handler, refresh logic, transactions,
and saved grant reads are real.
- Live acceptance remains unverified. The available CLI token cannot
list GitHub App installations. The affected staging deployment still
serves the old code. This PR does not deploy or recover the original
task.

## Risks

- Removing an access fence could admit stale permission data. The access
revision and refresh timestamp checks remain. The real suspension
webhook regression confirms that stale provider data cannot restore
removed access.
- This change adds no retry, timer, schema change, or credential
forwarding.

## Model Used

OpenAI Codex, based on GPT-6. The exact runtime model ID and context
window are not exposed in this session. Used reasoning, code editing,
and terminal tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 11:15:49 -06:00
DottaandPaperclip 58308a3e63 perf(apps): scope sidebar review counts to one connection (#18688)
## Thinking Path

> - Paperclip helps people manage AI agents and their governed app
access.
> - The Apps sidebar displays a Review badge for one connection.
> - It polls the full company attention service every 30 seconds to get
two counts.
> - That service also loads every profile, rule and active catalog
entry.
> - This pull request adds an authorized connection read for only those
counts.
> - The same badge now avoids unrelated profile evaluation and database
result decoding.

## Linked Issues or Issue Description

**Current behavior**

The contextual sidebar polls company-wide attention. It needs only one
connection's signed pending action count and whether any catalog entries
are quarantined.

**Proposed behavior**

Both sidebar variants poll a connection-scoped count endpoint with
existing connection read authority. The full attention API stays
unchanged.

**Reason and benefit**

The retained 200-profile fixture performs seven service queries and
returns 80,115 rows for this badge. The new reader performs one query
and returns one row. This reduces local service CPU and avoids
unnecessary decoded profile/rule data. Related merged evaluator fixes
are already in the baseline: Refs #18673, Refs #18681. A search found no
existing PR for this scoped count reader. ROADMAP.md contains no
duplicate planned work.

## What Changed

- Add the narrow shared DTO, typed client, guarded REST/OpenAPI route
and catalog-owned reader.
- Count independent aggregates with both company boundaries and exact
signed-argument semantics. Keep archived counts at zero.
- Update both contextual sidebar variants. Keep the exact badge, 30
second interval and existing attention-prefix invalidation.
- Add PostgreSQL differential tests, authenticated HTTP tests, sidebar
tests, a repeatable benchmark and access documentation.

## Verification

- Baseline: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`, including both
merged evaluator fixes. Service treatment: `0827f7d0b3`; route/fixture
refinement: `230c42ee15`; final local existence gate: `390a218254`.
- Passed: focused PostgreSQL differential test; assembled HTTP test with
board session and agent JWT; 14 sidebar component tests; token gates;
map checks; diff check.
- Fixture: 200 profiles, 76,000 rules, 1,900 active tools, 695 removed
references, 12 agents, a sibling connection and 2,000 unrelated
historical invocations. Complete normalized two-count results are equal
on both paths.
- Quiet local measurement: seven alternating warm pairs after two warmup
pairs, Node 24.22.2, macOS, real disposable PostgreSQL. Legacy/new
service CPU median: 213.891/1.680 ms; ranges:
209.329–228.552/0.515–6.137 ms. Wall median: 156.867/1.248 ms; ranges:
151.739–162.289/1.115–1.386 ms. Queries: 7/1. Returned rows: 80,115/1.
- Run `cd server && node --expose-gc --import
./node_modules/tsx/dist/loader.mjs
scripts/benchmark-connection-review-counts.ts`. See
`doc/connection-review-counts.md` for scope and failure history.
Byte-volume accounting runs outside timing; no HTTP payload reduction is
claimed.
- Initial raw scalar SQL failed with ambiguous columns. Typed subqueries
fixed it and passed real PostgreSQL equality. Initial fixture agents
lacked permissions and correctly received 403. The final fixture grants
explicit authority and tests revocation. An initial benchmark included
byte serialization in timing; the reported run excludes it.
- Passed required local `pnpm -r typecheck` and `pnpm build`. The
initial sandbox typecheck could not open the tsx IPC socket; the
escalated rerun passed. Full monolithic `pnpm test:run` was not run
locally: repeating a 15–25 minute suite serially across both performance
PRs adds no distinct evidence. Broad suite acceptance uses final-head
CI, which runs the same general/serialized test partitions and the
native Runner lane. Final-head CI is in progress.
- Passed built-static Chromium journey on an owned local-trusted server
and disposable PostgreSQL: Review 3 appears, only scoped counts are
requested by this sidebar, resolving requests/clearing quarantines then
reloading removes the badge, and archived counts are zero. The first ad
hoc launcher omitted required decision-service options and failed at
startup; using the normal no-op wake callback fixed that harness.
- Greptile identified an invalid resolved-request fixture label.
Corrected `denied` to the declared `rejected`, then reran the real
PostgreSQL differential test. The count SQL and non-pending exclusion
are unchanged.
- The first final-head CI failed the static cross-tenant
existence-oracle guard because the new explicit resource company
assertion had no nearby `hasCompanyAccess` gate. Existing management
admission already rejects foreign resources with 404. The route now also
follows the standard local two-step gate. The unchanged guard plus
authenticated HTTP negative cases passed (3 tests). No allowlist or test
was weakened.
- These local service measurements do not prove hosted performance. No
deployment is included.

## Risks

Low risk. New read admission reuses existing current-authority checks.
The old attention API is unchanged. Counts still require database work
over relevant pending/catalog rows. No cache, schema migration or
telemetry is added. Roll back this PR if the badge differs or scoped
reads deny previously permitted users.

## Model Used

OpenAI `gpt-6.1-sol` (Sol 6.1), with reasoning, tool use and code
execution. The host does not expose the configured context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 17:12:22 -06:00
DottaandPaperclip 554a770c6f perf(apps): scope connection profile configuration reads (#18687)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Apps Permissions page reads saved access profiles for one
connection.
> - The page fetched every company profile, rule, and binding.
> - It used only the app profile and eligible task-granted profiles for
that connection.
> - This pull request adds a saved configuration read owned by the
profile service.
> - The route keeps current connection and per-profile management
checks.
> - The benefit is fewer database rows, less Node work, and a smaller
response for the same displayed configuration.

## Linked Issues or Issue Description

**Subsystem affected**

Server profile configuration reads and the Apps Permissions and Review
data caller.

**Problem or motivation**

One connection page loaded and serialized unrelated company profiles. In
the local fixture, this produced a 32.35 MB response with 76,001 rule
rows.

**Proposed solution**

Select the displayed connection's app profile and exact eligible
connection-intent profiles before loading their entries and bindings.
Return an explicit configuration DTO with complete selected arrays. Keep
summary and pending-review work on the existing full API.

**Alternatives considered**

Filtering the full response in the browser retains the database and Node
cost. Truncating the full API would break other clients. Filtering
entries to the connection would lose selectors and bindings needed by
saves.

**Roadmap alignment**

This improves the existing governed Apps capability. It does not add a
new access policy. Related merged work: #18673 and #18681. Related
harness cleanup: #18683. Public PR searches found no duplicate
connection configuration projection.

## What Changed

- Add `ToolProfileConfiguration` and `GET
/api/tool-connections/{connectionId}/profile-configuration`.
- Select `app:<connectionId>` at any status and exact active, correctly
bound `connection_intent` profiles. Run the existing per-profile
visibility filter before configuration queries.
- Preserve complete ordered entries and bindings. Keep all save
behavior, empty-binding install fallback, and the existing full profile
API.
- Switch AppDetail and AgentConnectionAccess to the configuration DTO.
Use a distinct cache key below the existing profiles prefix so mutation
and reconnect invalidations still refresh it.
- Add real PostgreSQL differential and authenticated HTTP tests, UI
caller coverage, a reproducible benchmark, and access-map documentation.

## Verification

Baseline: `3f8ebfff8aa9febd0733c6f65b07485dcadfd0e2`, after both #18673
and #18681. Final head: `b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`.
Current implementation and reproduction details are in [the performance
note](https://github.com/paperclipai/paperclip/blob/b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22/doc/performance/2026-11-02-connection-profile-configuration.md).

- AppDetail: 79 tests passed. Existing access/action saves, install
fallback, grant removal, errors, and refresh cases use the new caller.
- Real PostgreSQL differential: passed. Full ordered selected
configuration equals the old full-list selection. Cases cover extra
bindings, sibling selectors, wrong source/key/status/binding, null and
non-string agent metadata, fresh mutations, empty app bindings,
visibility filtering, and company scope.
- Production HTTP authentication: API key and run JWT cases passed.
Hidden personal accounts omit spanning profiles. Private connection
reads fail. Restored ownership restores visibility. Revoked management
grants deny fresh calls. Final UID canonicalization and foreign-company
404 assertions passed on the current head.
- Map structural check, token gate, and `git diff --check`: passed.
- Required local `pnpm -r typecheck` and `pnpm build`: passed on the
current head. `pnpm test:run` was started once. Its first chat shard
passed (337 tests, 151.87 seconds); the second shard was intentionally
interrupted after the complete final-head CI suite passed. This partial
local attempt is not a full local test pass. The same stable-runner
general and serialized partitions and Runner lanes passed in [final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/41538805231),
including aggregate `ci / verify`. No local assertion failures were
observed before interruption.
- Final-head focused acceptance: four files, 85 tests passed. Full token
gates, maps, and module boundaries passed.
- Actual browser acceptance passed on the current checkout with isolated
real PostgreSQL and a local trusted board. An action saved as Off
persisted after reload. Both app agent bindings stayed intact. Removing
an agent-only task grant persisted after reload. Requests used only the
narrow profile endpoint. No browser exceptions occurred. No responses
were mocked. No provider calls or hosted changes were made. The helper
stopped the owned runtime and database after the check.
- Final exact-head checks: 54 success, one clean neutral Paperclip
Review, three skipped, zero pending or failures. Legacy Snyk status:
success. No inline review comments.
- Greptile reviewed exact head
`b40c1ed7c0833d403e20ee8a5ff9d09f1bad6d22`: 5/5, no findings, completed
successful check.

Five alternating unprofiled pairs ran with real PostgreSQL on
macOS/arm64, Node v24.22.2, and forced GC. The fixture has 210 profiles
(200 heavy profiles plus ten eligibility cases), 76,001 rules, 1,900
active tools, 695 removed tools, and 12 agents. The selected
configuration has two profiles and 1,391 rules. Each pair asserts
complete ordered selected DTO equality.

| Measurement | Full list | Configuration read |
| --- | --- | --- |
| Service wall ms, median [min, max] | 200.927 [193.319, 213.390] |
6.414 [5.508, 6.973] |
| Node CPU ms, median [min, max] | 262.751 [246.727, 271.573] | 27.525
[22.576, 29.308] |
| Select queries / returned rows | 7 / 78,139 | 3 / 1,397 |
| Actual response JSON bytes | 32,353,534 | 592,233 |
| Serialization wall ms, median [min, max] | 90.714 [88.715, 99.304] |
1.644 [1.637, 1.745] |
| Serialization CPU ms, median [min, max] | 90.644 [88.569, 96.778] |
1.654 [1.637, 1.807] |

Reproduce: `node --expose-gc --import
./server/node_modules/tsx/dist/loader.mjs
server/scripts/benchmark-connection-profile-configuration.ts 5`.

The benchmark measures the service and actual response JSON separately.
It excludes HTTP and authorization work through an identity visibility
callback. Node CPU does not include PostgreSQL CPU. No hosted or
foreground-under-load improvement is claimed.

Preserved failures: offline install lacked a locked tarball; frozen
online install passed. Sandbox localhost and tsx IPC restrictions
blocked initial DB/typecheck attempts; permitted retries use the same
checks. The first HTTP suite needed the unbuilt plugin SDK; its owning
build helper fixed setup. Shared typecheck caught a missing barrel
export, now fixed. A broad typecheck inherited the host's unpinned
nested pnpm executable; the owned stalled process was stopped and
restarted with pinned pnpm 9. The complete current-head CI suite and
aggregate verify passed. The separate Paperclip agent review published
5/5 with no findings at this head and completed with a clean neutral
conclusion. An added foreign-company assertion expected 403; the
existing guard correctly returned 404. Only the test expectation
changed. The first real browser attempt saved and reloaded the action
correctly, but grant deletion returned 422: the stress fixture had a
company binding, which makes that profile the company default. The UI
showed the existing deletion error. The second browser seed used the
production agent-only intent shape and passed removal and reload. This
does not change the stress fixture or product deletion gates. Initial
external browser script attempts also needed an ESM extension and the
correct dependency path. Logs, first-failure screenshots, traces, and
result files are retained in the local evidence directory.

## Risks

- SQL selection must stay aligned with connection-intent eligibility.
Differential fixtures cover the exact current predicate.
- A selected connection can still have many rules. This change preserves
all selected configuration.
- Per-profile authorization still reads current selected resources and
can add work. Connection visibility does not bypass that check.
- No migration, default-allow, effective-access, policy-review,
provider, or deployment behavior changes. The old endpoint remains
available. Rollback restores the old AppDetail caller and removes the
additive endpoint.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, with code execution and repository tools.
The host does not expose the context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 16:55:56 -06:00
DottaandPaperclip d0ffa179b4 perf(tools): index profile selectors per resolution (#18673)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Core resolves tool profiles before it prepares runtime MCP
assignments.
> - A selected profile can contain a rule for each individual tool.
> - The resolver and profile summary scan these rules for each catalog
row.
> - This repeats millions of exact comparisons for a large selected
profile.
> - This pull request indexes each selector kind once per computation.
> - The benefit is less server CPU with the same current access
decisions and result contract.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Effective tool assignment and profile summary computation for large
selected profiles.

**Current behavior**

Each catalog row scans the profile's includes and excludes. The work
grows with catalog size times rule count. The runtime assignment path
also computes a profile summary with the same scans.

**Proposed behavior**

Build five selector sets for each include or exclude group. Match
catalog rows with exact set membership. Retain all current database
reads and the full result contract.

**Reason and benefit**

Reduce repeated JavaScript comparisons. On the synthetic PostgreSQL
fixture below, median process CPU per resolution falls from 50.068 ms to
10.129 ms. This is independent of
[#18371](https://github.com/paperclipai/paperclip/pull/18371), which
removes unused administrative results and narrows instruction
descriptors. This PR retains those results and does not include that
change.

## What Changed

- Index application, connection, catalog-entry, tool-name, and risk
selectors in the profile owner.
- Use indexed matching in effective access resolution and profile
summaries.
- Add a differential test against the original single-rule matcher.
Cover null applications, exact special-character names, unknown
selectors, and empty rule groups.
- Add a PostgreSQL regression for per-profile exclusion precedence, the
union of multiple effective profiles, and current rule removal.
- Add a real PostgreSQL resolver benchmark and document reproduction and
limits.

## Verification

Baseline: `410b0e27015495ab8951768b5b0b94727de33785`. Treatment:
`895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`. Node 24.22.2 on macOS arm64.
Source modules run with `tsx`; these are local service measurements, not
production-build or hosted acceptance.

The real owning resolver ran against disposable PostgreSQL with 1,890
active catalog tools, 1,575 individual include/exclude rules, one
deny-by-default effective profile, and 12 agents. The installation
dependency returns an empty list because the fixture has no installed
connections. There are no provider calls. Full returned assignments,
profile summaries, rules, bindings, and names are identical to baseline.

After two warmups per version, seven alternating samples ran ten
resolutions each. Other workers paused CPU measurements and builds.
Unrelated host work was not controlled.

| Per resolution | Baseline median (range) | Change median (range) |
| --- | --- | --- |
| Process CPU | 50.068 ms (46.380–58.880) | 10.129 ms (8.089–11.072) |
| Elapsed time | 50.362 ms (44.799–106.408) | 8.853 ms (7.719–18.590) |
| Queries | 6 | 6 |
| Returned database rows | 3,480 | 3,480 |

Median process CPU falls 79.8%; median elapsed time falls 82.4%. The
long baseline elapsed sample had no matching process CPU increase. Query
and row counts do not decrease. Small or broadly allowed profiles can
save less. This does not establish whole-run latency or hosted
performance.

Reproduce with `server/scripts/benchmark-tool-profile-matching.ts`;
baseline module preparation is in `doc/mcp-discovery-performance.md`.

Passed on the treatment source:

```sh
node node_modules/vitest/vitest.mjs run server/src/__tests__/tool-profile-matching.test.ts server/src/__tests__/tool-effective-assignments.test.ts server/src/services/native-runtime/runtime-context.test.ts server/src/__tests__/heartbeat-runtime-mcp-servers.test.ts
```

All 45 tests in four suites passed. `git diff --check` passed. The first
PostgreSQL test attempt failed at setup because the sandbox blocked its
loopback listener (`EPERM`); the authorized unsandboxed retry passed. No
behavior failure was discarded.

At measured head `895ee2eec15cf4cdedee4c257fcdddb2a7000a1b`, CI
typecheck, build, canary, Runner checks, and all eight browser shards
passed. Server general shard 8/12 failed an unchanged
background-quietness assertion in `heartbeat-run-dispatch-work.test.ts`:
expected 235 select calls, observed 236 after advancing fake timers. Its
cause remains unverified; no baseline reproduction was run.

That head also failed the module-boundary policy because the benchmark
inserted synthetic agents outside the test fixture tree. Follow-up head
`e20165b6eb` moves the fixture body byte-for-byte into
`server/src/__tests__/helpers/tool-profile-matching-fixture.ts`,
following the existing listing benchmark pattern. The production matcher
and benchmark measurement loop are unchanged. `node
scripts/check-module-boundaries.mjs`, a Node 24/tsx fixture import
check, and `git diff --check` pass. The benchmark was not rerun for this
extraction. New-head CI and review are pending. Greptile gave the
measured head 5/5 with no actionable issues. No deployment or live
runtime mutation was performed.

## Risks

Exact equality must remain identical to the original matcher, especially
for null application selectors and exclusions. The differential and
PostgreSQL tests cover these cases. Sets add linear temporary memory in
rule count. They live only for the current computation. Current
company-scoped state reads, profile precedence, revocation, and runtime
assignment digest checks remain in place. Revert this commit to roll
back.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with source inspection, code
execution, tests, and local PostgreSQL benchmarking. The context window
size is not available in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 16:13:18 -06:00
DottaandPaperclip 482706e7dd perf(tools): index pending profile review scopes (#18681)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - The Apps UI lists tool profiles and shows new tools that need
review.
> - Both services project new-tool review for each profile.
> - The projection scans include rules and exact rules for each catalog
row.
> - This repeats millions of comparisons when many selected profiles
exist.
> - This pull request builds scope and selector indexes once per
projection.
> - The benefit is less server CPU with the same fresh review results.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

New-tool review projection in profile listings, profile review, and app
attention.

**Current behavior**

Each newer catalog row scans the profile's included scopes. Included
catalog entries require a map lookup for each scope comparison. Rows in
scope then scan all rules again to omit tools that were already
reviewed. Profile listing and app attention repeat this work for every
profile.

**Proposed behavior**

Index covered application and connection scopes once. Expand included
catalog entries to their existing connection and non-null application.
Index exact selectors across all rules once. Keep the ordered catalog
filter and returned metadata.

**Reason and benefit**

A recent main-thread profile identified this scope-matching path as a
large CPU consumer. The local reproduction below confirms multi-second
CPU work in the actual owning services. This PR removes the repeated
scans without caching authorization or changing database reads.

Related [#18673](https://github.com/paperclipai/paperclip/pull/18673)
indexes effective access and profile summaries. It leaves this
pending-review path unchanged. This PR targets the separate
pending-review mechanism and has no dependency on that PR. The selector
helper is identical so the owners can share it when both changes land.

## What Changed

- Build covered application and connection sets for included scopes.
- Build exact selector sets for all rules, including exclusions, to omit
already reviewed tools.
- Preserve null application selectors, missing catalog references,
review watermarks, status checks, output order, and names.
- Add differential tests against the former scope/matching evaluator.
- Add PostgreSQL checks through all three affected service methods,
including current catalog and rule changes.
- Add a reproducible PostgreSQL benchmark of complete profile-list and
app-attention results.

## Verification

Baseline: `1f8f2e195f4897cd1c237bdc068de95780ab0c3c`. Treatment:
`fd59c0183ce64c44696f79604516f4d51ca5662b`. Node 24.22.2 on macOS arm64.
The production projection file at this baseline is identical to the file
at `410b0e27015495ab8951768b5b0b94727de33785`.

The benchmark exercises the real
`createToolProfileService(...).listProfiles` and
`createToolAppAttention(...)` methods against disposable PostgreSQL. It
uses 200 selected profiles, 1,900 active tools, 76,000 rules, and 12
agents. Profiles alternate between 65 and 695 rules. Half reference 695
removed catalog entries; half select current tools. This tests both the
slow failed scope scans and the slow exact-rule scans. No provider calls
or hosted traffic are generated.

The complete ordered results from both services match baseline. After
one equality/warmup invocation per version, five samples alternate
implementation order. Forced GC runs before each measured pair of
service calls. Other workers paused owned builds and benchmarks;
unrelated host activity was not controlled.

| One profile listing plus one attention listing | Baseline median
(range) | Change median (range) |
| --- | --- | --- |
| Process CPU | 5,406.512 ms (5,221.533–5,648.262) | 1,770.116 ms
(1,596.232–2,045.617) |
| Elapsed time | 5,279.231 ms (4,932.419–5,537.105) | 1,532.414 ms
(1,499.199–1,610.428) |
| Queries | 14 | 14 |
| Returned database rows | 156,215 | 156,215 |

Median process CPU falls **67.3%** and median elapsed time falls
**71.0%**. Profile summaries retain their old nested matching in this
comparison, so the result isolates pending-review indexing. Query counts
and row counts do not decrease. This is local source/tsx service proof.
It does not establish hosted improvement, whole-run latency, or HTTP
tail latency.

Reproduce with `server/scripts/benchmark-profile-review-projection.ts`;
baseline-module setup is documented in
`doc/mcp-discovery-performance.md`.

Passed:

```sh
node node_modules/vitest/vitest.mjs run server/src/__tests__/tool-profile-pending-projection.test.ts server/src/__tests__/tool-profile-review-service.test.ts server/src/__tests__/tool-profile-entry-order.test.ts
node scripts/check-module-boundaries.mjs
```

All five tests in three suites passed. The differential tests cover all
selector kinds, include/exclude effects, null scope behavior, missing
references, review watermarks, and exact special-character names.
PostgreSQL checks preserve current pending counts after exclusions,
catalog removal, and review; another company remains separate. `git diff
--check` passes.

The first new integration assertion used `summary` instead of the
existing attention response's `totals` field. It failed before the
mutation checks. The test assertion was corrected; no production code
changed for that correction. The final run above passed.

Server typecheck was attempted but did not pass local setup. The
packaged command did not reach TypeScript because a global pnpm child
tried to fetch and verify the project's pnpm version through an
unavailable registry. A direct Node 24 TypeScript invocation then
reported missing built `@paperclipai/paperclip-runner` and related
package declarations, with cascading type errors in unchanged consumers.
No error points to the changed projection. The prerequisite build was
not expanded into a full workspace build. Full local unsharded tests and
build were not run; final-head CI must supply broad acceptance. No merge
or deployment was performed.

## Risks

Scope expansion must preserve existing semantics. An individual catalog
rule covers its connection and only a non-null application. An explicit
null application selector still matches null applications. Removed
references contribute no scope when absent from the supplied catalog.
The oracle and PostgreSQL tests cover these boundaries.

The indexes add linear temporary memory in rule count and live only for
one projection. Current company-scoped database reads and review
mutations remain in their owners. Exclusions remain explicitly reviewed;
they do not create new review items. Revert this commit to roll back.

## Model Used

OpenAI GPT-6.1 Sol (`gpt-6.1-sol`) with source inspection, code
execution, tests, and local PostgreSQL benchmarking. The context window
size is not available in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 16:10:42 -06:00
DottaandPaperclip eb12cfc96b perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at f77620b9a4.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 15:38:43 -05:00
DottaandPaperclip f1d7808382 fix(tools): stabilize tied profile entry order (#18427)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The Advanced YAML editor reads shared tool profiles through their
owning service.
> - The editor must preserve hidden settings and reject real concurrent
changes.
> - Batch-created entries share a creation time. Queries sorted only by
that time can return the same entries in a different order.
> - The editor can then report a false concurrent tool change before it
sends a profile binding request.
> - This pull request adds an ID tie-breaker to profile entry reads.
> - The benefit is a stable saved configuration without removing the
concurrent-change guard.

## Linked Issues or Issue Description

Refs #18356. Related: #18168.

**What happened?**

The compact YAML browser case failed at its second save. Save stayed
enabled and the profile binding write did not occur. The original CI
artifact has no response bodies. It does not establish the exact entry
order from that run.

A controlled browser test reproduced this failure. It changed only the
order of 30 equal-time entries in the second preflight response. The
client reported `tools changed while you were editing. Reload saved
before saving.` No binding request followed. A separate first-save
response barrier showed that the pending-save guard worked.

**Expected behavior**

Reads of unchanged entries must have a stable order. Real entry changes
must still trigger the existing configuration checks. Entry order does
not set policy precedence; exclusions still win over matching includes.

**Steps to reproduce**

Create a profile with entries that share one creation time. Read the
profile, change the physical row layout in a disposable PostgreSQL
database, and read it again. The old service has no tie-breaker for
these entries. The new regression fails against that service.

## What Changed

- Sort list and detail entry reads by creation time, then ID, in the
profile service. These reads supply saved configuration snapshots. Other
reads retain their existing behavior.
- Add one real PostgreSQL regression. It changes heap layout in both
directions, checks stable list and detail values, preserves
chronological ordering, checks company boundaries, and checks
include/exclude results and a real entry change.
- Include the exact test-only recovery prerequisite from commit
`36c8d9ddd76d0b7ddb28a9a6c674d9e538f6df88`. Its file blob is unchanged.
It removes a one-sweep fixture assumption while keeping bounded recovery
assertions. No document UI or journey code is imported.

## Verification

- Base: one frozen public master snapshot,
`8c9dd1e4445e58cc9ca08076dfe9c0b27038d6ee`. The affected service,
schema, projection, resolver, and save guard match the failed CI merge
`94d98d5c75c8f331aad4bfab2111ee587b0c63fe`.
- RED: the new real PostgreSQL test fails on the old service. Both
physical layouts were verified before the ordering assertion failed.
- GREEN: `cd server && pnpm exec vitest run
src/__tests__/tool-profile-entry-order.test.ts
src/__tests__/tool-profile-entry-validation.test.ts` — 18 tests pass
with real PostgreSQL 17.
- GREEN: `cd ui && pnpm exec vitest run
src/lib/agent-configuration/save.test.ts
src/lib/agent-configuration-yaml.test.ts --maxWorkers 1` — 51 tests
pass.
- GREEN: one focused Chromium treatment of the original compact YAML
case passes. The treatment changes the actual disposable database heap
after the first save. It does not transform HTTP responses. The 30
equal-time entries stay identical through the API, the second bind
returns 201, and the original hidden-setting and disabled-Save
assertions pass.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- Local tests use Node 24, a macOS source server, Chromium, and an
isolated loopback PostgreSQL 17 container. The server and container are
stopped. Private traces and control source are retained outside Git.
These controls do not prove a deployed Cloud result.
- A review follow-up removes two unnecessary query changes. Fresh real
PostgreSQL verification still passes all 18 tests. The list/detail
queries and browser path are unchanged from the passing heap-reorder
treatment.
- Prior head `59edcade203d6a05923955f5a0118e62c7e549c5` had a separate
browser failure in run `41169427774`:
`disconnect-write-recovery.spec.ts:204` kept the original title after a
mocked session lookup failure. Its relevant test and UI source are
unchanged from the base. The cause is unresolved. The two removed query
changes do not repair it. The original log and artifact are retained; no
unchanged job retry was issued. On the current head, this case passes in
14.3 seconds. The compact YAML case also passes in 11.2 seconds. The
later success does not establish the prior failure cause.
- Final exact head: `81c584aeac116b9d7a126b55dc99d5f6c70c195f`. [CI run
41169978784](https://github.com/paperclipai/paperclip/actions/runs/41169978784)
completes with success: all 47 workflow jobs are terminal, and all 57 PR
check/status entries are green or accepted neutral/skipped states. Full
typecheck, build, Runner checks, security checks, and `ci / verify`
pass. They were not repeated as broad local commands because the shared
host is under load.
- The new PostgreSQL regression also passes in Linux CI in 911 ms.
Server shard 12 has 1,839 passing tests. Browser shard 3 has 45 passing
tests, including the original compact YAML case.
- Fresh Greptile and Paperclip reviews both give this exact head 5/5
with no findings. All review threads are resolved. The branch is
mergeable and clean.

## Risks

- Entries with the same creation time now use ID order. Their prior
relative order was undefined. Entries with different creation times keep
their existing order.
- The query still sorts the same rows and adds one key to the sort. It
adds no query, polling, schema change, or migration.
- The configuration equality and concurrent-change checks are unchanged.
This fix does not ignore real tool changes.
- The original CI response order is unknown. The regression proves a
permitted ordering defect and the matching failure mechanism, not the
unique historical cause.

## Model Used

- OpenAI Codex, based on GPT-6, with reasoning, source inspection,
terminal tools, and browser test execution. The exact serving model ID
and context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-31 00:07:06 -05:00
DottaandPaperclip 100b364264 Preserve selected Google services when setup resumes (#18048)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections control which external services an agent can use.
> - Google Workspace setup lets a person choose services and read or
write access.
> - A draft lost that choice after a declined or expired OAuth handoff.
> - Finish setup could then show all services with write access instead
of the saved narrow choice.
> - This pull request saves the exact draft choice without treating it
as granted access.
> - The benefit is safe setup recovery that preserves the person's
choice and existing account limits.

## Linked Issues or Issue Description

Refs #15545. The duplicate search found no open PR for Workspace draft
consent recovery. Related #18000 concerns satisfied gateway access
requests, not saved OAuth setup intent.

**What happened?**

A cold Workspace draft resumed with the new-account service defaults.
The selected request lived only in short-lived OAuth state. Denial
consumed that state. The picker ignored draft connections.

**Expected behavior**

Finish setup restores the same person's saved service choice. It does
not use another person's choice, a broad profile hint, or a credential
grant as draft intent. An old draft with no reliable choice asks the
person to choose services.

**Steps to reproduce**

1. Start a new Google Workspace connection. Select Gmail read-only. Turn
off Drive and Calendar.
2. Continue to Google, then decline authorization.
3. Close the setup page. Return through Finish setup.
4. Open the service picker. Before this fix, all three services can be
enabled with write access.

**Paperclip version or commit**

Reproduced in tests on master
`820c458465fa40137c81fe510f3e6dc573b4f895`.

**Deployment mode**

Authenticated and local source builds. The source regression uses real
PostgreSQL fixtures and the production React setup components. Live
provider acceptance of this repair is still pending deployment.

## What Changed

- Save validated, non-secret draft intent before managed or
customer-owned OAuth. Bind it to company, connection, actor, grant kind,
and subject.
- Restore that intent after denial, state expiry, and cold Finish setup.
Keep active reconnect bound to its saved grant. Do not change
new-account defaults.
- Preserve agent-to-human setup handoffs. Consume only the addressed
person's reviewed proposal. Keep other people's proposals.
- Lock and merge the current draft before saving intent. Return 409 if
its credential identity or method changed during setup.
- Preserve the latest saved choice across overlapping customer-owned
starts. Merge both current configuration records under short locks.
Reject stale client-binding changes. Keep each OAuth state's own
requested scopes.
- Add shared, service, handoff, picker, and cold-page regression tests.
Update the connector contract and existing feature plan.

- Route the default ungrouped test/release paths through six sequential
exact-cover chat partitions. Preserve remaining server/native coverage,
explicit grouped CI, selection-first execution and shared-database
paths.
- Correct the retention test driver so pending owned real I/O does not
advance logical deadlines. Pump zero-duration connection timers and add
a held-I/O regression; keep production scheduling and original timing
assertions unchanged.

The shared helper is needed by the server and picker to validate the
same narrow binding. It does not resolve credentials or grant access. No
production schema migration, new route, provider scope or deployment
configuration is added. The test-harness changes do not change
production timers.

## Verification

### Current candidate: source qualification passed; live acceptance
pending

- Current published candidate:
`82d13d4d698a5579378b5d506cb7a1f9cb64b5c8`, tree
`78d6a821ce53f481fe3e9991edce56f0acffb22f`. It combines the Workspace
repair with launcher commit `bf29c9` and retention-test commit `e25987`,
then integrates master `493e229d93ba7798e80152f4549e2e9177ce50ae`. The
integration brings 40 upstream commits across 713 files. One launcher
conflict keeps upstream validated CI selection and its selection-first
path, then applies six default chat partitions only without a selection.
The authored PR diff has 16 files, 833 additions and 100 deletions; it
adds no authored lockfile or `.github` changes. The public head is
confirmed at this revision. Fresh CI and review passed as recorded
below. Source qualification is complete using exact-tree full CI and the
local results below. The separate local whole command was interrupted by
host disk protection and is not called a pass. Live repaired-draft
acceptance remains pending deployment.
- Exact clean `82d13d4` passed one Node 24.19.0 builtin-only run: 41/41
cases (37 launcher and four CI-workflow cases), zero failures/skips.
HEAD, tree and relevant file hashes stayed unchanged and clean before,
at start and after. This proves launcher fixtures, not real chat
execution, current typecheck/build, the dependency-backed selection
graph, or full feature acceptance.
- The new default ungrouped whole command uses six sequential
exact-cover chat partitions, then the remaining server suites,
workspaces and serialized suites: 28 expected invocations. It preserves
all native suites, including upstream usage coverage. The plain
ungrouped release launch path also changes. Explicit grouped CI,
validated selection and shared-database execution stay unchanged.
Launcher hard RED had three expected body failures and two unchanged
passes; the repaired focused five and final 37 passed. A later 41-case
staged-merge pass is retained separately; the clean 82 run above is the
frozen proof.
- Fresh [CI run
41153113944](https://github.com/paperclipai/paperclip/actions/runs/41153113944)
completed successfully on October 31 at 01:14:53 UTC. Current head has
52 successful check runs, four conditional skips and no pending/failing
checks; legacy Snyk also passed. The skips are test-file selection, the
optional paired benchmark and two Storybook checks. The full
server/chat/workspace/serialized matrix, all eight e2e shards, normal
build, build-gap typecheck, Runner checks and final aggregate gates
passed. Build and all three chat job logs identify checkout
`e5e831fd3a8534d0d0f1e892118c0f8fbe12dd92`, merging this head into
`493e229d93`. Its Git tree is exactly
`78d6a821ce53f481fe3e9991edce56f0acffb22f`, identical to the clean
candidate. The six real chat partitions passed 314, 358, 273, 239, 321
and 311 cases: 1,816 total. Other cases shown as skipped in each
partition were selection exclusions, not extra passed cases. This is
current tree-equivalent CI proof, not a pass of the separate default
ungrouped local whole command.
- Fresh [Greptile
summary](https://github.com/paperclipai/paperclip/pull/18048#issuecomment-6473856992),
edited October 31 at 01:05:04 UTC, explicitly reviews exact head
`82d13d4` at 5/5 with no actionable code issue. Its exact-head check
passed. The prior overlapping-start discussion is resolved. This review
is source-only, not provider or live acceptance.
- Exact clean `82d13d4` passed full recursive typecheck and normal `pnpm
build` on the independent darwin/arm64 snapshot. Typecheck finished on
October 31 at 01:27:33 UTC; build finished at 01:28:33 UTC. Both exited
0. Normal native preparation and code signing passed. Source HEAD, tree
and tracked/untracked state stayed unchanged and clean. `pnpm
check:token-gates` also passed all token, palette and ClipLab checks on
this candidate. The earlier forced-clone `ENOSYS` preflight remains
recorded as setup failure, not an assertion result.
- The unchanged default `pnpm test:run` started at 01:51:33.796 UTC on
clean exact `82d13d4`. All six real chat partitions passed: 358, 314,
273, 239, 311 and 321 cases, or 1,816 actual cases. The original FIFO
and ambiguous-retry cases both passed under their unchanged 15-second
limits. General-server then reported 37 completed file rows with 5,438
passed cases and five default remote-continuity skips, including all 498
tool-access service cases and all 83 connection-route cases. At 02:39:44
UTC, the unchanged 5-GiB host-disk guard terminated the owned run. The
child ended with SIGTERM and the container/harness exited 143, with no
OOM or recorded assertion failure. Only seven of 28 RUN invocations
started. Retention had no completed local row. Later invocations did not
run. This whole local command is interrupted/incomplete, not passed.
- A coverage check found no default suite missing from current full PR
CI. The maintained launcher tests prove exact coverage for all 12 PR
server shards, chat and five native suites; all configured workspace
projects; and all nine serialized shards. The final Runner CI shard
executes the five native server suites. The 41 local launcher/workflow
cases cover six sequential chat children, per-child isolation, native
retention, fail-fast propagation and unchanged explicit
selection/shared-database paths. The actual six-chat run above exercises
that changed default path. The remaining unqualified local difference is
uninterrupted sequential whole-command composition and resource
accumulation, not omitted test selection. Current exact-tree full CI
supplies broad source-test qualification; no local whole-pass claim or
provider acceptance is inferred.
- A task-owned remote runner was considered to complete that separate
composition check. Its provider rejected the requested allocation with
per-sandbox limits of 8 GiB RAM and 10 GiB disk. No sandbox was admitted
and no remote tests ran. No tests, timeouts, assertions, database
synchronization, source or provider settings were changed to avoid the
capacity limit. Historical a0 failures remain below. No deployment or
live repaired-draft acceptance has run.

### Bounded follow-up on the prior source

- On a0, one finite phase-traced chat diagnostic executed exactly the
two failed cases. Both passed (1,374 and 1,767 ms); 1,814 other cases
were filtered, not passed. This did not reproduce a blocked await or
establish the original monolithic timeout cause. No assertion or
15-second limit changed.
- The retention diagnosis preserved an initial zero-body selector
failure, then corrected only the selector: one original case passed in
783 ms, with 42 filtered. Trace showed the old settling helper advancing
logical time while real I/O was active. A new held-I/O regression then
failed hard because the old helper advanced 1,000 ms. This proves a
test-driver defect, not the precise cause of the old 5,340 ms full-run
gap.
- The first manual-clock repair failed the complete 44-case file: 43
passed, one body hit 15 seconds and cleanup hit 30 seconds. Its planned
mutant did not run. Source then identified a fresh PostgreSQL
connection's zero-duration timer. The corrected driver pumps
zero-duration timers while initial readiness or tracked I/O is pending,
without advancing logical Date. That pre-integration file passed 44/44
with no skips. A separate scratch-only 5,000 ms normal-continuation
mutant failed the original hard `5000 < 3000` assertion; its expected
failure is not a test pass. The original page-count, fairness, expiry
and quiet assertions stayed intact. The authored repair does not change
the production worker. The merged master changes an older retention case
and dependencies; these pre-integration results are not a complete 82
retention or whole-suite pass.

### Historical qualification: SHA-bound, not 82 acceptance

- Historical a0 candidate: `a0f9089fc1ff466bf4e1eeddc5b1a910d09e380b`,
tree `b9d73713319dcd52b9eaa6053008b5190e43fdf1`. It adds the
overlapping-start repair to parent `6c18`, which already merged pinned
master `1511c92965325249ed6b52dd5be52eb6d736fbfa`. That candidate
changed 12 files. Its CI and review passed; its local full qualification
remained failed/incomplete.
- Greptile's previous `6c18` review found P1 [Latest service choice gets
lost](https://github.com/paperclipai/paperclip/pull/18048#discussion_r4425965828),
confidence 4/5. Two hard-assert regressions reproduced it on untouched
`6c18`: the already-bound path overwrote the later choice, and real
first-use client stamping also restored the old choice. Existing OAuth
states/grants were not widened, but a later setup could restore an older
request. The new repair covers both write boundaries without holding
locks over external work or copying another attempt's scopes.
- The complete affected service file passed 498 tests with no failures
or skips. All 31 required controls ran, including both unchanged RED
cases, 14 new client-drift cases, seven earlier identity guards,
reconnect/cold-resume cases and upstream transaction tests. The tested
three-file patch is byte-identical to this commit; the connector
contract update is separate. Independent scoped review found no
additional actionable issue. Fresh Greptile reviewed exact `a0f` at 5/5
with no actionable findings. The previous P1 thread is resolved.
- On exact `a0f`, the five shared/UI files passed 312 tests. An initial
complete connection-management run then failed: 74 passed, eight timed
out at the unchanged 15-second limit, and one AI-account verification
returned 422 instead of 201. The Google addressed-user handoff passed.
The container exited 1 without OOM and stopped before chat/token/whole
tests. This material failure remains retained and its cause
unclassified; the later controlled recovery below is not a causal
explanation.
- One diagnostic executed the first failed gateway case and the two AI
cases with temporary phase-only tracing. All three passed at 8.516,
5.511 and 3.523 seconds; 80 other cases were filtered. Original hard
assertions and timeouts stayed unchanged. No timeout or suspected late
provider cleanup reproduced, so the complete-file failure remains
unresolved. Vitest exited 0; the harness exited 1 because its trace
parser missed ANSI prefixes. All 203 events were recovered from retained
logs without rerunning tests. Instrumentation was removed. This is not
acceptance.
- Historical a0 CI
[41097596364](https://github.com/paperclipai/paperclip/actions/runs/41097596364)
completed successfully: 52 successful checks, two expected Storybook
skips and no failed/pending checks. Legacy Snyk passed. All 48 CI jobs
passed, including all 1,816 chat cases and explicit passing lines for
the original 20 local failures. [Server
11/12](https://github.com/paperclipai/paperclip/actions/runs/41097596364/job/124207469276)
explicitly passed the complete 83-case connection file in 129.49
seconds, including all nine local failure titles. The actual checkout
was synthetic merge `06851da6eb969d63e1c7dc9f7c95f08d09fa4ff1`,
combining `a0f` with `aa58fa107389e8b9ae5949f7ad815943dfd94292`. This is
not standalone local `a0f` qualification and does not erase the nine
local route failures.
- Linux admission uses Node 24.22.2, pnpm 9.15.4 and Rust 1.97.1. Frozen
offline install, exact native hashes and real migrated PostgreSQL 18.1
passed. Native/protocol/build/lock inputs are identical between `6c18`
and `a0f`; validated cache reuse retains the actual `6c18` native-build
provenance. The selected packaged release binary is not called a fresh
`a0f` build. Database synchronization settings stayed on. This admission
alone does not explain the route failures.
- A fresh Mac snapshot of exact `a0f` passed frozen install, workspace
prerequisites, full sequential recursive typecheck and normal `pnpm
build`, both on their first attempt. Typecheck completed at 17:43:05
UTC, build at 17:44:43 UTC; both exited 0 with clean tracked/untracked
source. Pinned Node 24.19, pnpm 9.15.4, Rust 1.97.1, package concurrency
one, Cargo jobs one and a 12 GiB V8 heap ceiling were retained.
Largest-process maximum RSS was 7,668,154,368 bytes for typecheck and
10,038,837,248 bytes for build, not aggregate tree peaks. The first
build correctly withheld its freshness stamp because 125 ignored, pinned
Copilot assets appeared during preparation; exact path/hash analysis
explained this. The normal build then recorded a genuine stamp and
reused Cargo outputs in 1.18 seconds. No manual stamp, source workaround
or gate retry. This is darwin/arm64 static/build proof, not Linux
packaging/runtime or full tests.
- The historical a0 candidate's token checker passed all four gates
across 1,410 files, run directly with pinned Node 24 against the script
behind `pnpm check:token-gates`.
- The single controlled complete-file comparison passed: baseline `6c18`
83/83 in 287.07 seconds, then candidate `a0f` 83/83 in 293.03 seconds,
with zero skips. All nine earlier failure titles passed on both. Both
child/container exits were 0, no OOM/interruption; source and
native/resolver/PostgreSQL guards matched. This ran only after all owned
Mac compilation ended, with fresh processes/databases and identical
four-CPU-quota/6-GiB/overlay inputs. No filtering, instrumentation,
assertion/timeout/source changes or retry loop. This establishes
standalone complete-file recovery, not the cause of the earlier eight
timeouts/AI 422.
- The first unchanged full `pnpm test:run` on exact clean `a0f` ran from
October 30 at 18:01:43 UTC to 21:06:19 UTC. It failed naturally with
test and container exit 1, without OOM. General-server reported 1,257
passed, one failed and seven skipped files; 28,633 passed, one failed
and 103 skipped tests. Only the first of 22 invocations ran. The later
four groups and 21 invocations did not run. The sole failure was the
unchanged chat-shard CLI meta-test: its nested projectless pnpm lookup
hit the offline network guard. This was a failed full gate, not Google
provider acceptance.
- A finite environment pair reproduced that one meta-test failure, then
passed the same unchanged case after supported `corepack prepare
pnpm@9.15.4 --activate` as the existing non-root node user. The cached
pnpm payload was identical. Only the missing node-user last-known-good
record changed. Core source, dependencies, lockfile, fixtures,
assertions, timers and offline guards stayed unchanged. This isolates
the nested collection defect; it does not explain the earlier route
failures or complete the full gate.
- The corrected unchanged full command ran from October 30 at
21:29:03.500 UTC to October 31 at 00:02:44.207 UTC on the same exact
clean `a0f`. It failed naturally with exit 1, signal null and no OOM.
Source stayed clean before, at start and after. It retained two visible
CPUs/quota, 6 GiB, private IPC, normal overlay, matching native inputs,
synchronized real PostgreSQL 18.1, and the default five groups/22
invocations. Only the first general-server invocation ran; the other 21
did not run. General-server reported 1,265 files: 1,256 passed, two
failed and seven skipped. It reported 28,737 tests: 28,631 passed, three
failed and 103 skipped. The two chat cases emitted `Test timed out in
15000ms` at `chat-channels.integration.test.ts:77310` and `:81790`, with
durations of 15,045 and 15,058 ms. These were the durable pre-run FIFO
notice and ambiguous GitHub text restart/retry cases. The retention case
emitted `expected 5340 to be less than 3000` at
`skill-version-retention.test.ts:593`. Its real duration was 1,727 ms;
5,340 ms was the recorded fake-time directory-page gap, not a deadline
timeout. The timeout stacks do not identify the blocked internal await.
The cause of these failures remains unclassified. Inside this same
failed run, the complete 83-case connection file and 498-case service
file passed. The former Corepack meta-case also passed in 6,396 ms.
These component passes do not waive the failed/incomplete full gate. No
rerun, cancellation, timeout/assertion change or full-pass claim. A
later fresh two-case phase trace passed both chat cases; it did not
reproduce or establish the original timeout cause. At that checkpoint,
the PR stayed draft.
- Prior `6c18` targeted checks passed 930 tests across eight files and
token gates. Its selected queue group passed all 53 cases, including the
original 20 failures and 12 new named controls. Backlog cases took 10.49
and 8.45 seconds under the unchanged 15-second limit. Another 1,763 chat
cases were intentionally filtered. This is historical targeted recovery,
not a full pass or proof of the new repair. No timeout or assertion was
relaxed.
- Prior `6c18` full local qualification failed during serial typecheck:
server TypeScript was killed with code 137. Docker confirmed OOM in the
6 GiB container; full tests and build never started. The failure is
retained. The new Mac resource plan avoids repeating that undersized run
and does not change the shared Docker VM.
- Prior CI
[41089793558](https://github.com/paperclipai/paperclip/actions/runs/41089793558)
completed successfully for synthetic merge
`a73bffcfa3ede051961bcabcfa576c703d10c932` (`6c18` plus
`abc270ee4c03e13b4d6eaad156a41b4ebed4bb0a`). All 48 CI jobs passed; all
1,816 chat cases passed, including explicit log lines for the original
20. Check runs were 51 success, two expected skips and one Greptile
failure; legacy Snyk passed. These results do not qualify `a0f` or erase
its local route failures.
- Red proof on the unmodified master baseline: cold picker and Finish
setup restore broad defaults; managed and customer-owned drafts lack
saved intent. The stale setup tests also fail before the lock guard.
- Live staging baseline `b25f3e2af8e77d882cd22d36ca55640e6468e46f` also
reproduces the ordinary failure: select Gmail read-only only, cancel at
Google, reload the connector catalog, then Finish setup. All three
groups return enabled at read/write. Authorization was declined; no new
Google access was granted. This is baseline failure evidence, not a
deployed-fix pass.
- Historical affected suites: 800 tests passed before the final lock
guard. The final focused service and handoff cases passed 126 tests
after that guard. These are not results for the integrated candidate.
- Historical `pnpm -r typecheck` and final lock-guard server typecheck
passed. The historical a0 candidate passed the full recursive gate
separately, as recorded above.
- `pnpm build`: passed again on the final lock-guard source in commit
`1c31cde0ed199fbf4e41b7649590554f7886dfc2`.
- Historical `pnpm check:token-gates` passed; the historical a0
candidate's direct checker pass is recorded above.
- `pnpm test:run` on macOS failed on the frozen commit. It exited 1
after 9,432.41 seconds. General server results: 1,195 files passed, 10
failed, eight skipped; 26,706 tests passed, one timed out, 157 skipped.
Nine setup hooks timed out while starting embedded PostgreSQL. The
retained log records host-wide semaphore exhaustion (`semget`), not disk
exhaustion. One unchanged publication-text performance case also timed
out. Its cause is not established. Later workspace and serialized groups
did not run. This is a failed and incomplete local gate.
- The independent full-source Linux run also failed and was incomplete
on the same frozen commit. Native and real-database admission passed.
Chat integration reported 20 failures in 1,593 tests. Tool-access passed
all 479 tests. Native-session-executor passed all 965 tests. Docker
interrupted the run after about 48 minutes on October 29. The container
exited 143 and was not OOM-killed. The run produced no final group
totals or detailed assertion summary. Shell trap metadata reports zero,
but this is not a valid test-pass result. Source and evidence remain
unchanged. The focused diagnosis below retains those failures; their
cause is not assumed from the interruption. Neither historical local
full run passed.
- A fresh focused replay on the frozen source then executed exactly
those 20 case bodies. All 20 failed; this was not a setup-hook failure.
The two backlog cases reached the unchanged 15-second timeout. The 18
FIFO cases failed on missing notices or receipt rows. All 18 FIFO cases
passed when run alone in a fresh process and database. This supports an
order interaction, not a complete cause or baseline conclusion. A
separate two-backlog control used RAM-backed temporary storage and real
PostgreSQL with synchronization settings on. One backlog case passed in
10.65 seconds; the other still timed out at 15 seconds. No source,
assertion or timeout was changed. The full local gate remains open.
- Earlier mixed-revision run: it began before the final guard and
imported older source before collecting later regression cases. It
failed those new cases with that older cached module. This run is
retained, not counted as candidate acceptance. The fresh focused run
passed all seven stale-setup cases. No case was skipped to hide a
failure.
- Historical remote gates for `1c31` were terminal and clean: 53
successful contexts and two expected Storybook skips. Main CI
[40720536514](https://github.com/paperclipai/paperclip/actions/runs/40720536514)
passed. Greptile reviewed exact head
`1c31cde0ed199fbf4e41b7649590554f7886dfc2` at 5/5 with no actionable
findings; the separate review job also passed, and there were no inline
review comments. These results do not qualify `6c18`.
- CI source qualification: the chat jobs checked out synthetic merge
`ef36746b3d2ec52ba5727dcbe4da265a03421fe4`, which merges the PR into
`7d75ae0569e41786640cf09647ae8849411d2f22`. All 20 locally failing case
names have explicit passing CI log lines. These passes do not establish
standalone local acceptance of `1c31`. The merge adds two upstream Teams
tests, so it collects 1,595 chat cases instead of 1,593. Neither
backlog/FIFO assertions nor their timeouts changed.
- Independent source review found an earlier credential-identity race.
The guard and seven deterministic cases resolve that finding. A later
integration review missed the separate overlapping-start race. Both RED
failures and that review miss are retained. The narrow follow-up repair
passed 498 service cases, independent source review and fresh remote
CI/5-of-5 review; full local qualification remains incomplete.
- Source qualification is ready for merge. Live repaired-draft recovery,
delivery and Google approval are not established by these results. The
local whole command remains explicitly incomplete.

## Risks

- Historical drafts without a matching intent now need an explicit
service choice. This is deliberate and prevents silent broadening.
- A concurrent credential or method edit now returns 409. Reloading lets
the person review the changed setup.
- Overlapping starts keep separate requested scope sets. Updating draft
intent does not relabel an earlier OAuth state's consent request.
- Saved intent is not consent. Provider-reported scopes and current
action policies still control discovery and dispatch.
- No migration is needed. No existing token or grant is rewritten by
saving draft intent.

- Default ungrouped launch paths now execute six chat children in
sequence and 28 whole-command invocations. Exact coverage, local
launcher contracts and all six real default chat partitions passed. A
complete uninterrupted local whole-command run remains unqualified
because of host capacity.
- The retention test-clock repair depends on owned-I/O and zero-duration
connection-timer progress. The controlled regression, sensitivity mutant
and current exact-tree CI protect the original assertion. It did not
reach a completed row in the interrupted local whole run.

## Model Used

OpenAI Codex, described by the runtime as GPT-5 based. The exact runtime
model ID and context-window size are not exposed to this task. Used
reasoning, code execution, browser verification, and independent
delegated review.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, including the runtime metadata
limit
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the bug report template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket or instance identifier
- [x] I have run tests locally and they pass — exact clean 82 passed 41
launcher/workflow cases and all 1,816 real default chat cases, full
recursive typecheck, normal build and token gates. Full exact-tree PR CI
passed. The separate local whole `pnpm test:run` was interrupted by disk
protection and remains incomplete; it is not claimed as a local whole
pass.
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [x] All Paperclip CI gates are green — current exact-head checks
passed; the verified CI checkout tree equals clean 82. The separate
interrupted local whole command is disclosed above.
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups —
fresh exact-82 summary and check passed; the earlier finding is
resolved. This is source review, not acceptance.
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-30 22:02:58 -05:00
4d0847c4ca fix(server): settle external conversations from committed work (#17772)
<!-- entire-trail-link-start -->
https://entire.com/gh/paperclipai/-/trails/6/changes/paperclip/6
<!-- entire-trail-link-end -->

## Thinking Path

> - Paperclip manages AI agents and the tasks that they execute.
> - External conversations must become idle after an authorized response
is published and all related work is complete.
> - Native replay and stranded-task recovery still discover this
settlement work through recurring scans.
> - A new deadline worker must preserve response ordering, monitor
handoffs, and unfinished work.
> - This pull request gives settlement one shared readiness predicate
and activates its worker from committed changes.
> - Existing rows provide recovery truth. No new table or per-feature
timer is added.
> - The benefit is a settlement worker that becomes quiet and resumes
when relevant work changes.

## Linked Issues or Issue Description

Refs #17758. This is slice 56. Slice 55 has merged. Related: #17606
(chat milestones), #17663 (bounded run reads), #17665 (monitor proof),
and #13809 (answered Slack conversation lifecycle). The provider
applicability remains Slack and Speko-owned conversations. This does not
extend Telegram settlement from #16559.

**What happened?**

Conversation settlement depended on periodic native replay and generic
task recovery. An exhausted monitor also cleared its schedule before
recovery admission. During that gap, an answered conversation could look
finished. Failed or deferred admission lost the monitor schedule.

**Expected behavior**

Relevant committed changes activate settlement. The worker uses the
owning service's evidence and becomes quiet when no settlement is owed.
Monitor recovery stays pending until admission, and restart does not
admit an already accepted recovery twice.

**Steps to reproduce**

1. Create an active external conversation with a successful run and an
authorized published response.
2. Keep execution, approval, a goal, or a monitor pending. Let the
settlement worker become quiet.
3. Resolve that prerequisite through its owning service. Verify that the
conversation settles without another periodic sweep.
4. Delay or fail exhausted-monitor recovery admission. Verify that the
monitor remains pending and the conversation cannot settle during the
handoff.

**Paperclip version or commit**

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Parent #17758 merged as `8ac5a128e66fb672e88106412f76cf278599bfc4`. The
publication adopts that actual squash through a normal merge with an
unchanged complete child tree.

## What Changed

- Preserve quiet after coarse company-only maintenance hints. The shared
inbound-notice and response-availability helpers keep settlement enabled
by default. Only the milestone dispatcher suppresses their duplicate
settlement fanout. Its scoped issue-review signal and the explicit
company-authority signal still wake real work. Four database cases prove
both quiet and real company-wide producer delivery.
- Preserve source-generation checks for chat handoff,
task-before-comment locking, issue-scoped decision invalidation, and
current conversation event columns from master.
- Track initial external conversation settlement through shutdown
without blocking HTTP readiness on endpoint or task locks.

- Reuse the real queue's coalesced recovery receipt after restart,
without admitting the same monitor recovery again.

- Share the existing conversation readiness predicate between bounded
discovery and locked settlement. Preserve current provider policy,
run/publication proof, company boundaries, and endpoint-to-issue lock
order.
- Register a scoped deadline worker through the delivery coordinator.
Preserve startup, restore/worktree suppression, drain/resume, failure
recovery, and joined shutdown.
- Extend existing review, milestone, wake-notice, and finalization
signals. Add notifications at independent issue, comment, goal, session,
and wake-receipt unblocks.
- Keep monitor owner recovery pending until guarded admission. Reuse an
accepted receipt for the same monitor schedule after restart, even when
its limit reason changes; retain compatibility with legacy receipt keys.
Commit board escalation and its monitor clear together.
- Delegate periodic native settlement while preserving resource
callbacks. Keep direct/manual defaults and preserve generic recovery
ordering with a read-only readiness guard.
- Add the producer and lifecycle audit in
`doc/plans/2026-10-26-external-conversation-settlement-sleep.md` and
update the stack ledger.

## Verification

Accepted production source: `50f4432ed88537b8d522cf1dce24c33588db0bac`.
Publication head: `b256bb52647affd9f5279ea8c40f48bbe40f031f`.
The complete accepted child tree stays unchanged when adopting merged
parent #17758. This publication adds only the audit document.

- **3,129 distinct cases in 37 complete server files pass**, plus **18
selected chat-response cases** (3,147 selected cases). The other 1,796
chat cases were not run locally. Full settlement, queue, scheduling,
run-state, goals, interactions, task Retry, decision targets, status
reassessment, workspace, process recovery, and review consumers are
covered.
- **Preserved failure:** the first 35-file command passed 2,445 cases
and timed out one restored-work handoff at its existing 15-second limit.
That aggregate command failed. The unchanged full 89-case actions file
then passed on a fresh fixture; the affected case took 22 ms. Root cause
was not established. No timeout or assertion changed. The 34 other files
passed originally. Distinct counts exclude duplicate actions passes.
Fresh CI remains a separate gate.
- **Indirect wake regression:** the original unconditional baseline
failed two quiet checks. Guards alone left the chat case failing with
three candidate queries. The final controlled indirect baseline fails
one of four cases; the repair passes all four with the same test hash.
Positive controls preserve real company-only producers.
- **Source and fixture controls:** all 38 selected test files restore to
their exact committed hashes after fixture imports. Fresh PostgreSQL 17
fixtures use 413 migrations. All five manual merge resolutions preserve
current authority, locking, and event-column contracts.
- **Static checks pass:** database build; server typecheck and build;
module boundaries; migration order; Copilot profile identity. Runner
packages equal the accepted parent byte for byte. No new Runner/package
qualification is claimed.
- [Current integration and wake
audit](https://github.com/paperclipai/paperclip/blob/b256bb52647affd9f5279ea8c40f48bbe40f031f/doc/plans/2026-10-26-external-conversation-settlement-sleep.md#current-master-integration-and-indirect-wake-audit-2026-10-29).
Latest-head CI and exact-head Greptile 5/5 must pass before merge.

<details><summary>Earlier revision verification and preserved failure
history</summary>

Receiver failure fairness: a controlled 1,100 ms batch reproduced a
30-second delay for healthy due receivers. The worker now supplies its
existing per-item deadline through `nextRunAtOnError`. All 20 receiver
lifecycle tests pass on the owner and combined tip
`31c12f1d68ff4a51e06fc2be2abf2c2aee9f9d3a`; server production
typechecking passes on the owner. The failed receiver retains its own
backoff, and settled receivers return to zero recurring SQL. The
five-second assertion budget is unchanged. See [the owning
audit](https://github.com/paperclipai/paperclip/blob/83204a67880d980cfbea3771cee19a3cda0022b7/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#healthy-receiver-progress-after-a-slow-failed-batch).
Current-head CI and review remain required.

Approved delegated Test-tab calls now evaluate caller and target
policies in the existing claim transaction. Real HTTP regressions first
showed a missing caller charge and shared-limit self-rejection. All 78
connection-management cases and 92 gateway cases now pass on the owning
branch, along with server production typechecking. All 78
connection-management cases also pass at combined tip
`d25f830db89c1623bff3547bfc68015c1c66e0cf`. The three new cases cover
shared charges, separate charges, rollback on denial, duplicate
approval, and rejection of a later authorized call. Intermediate
counter/identity fixture failures and a reused-database Slack collision
are recorded in [the owning
audit](https://github.com/paperclipai/paperclip/blob/ef9ae5767430d0093efc736ae0b303ba5afaae65/doc/idle-approved-tool-recovery.md#delegated-test-tab-admission-after-policy-changes).
The earlier combined checks below remain evidence for their stated
revisions. Current-head CI and review remain required.

Inherited voice-cleanup verification: the controlled slow-provider
regression and all 13 lifecycle tests passed on its owner and combined
tip `ab22d8ed0c09028315607414b83b375e5cc0a60a`. That repair changed only
the fixture clock advancement and its audit note.

Current head: `83204a67880d980cfbea3771cee19a3cda0022b7`. This remains a
single-duty PR stacked on #17758.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-24-recurring-server-sleep-stack.md),
[2026-10-26-external-conversation-settlement-sleep.md](https://github.com/paperclipai/paperclip/blob/d6d8e4ffa6b330ebf4647aec997ee2dde6abc321/doc/plans/2026-10-26-external-conversation-settlement-sleep.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.


</details>

## Risks

- Signals cover supported in-process writers. They do not supply change
capture for independent SQL or other replicas.
- Real company-wide changes can inspect active candidates. Coarse
maintenance hints do not activate settlement. Configured idle and
retained history must remain quiet.
- Monitor receipt binding and admission order must preserve company,
agent, issue, and exact schedule identity. Current parent handoff locks
and source-generation checks remain intact.
- The local restored-work timeout is retained above. A fresh full-file
pass does not establish its cause; latest-head CI must also pass.
- Generic recovery and Muse/resource duties remain separate slices. This
does not make the whole server idle. Existing Cloud sleep blockers
remain. No deployed sleep, database suspension, external alarm coverage,
fleet cost, or live provider qualification claim is made.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution, PostgreSQL
tests, and GitHub tools. The exact runtime model identifier and
context-window size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 06:02:43 -05:00
df6156c2e3 fix(server): wake native status reassessment from committed changes (#17709)
## Thinking Path

> - Paperclip manages agents, tasks and their durable native results.
> - Changed evidence, task authority or review retirement can require a
new status assessment.
> - Periodic orphan recovery repeatedly reads history to find these
changes.
> - This change gives reassessment an owner driven by committed changes
and saved deadlines.
> - Existing result, assessment, decision and task rows remain durable
truth.
> - The status committer still verifies current sources and authority
before writing.
> - Settled or insufficient evidence becomes quiet until a supported
producer changes it.

## Linked Issues or Issue Description

Refs #17702 (merged), #17679 and #17680. This is slice 50, now based on
master. It owns native status reassessment; workspace settlement,
retained sessions and other orphan duties remain separate.

**What happened?**

Periodic recovery repeatedly inspects completed native results. Removing
that scan without complete writer coverage or current authority checks
could strand new evidence or commit an old result over newer work.

**Expected behavior**

Reassess from existing durable facts after relevant outer transactions
commit. Retain real retry/lease deadlines and restart/drain recovery.
Preserve current authority, quality-review and supersession checks. Make
no recurring SQL when work is settled or dormant.

**Steps to reproduce**

Complete a native result awaiting evidence or review, leave it idle,
then approve a real work product or retire the selected automatic
review. Its saved result should advance once when authorized, or record
the existing nonterminal business gate, without a periodic scan.

## What Changed

- Extract reassessment and its pure status policy from finalization
recovery into their owning modules. Add a scoped worker using the
existing delivery coordinator and scheduler; no queue table or migration
is added.
- Retain the three durable discovery triggers: changed work-product
history, authoritative task status/version, or retirement of the
selected automatic completion review. Preserve selected-source,
ownership, effects, retry and lease fences under the existing committer.
- Notify from actual task/work-product, claim release, approval,
interaction, OAuth, event evidence and workspace-settlement owners
before their transaction writes. Keep provider calls outside short
transactions and nested writes behind outer commit.
- Scope ordinary milestone/finalization/coalesced-input wakes to an
issue/run. Keep genuine company authority restoration explicit. Real
Photon and coalescer regressions cover all three response/status queues.
- Move the current parent's required quality policy into the owner and
apply it to every reassessment that could complete a task. An unmet
review records the existing in-progress decision and next action,
avoiding an operational retry loop. Preserve conversation/session reset
locks and the workspace omission-barrier signal.
- Skip reassessment in periodic recovery while preserving
explicit/startup recovery and the historical supersession guard before
independent workspace work. Document the source contract, producer
inventory, failures and qualifications in the linked audit.

## Verification

- Broad source `9fddb3f0b1157df32b64fb1f22e9d6e9624470e8` passes **2731
distinct cases in 28 complete files**, with no skips: 2458
producer/owner/authority/workspace/quality cases in 394.43 seconds and
all 273 governed-response cases in 53.74 seconds. Nineteen dedicated
fresh PostgreSQL 17 databases migrate through 0413; eighteen
embedded-only invocations preserve fresh-per-call isolation through the
existing recovery helper. All 22 temporary imports restore exactly and
all 28 hashes match.
- Final runtime source `5ea3b0fd016bf7e63ba82a7044825a46a0cd2082` passes
**1211 cases in five complete affected files** in 108.23 seconds after
the narrowly reproduced evidence-trigger fix: all 43 worker, 145 corpus,
32 finalization-recovery, 968 native-session and 23 quality-review
cases. Four fresh databases plus the quality helper's own fresh fixture
use PostgreSQL 17. Its one adapted import restores exactly; all five
hashes match. The other 27 broad-run test files are unchanged. These
qualifications are source-scoped and not cumulatively double-counted.
- Required/optional real regressions reproduce retry loops on
retired-review and evidence-approval paths before repair; both become
SQL quiet afterward with unchanged assertions/time limits. Controlled
removal of only ordinary scope guards reproduces two coarse-hint SQL
failures; exact guard restoration passes. Real producer, outer
transaction, claim, authority, 103-item backlog, retry and drain cases
pass. Photon transport is mocked; callbacks, locks and observed SQL are
real.
- Normal merges preserve 12661 parent/unrelated entries, 35 owned
entries and 21 automatic shared results, with five original conflicts
reviewed explicitly. The parent's actual squash reconciliation preserves
the entire held child tree. The final quality fix changes only its owner
and tests. Fourteen added comments are removed with identical nontrivia
tokens; no directives are removed and the final comment check is empty.
- Database build, server types/build, boundaries and exact-head
migration order pass. The database tree is identical to merged #17702:
no new SQL, queue table or snapshot is generated. Earlier benchmarks,
host limitations, failed assertions and their repairs remain in
`doc/plans/2026-10-26-native-status-reassessment-sleep-audit.md` and
`doc/idle-native-status-reassessment.md`.
- This qualifies targeted external PostgreSQL behavior, not embedded
bootstrap on the semaphore-exhausted host, a full local repository
suite, deployed sleep or measured fleet savings. Fresh exact-head CI and
review remain required.

## Risks

- Notifications retain the process-local supported-writer contract.
Independent SQL writers and uncoordinated replicas are not covered by a
new transport.
- Reassessment competes with current authority changes. Existing
row/source locks and retries remain required; arbitrary missing
identities have no row to lock, so this is not a general predicate-lock
guarantee.
- Discovery pages 100 rows but retains outstanding candidates in memory.
It remains proportional to backlog size, and cancellation occurs between
queries.
- Other orphan/workspace/session duties and Cloud sleep eligibility
remain unchanged. Missing supported producer coverage could strand
dormant work, so the producer audit and real callback/transaction tests
remain part of acceptance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (targeted scope is stated
above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 03:40:01 -05:00
79ce86630a fix(server): schedule governed review response recovery (#17702)
## Thinking Path

> - Paperclip manages agents, tasks and their external chat responses.
> - A completed chat turn can owe a response while an earlier review
gate remains pending.
> - Periodic recovery revisits that response history even when no work
can advance.
> - This change uses the existing delivery coordinator and saved retry
deadlines.
> - Approval, interaction and authority changes wake the affected issue
or run after commit.
> - The existing presentation owner verifies the durable result and
current authority.
> - Settled and dormant history becomes quiet without losing future
eligible responses.

## Linked Issues or Issue Description

Refs #17696 (merged) and #15625. This is slice 49, now based on master.
It owns governed-review response presentation. Accepted responses and
pending decision effects keep their existing owners.

**What happened?**

Native finalization recovery repeatedly inspects committed review
responses. Removing the scan without covering approval, interaction,
task-policy and authority changes would strand responses that become
eligible later.

**Expected behavior**

Recover an owed response once from existing durable facts. Enforce
authority at publication, wake after relevant outer transactions commit,
retain real retry deadlines and restart/drain recovery, and stop
recurring SQL when no response can advance.

**Steps to reproduce**

Complete an external chat turn while an earlier completion review
remains pending. Block its response behind an approval or interaction,
then resolve that blocker. The response must appear once without a
periodic scan or resolving the original review gate.

## What Changed

- Add a scoped governed-review worker using the existing coordinator,
finalizations, decisions and comment receipts. No queue table or
transaction wrapper is added.
- Keep recovery and publication policy in the native finalizer. Preserve
binding, source/session authority, ownership, deadline and
deleted-comment fences; match coordinator-before-issue lock ordering.
- Notify through actual approval, interaction, OAuth, task-policy and
lifecycle producers before their owned writes. Serialize approval
resolution with task linking so a concurrent link retains its wake.
Nested transactions still settle at outer commit.
- Scope ordinary milestones, finalization and coalesced input to their
issue/run. Use the explicit company authority boundary for genuine
global restoration, including membership changes. Both response workers
share this boundary.
- Register under maintenance and drain gates, disable the two periodic
repair entrances, and retain fresh finalization and explicit recovery.
Add the concurrent 0413 comment-receipt index and migration
registration; retain five snapshots.
- Preserve the producer inventory, source qualifications and earlier
failures in the owner audit. Retain a real Photon regression proving
routine ownership assertions stay quiet and concurrent authority
restoration wakes each worker once.

## Verification

- Scoped repair source `2341714720b5c273f0ce011b48345af0ef39227f` passes
**2053 distinct cases**: 1703 in 15 complete producer/authority/recovery
files, including all 273 governed-response cases and 95 wake-queue
adapter cases, plus 350 selected chat cases. Another 1463 chat cases are
outside that selection. Thirteen fresh PostgreSQL 17 fixtures migrate
through unchanged 0413; the one temporary queue-fixture import restores
exactly. All sixteen captured hashes match.
- Two actual repeated coarse-hint SQL regressions fail before repair and
pass afterward. Actual membership restoration passes before and after.
The same assertions and time limits prove dormant work stays quiet and
real authority restoration still publishes once. Real coalesced input
covers both queues after commit.
- Combined source `586f5674cd9798b33cd22437255e0480b5726d20`, including
independent master #18129, passes **1297 distinct cases**: 1093 in five
complete server files, 185 in five runner files, 14 Node protocol cases
and five UI registry cases. This includes the complete 22-case Photon
file for both response workers. The external Photon transport is mocked;
Paperclip callbacks, transaction fences and observed SQL are real. The
earlier and combined case counts are separate, with no cumulative
double-counting.
- A normal merge retains both complete owned and independent master
binary patches. All twelve new-context hashes and sixteen earlier repair
hashes match. Three dedicated fresh 0413 fixtures plus the live-event
helper's own fresh database use PostgreSQL 17 without source
adaptations. Database build, server types/build, boundaries, full-SHA
migration order and generated TS/Rust profiles pass. The new inherited
Runner vendor inputs rebuild successfully.
- Database bytes remain identical to the previously qualified 0413
source. The recorded populated 0412-to0413 upgrade retains all 411 prior
receipts and 38 companies, appends one receipt and none on replay; the
index is valid and ready. The earlier 2828-case acceptance, CLI journey,
setup failures and their repairs retain their recorded source scope in
`doc/plans/2026-10-26-governed-review-response-sleep-audit.md`.
- Parent #17696 merged with 58 clean checks, Greptile 5/5 and Ponytail
5/5. Its Photon warning was disproved by real callback/SQL evidence and
the security check became successful. This PR still requires its own
fresh exact-head CI and review gates. No deployed sleep, external alarm
or fleet savings claim is made.

## Risks

- Notifications use the existing process-local writer contract.
Independent SQL writers and uncoordinated replicas are outside its
coverage.
- Genuine startup and company authority reconstruction remain
proportional to the owed backlog. Queries page 100 rows, but discovery
accumulates the candidates; this is not a constant-cost global
operation.
- Current publication authorization, durable binding, ownership and lock
ordering remain mandatory. The worker does not resolve the original
review gate.
- The concurrent index uses existing migration replay handling. Other
recovery duties and Cloud sleep eligibility remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution and PostgreSQL tests. The runtime does not expose a more
specific serving snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:54:46 -05:00
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00
DottaandPaperclip e93732c3d4 fix(server): deliver voice replies from commits and saved deadlines (#17578)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Phone calls receive approved answers from the existing chat
publication queue.
> - The chat maintenance timer checks for voice replies every second,
including when no reply is ready.
> - A reply can be saved before its publication commits. An early wake
alone can miss that final transition.
> - Existing session, reply, publication, and action rows already record
delivery and recovery work.
> - This pull request uses committed changes and saved deadlines to
drive phone reply delivery.
> - Empty reply queues become quiet while authorization, ordering, and
uncertain-send protection remain in place.

## Linked Issues or Issue Description

Stacked on #17576. Related delivery infrastructure: #15625. Related
voice connection work: #11418.

**What existing behavior does this improve?**

Background phone reply delivery and recovery of abandoned dispatch
intents.

**Current behavior**

Chat maintenance calls voice reply delivery every second. It also scans
old dispatch intents. Publication transport starts a direct push before
the final publication commit. The timer masks that timing gap.

**Proposed behavior**

Committed publication, reply, session, and configuration changes wake
the shared coordinator. Ready replies drain in bounded passes. Saved
rate-limit and dispatch-expiry deadlines retain recovery. An empty queue
or a permanent delivery barrier has no recurring reply scan.

**Reason and benefit**

Remove recurring SQL from idle phone reply delivery without dropping
replies, replaying uncertain sends, or adding a queue table.

**Breaking changes**

No API or schema change. Explicit manual recovery remains available.
Inbound admission cleanup, history retention, and reports remain on
their current paths for separate migrations.

## What Changed

- Extract ordered reply state and multipart policy within the voice
owner. Retain current authorization and atomic dispatch claims.
- Register a scoped reply worker with startup reconstruction, fair work
batches, failure backoff, drain/resume handling and shutdown ownership.
- Extend the existing shared dispatcher with normalized partial-page
results to preserve the bounded parent recovery contract. Existing
complete-array consumers retain their calls.
- Signal before relevant writes in their existing transactions. Both
normal publication completion and operator resolution wake the actual
conversation after commit.
- Bound reply inventory to 100 total headers per refresh. Preserve
partial scans, concurrent changes and newer denied-call restoration
obligations across reconstruction.
- Track abandoned dispatch actions independently of live calls.
Quarantine expired intents without resending them.
- Use existing active-call authorization checks to reactivate replies
that were previously denied. Healthy call inspections do not rescan
empty reply queues.
- Remove the periodic reply scan and the early direct publication push.
Keep provider requests outside database transactions.

## Verification

Published head: `1435cab10be3f95c38ab6672d2a890ae49e9d300`. Voice
recovery #17576 has landed at master
`e1ba77f06771a27114b3999004b0ca5e3c172dd4`. A normal merge reconciles
squash ancestry while preserving the complete reviewed 31-file reply
patch byte for byte and all unrelated master changes. The PR now targets
master; fresh exact-head CI and Greptile are required. The prior
449-case/typecheck/build results below cover the identical owned source,
not a new full-suite run.

- **449 distinct targeted cases pass**: 111 complete voice-store/reply;
19 complete parent recovery; and 319 complete tool-gateway,
secret-proposal, shared dispatcher/coordinator/batching/startup/hostname
cases. Production server typecheck and build pass. The policy checker
and its Node tests pass. Later edits record evidence only.
- Two real normal-publication/operator-resolution tests seed 105
unrelated calls and 105 future dispatches. Five publications selected
**1,065/1,060 inventory rows before repair**; both paths now pass the
same **125-row total budget** and deliver every answer once. This counts
selected SQL results, not physical tuple visits or host savings.
- Publication commits now signal their actual conversation. The voice
reader returns at most 100 total session/action headers per refresh,
preserves phase/cursor state and retries, and resets on worker
registration. Current-state checks and dispatch claims still own
provider effects. The 105-action case measures the page cap, restarts
the same owner during reconstruction, waits at saved future deadlines
and quarantines every abandoned action without resending.
- A real 403 fixture proves that an older broad scan cannot erase a
denied call first observed behind its cursor. Its owner callback still
emits the committed restoration wake after scan completion. The existing
actual-service authority-restoration and final-publication timing tests
remain in the full suite. They are not replaced by direct helper calls.
- Guarded removal of conversation scoping, cursor reset or the denial
revision fence fails the intended budget, 100-versus-5 restart or
missing committed-wake assertion; source is restored. An initial
shadowed fixture name and an inadequate due-action reset probe are
retained as failed attempts in the [current
audit](https://github.com/paperclipai/paperclip/blob/d2c0a7de8c3e9cac8382b1268f3f5d883a2067b3/doc/plans/2026-10-25-voice-reply-delivery-sleep-audit.md#publication-inventory-and-paging-acceptance-october-28).
- Parent recovery still passes paging, restart, behind-cursor insertion,
feature changes, actual disablement, quiet health observations, current
authority, exact-scope failure fences and wildcard-failure isolation
through the shared worker. Four existing array-based worker consumers
retain their API and pass their full suites.
- The two voice fixtures self-migrate separate empty databases. Legacy
tool-review/secrets suites use separate migrated clones. Earlier
full-local attempts remain limited by embedded PostgreSQL setup and are
not passing full-matrix claims. Older counts apply only to their stated
revisions. Fresh exact-head Greptile, Ponytail if present, security and
all CI remain merge gates. No live-provider, multi-writer,
external-wake, deployed-sleep or savings claim is made.

## Risks

- Missing a supported producer can strand work. Tests cover real
enqueue, retrieval, normal publication, and operator publication paths.
Startup reconstructs existing records.
- Signals cover the process-local database owner. Independent SQL
writers or other replicas need their own wake contract.
- Authority restoration relies on the existing live-call recovery check.
This dependency must be preserved if that owner changes later.
- A provider acceptance receipt does not prove speech. Unknown outcomes
remain blocked from automatic resend.
- This change does not extend Cloud alarms, allow sleep during live
calls, or prove deployed suspension or cost savings.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, reasoning, code
execution and test tools. The runtime does not expose a more specific
deployment identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used with the available version and
capability details
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run the selected tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:50:27 -05:00
DottaandPaperclip e1ba77f067 fix(server): schedule live voice recovery from session changes and deadlines (#17576)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Voice calls retain durable session state and provider identities.
> - The chat maintenance timer scans live voice sessions even when no
call exists.
> - Session writes, saved expiry and active-call inspection deadlines
identify real work.
> - This pull request moves live-session recovery to the shared
coordinator.
> - Idle session recovery becomes quiet without losing exact-call
cleanup or current authorization checks.

## Linked Issues or Issue Description

Refs #17572. This PR is stacked on receiver discovery. It owns only live
voice session recovery. Reply delivery, inbound admission cleanup,
inbound history retention and call reports remain separate migration
slices.

**What happened?**
The one-second chat maintenance lane reads voice settings and scans live
sessions even when an instance has no calls or only terminal history.

**What did you expect?**
Reconstruct live calls on startup. Wake on committed session or
configuration changes. Inspect active calls and retry cleanup at actual
deadlines. Stop querying when no live obligation remains.

**Steps to reproduce**
Start an instance with no live voice sessions. Let startup settle. The
old maintenance method continues to read settings and session inventory
every chat maintenance pass.

## What Changed

- Add a voice-owned bounded deadline reader and scoped worker on the
existing coordinator. Retain concurrent writes across pages and reset
partial scans on restart. No tables or migrations are added.
- Separate generic connection health observations from configuration
changes. Existing receiver and command consumers retain their
notifications. Voice eligibility and provider operations remain in the
voice owner.
- Signal existing session transactions and configuration/feature
changes. Keep provider calls outside database transactions. Standalone
session writes use one small transaction helper.
- Preserve saved expiry, the two-minute unknown-creation grace,
active-call authorization and missed-hangup inspection, and
exact-provider cleanup. Recovery never creates a provider call.
- Retry transient authority lookup failures instead of treating them as
confirmed revocation.
- Fence expiration against a provider identity bound after discovery. A
late known call remains eligible for cleanup.
- Preserve work through rollback, drain, restart, concurrent refresh,
backoff and shutdown. Page inventory and rotate bounded batches across
companies.
- Leave the four other voice maintenance duties in the periodic lane.
- Document producer coverage and the supported writer contract in
`doc/idle-voice-session-recovery.md` and the dated audit.

## Verification

Published head: `f0b2e3cc22a18e7d08b10d4fc02aae4a19a52f30`. Receiver
#17572 has landed in master at
`dce004259394486f6361d0c0dd5a33a5272acc0a`. This normal merge reconciles
squash ancestry; the complete 22-file voice recovery patch is
byte-for-byte identical to the reviewed and tested `ad79c7e805` patch,
while all unrelated master changes are preserved. The PR now targets
master. Fresh exact-head CI and Greptile are running; interrupted
Ponytail turns are not completed verdicts under the maintainer’s
clarified merge policy, and an actual failing verdict still blocks.

- **210 distinct targeted cases pass on the current repair**: 19
complete recovery lifecycle cases and 191 complete voice-store,
coordinator, batching, startup and hostname cases. Production server
typecheck and build pass. Later edits record evidence only.
- CI `40633667141` on prior head `1b6323c164` failed the feature-change
assertion with `34050 < 25000`; its verification aggregate also failed.
The test now disables automatically advancing fake time, drives 20
controlled scheduler turns and requires all 205 calls to close before an
unrelated inspection deadline can mask a lost wake. Removing generation
replay fails with missing closures. The repair does not benchmark
database/host speed or raise the elapsed-time bound.
- A real committed exact-call regression reproduces wildcard-backoff
starvation. The worker keeps the broad retry but processes independent
exact hints and cached due calls. Each check reloads current state and
authority. Failed exact scopes remain fenced. Removing wildcard
isolation fails the intended assertion; a successful broad retry then
leaves the worker SQL-quiet. Full-row and lean queries now share the
state/scope/cursor predicate.
- Earlier acceptance on `712f3862b5` passed 725 distinct targeted cases.
Its measurements remain useful history: five real metadata writes and
one health check over 205 future calls selected 1,230 rows before the
inventory repair and zero afterward, without changing the saved
deadline. Those older test counts are not added to the current 210-case
count.
- Both external voice fixtures self-migrate; the current full
voice-store run uses a separate initially empty database. Earlier
fixture/method/observer failures, source mutations and partial edit
attempts remain in the [current
audit](https://github.com/paperclipai/paperclip/blob/ad79c7e805367fc7e2df3ea53c942ca6c7d27365/doc/plans/2026-10-25-voice-maintenance-sleep-audit.md#wildcard-failure-and-deterministic-generation-acceptance-october-28).
Source is restored after guarded mutations.
- Older CI `40632057899` failed because its trusted policy script was
absent from the checkout, while build separately received a runner
shutdown signal. E2E shards succeeded but their aggregate failed on
policy. The current checkout includes the landed policy tooling and its
checker/tests pass locally. The shutdown cause is unproven; neither old
failed run is counted as passing. Earlier full-local attempts stopped on
embedded PostgreSQL setup failures and did not pass.
- Fresh exact-head Greptile, security and all CI must pass before merge.
A completed failing Ponytail verdict blocks; an interrupted turn without
a verdict does not. No full current local repository matrix, live
provider acceptance, multi-writer support, Cloud wake, host suspension
or savings is claimed.

## Risks

- Process-local hints cover the existing active writer and shared
clients. Independent SQL writers or another replica need explicit wake
integration.
- Membership, task binding, phone preferences and other authority inputs
remain checked on the existing 30-second active-call inspection deadline
and on use. This is not an instant cleanup guarantee for every authority
mutation.
- Unknown creation retains its existing maximum-duration grace. Cleanup
failures retain retries. A provider request is never replayed merely
because its creation response is unknown.
- Genuine configuration changes retain bounded reconstruction. Metadata
writes can probe parent eligibility, while health-only observations
leave voice recovery quiet. Paging state is process-local and
reconstructs from durable rows after restart.
- Active calls, the other voice maintenance owners and connector
transports remain real obligations. This does not expand Cloud alarm
coverage, relax sleep eligibility or prove deployed suspension.
- The receiver and Discord parents have landed. Current-master
compatibility and exact-head CI and Greptile remain merge gates.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, reasoning, code
execution and test tools. The runtime does not expose a more specific
deployment identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:38:48 -05:00
DottaandPaperclip dce0042593 fix(server): wake chat receiver discovery from configuration and lease deadlines (#17572)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat receivers own persistent provider connections and fenced
database leases.
> - The chat maintenance timer repeatedly queries all eligible
receivers, including when none exist.
> - Configuration changes and known lease deadlines already identify
when discovery is needed.
> - This pull request moves receiver discovery to the shared delivery
coordinator.
> - Empty or settled discovery stops querying, while ownership and
recovery remain explicit.

## Linked Issues or Issue Description

Refs #17557. This is the next stacked slice after Discord command
registration. The base is `codex/sleep-19-discord-command-registration`.

**What happened?**
The recurring chat maintenance lane scans receiver inventory and
reconciles Discord and Photon runtimes even when configuration has not
changed.

**Expected behavior**
Reconstruct receiver work at startup. React to committed configuration
changes and local listener retirement. Retry foreign ownership at the
saved lease expiry. Stop discovery when there is no remaining
obligation.

**Steps to reproduce**
Start an instance with no chat endpoints. Let startup finish. Observe
the recurring receiver inventory query from chat maintenance. With a
configured receiver, remove its ownership token while listener shutdown
is delayed and signal a configuration refresh: the old implementation
can replace the listener before shutdown completes.

## What Changed

- Preserve healthy receiver progress after a slow failed batch while
retaining per-receiver retry backoff.

- Register a receiver discovery worker with scoped refreshes, paged
startup inventory, fair batches, and per-receiver or scope failure
backoff.
- Keep owned receiver identities cached without a discovery deadline, so
deletion still stops their listeners.
- Use existing transaction signals for configuration, credentials,
health, access revocation, company deletion, and provider lifecycle
changes. Rename the generic command-only helper to describe its broader
configuration contract.
- Keep lease-key selection with the receiver owner. Shared scheduling
contains no provider protocol.
- Retain stopping ownership until listener shutdown and lease release
complete. Concurrent acquisition joins the existing stop promise.
- Remove receiver discovery from the periodic chat lane. Preserve voice
and the other unmigrated duties.
- Record producer coverage, failure evidence, limits, and remaining work
in `doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md`.

## Verification

Published head: `ab39569250f4b699a0ab7a829e66dbe27c15020b`. Discord
#17557 has landed at master `5bf0142d45`. A normal merge reconciles its
squash ancestry by applying only the recorded 27-file receiver patch
from parent `547fdf2139`. Every other path matches that master. Every
server and package production/test file is byte-identical to the
published `6e88ca0201` source that passed the 131-case run, typecheck
and build. Unrelated upstream UI/eval/documentation changes are
preserved. This PR targets master. Fresh exact-head reviews and CI are
required.

- **131 distinct targeted cases pass.** Five complete receiver,
coordinator, batching, startup, and GitHub health error-projection
suites pass 99 cases. Twelve real Discord
automatic-registration/receiver and Slack held-transaction cases pass
separately, with 1,581 other chat integration cases deselected. The
complete Photon channel suite passes 20 cases.
- The standalone receiver and Photon fixtures each reproduced
missing-table failures on separate initially empty PostgreSQL 17
databases. Both now apply pending migrations before client use; their
complete 20-case files pass on those same databases. The large chat
producer suite uses a separate migrated clone.
- The old review concerns already have source fixes: discovery stops
suppress their own completion wake at `1604a9fcba`; slow failed batches
supply `nextRunAtOnError` at `ab56393acc`. Guarded mutations prove the
existing regressions: reenabling the stop wake yields two inventory
reads instead of one, and removing the error deadline fails the 1,100 ms
provider-delay next-wake assertion. Both original source files are
restored byte for byte. Normal full worker and real producer suites pass
with those fixes.
- All 131 distinct targeted cases, production server typecheck and build
pass on source `6e88ca0201`. Exact source comparison carries that
evidence to this normal merge; the unchanged suites were not rerun just
for unrelated upstream changes. Receiver and Photon fixtures
self-migrate separate initially empty databases; the large producer
fixture uses its own migrated clone. Source is unchanged after these
checks. Recorded listener-count warnings and controlled provider failure
logs remain; no assertion was weakened. An initial whole-file identity
check rejected expected parent queue additions; precise patch comparison
confirms those declarations are correctly preserved.
- Old server shard 12 failed because GitHub reports that its self-hosted
runner lost communication. The grouped-test step is cancelled, and the
retained log has no failing assertion or completed summary. The
underlying host/resource/network cause is unproven. That interrupted job
is not a pass; the updated parent and fixture source require a fresh
full CI run.
- [Current acceptance and failure
record](https://github.com/paperclipai/paperclip/blob/ab39569250f4b699a0ab7a829e66dbe27c15020b/doc/plans/2026-10-25-chat-receiver-discovery-sleep-audit.md#compatibility-after-discord-squash-merge-october-27).
Prior 218-case and full-workspace results describe earlier revisions. No
full local repository-suite, multi-writer deployment, Cloud wake, host
suspension, or fleet cost claim is made.

## Risks

- Process-local transaction hints cover the active database owner and
its shared clients. Arbitrary direct SQL or independent configuration
writers need explicit wake integration. Startup/resume reconstruction
and known foreign-lease takeover remain supported; this does not add
general multi-writer discovery.
- Active receiver sockets and lease renewals continue. This change does
not prove host suspension or extend Cloud alarm coverage.
- Configuration signals can arrive during drain, initialization, or
shutdown. The worker preserves dirty scopes and the owner joins listener
retirement before replacement.
- Voice, Teams, milestone projection, publication maintenance, GitHub
recovery, and the remaining stack inventory are separate work.
- The Discord parent has landed and the receiver child now targets
master. Fresh exact-head reviews and all CI must pass before merge.

## Model Used

OpenAI GPT-6 via Codex, with repository inspection, code execution, and
test tools. The runtime does not expose a more specific deployment
identifier or context-window limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked existing issues or described the issue in-PR
following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links
- [x] My branch name describes the change and contains no internal
ticket identifier
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 07:25:20 -05:00
DottaandPaperclip 5bf0142d45 fix(server): schedule Discord command registration from saved deadlines (#17557)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Chat connections register and verify provider commands in the
background.
> - Discord command recovery ran inside the recurring Gateway runtime
scan.
> - Its existing durable records already store verification and retry
deadlines.
> - This pull request gives command registration its own
shared-coordinator worker and scoped commit notifications.
> - Idle command work stops querying between real deadlines. Existing
command authority and recovery remain intact.

## Linked Issues or Issue Description

Refs #17554 and #17561. Both foundations have merged. This slice targets
master. Runtime discovery and the other chat maintenance duties remain
separate work.

**What happened?**

The chat runtime maintenance loop called Discord command registration on
every reconciliation. It repeatedly read durable command state even when
the next remote verification or retry was minutes away.

**Expected behavior**

Use committed producer changes and saved deadlines. Keep the existing
five-minute verification policy, Retry-After, uncertain-write
quarantine, namespace ownership, and credential fences. Do not add a
database table for the timer.

**Steps to reproduce**

Configure a Discord connection and let command registration settle. In
the parent revision, repeated runtime reconciliation still invokes
command recovery. This revision reconstructs command deadlines at
startup and removes that call from runtime reconciliation.

## What Changed

- Expose normalized deadlines from the existing registration store. Park
indefinite retries and invalid or foreign-owned records. Signal before
store-owned claims, receipts, and retry-result writes.
- Discover eligible endpoints in bounded pages. Refresh only affected
endpoint, connection, or company scopes after commits. Dispatch at most
25 due commands per pass with fair company rotation.
- Wire configuration, credentials, provider lifecycle/failure paths,
inbound activation, generic connection mutation/removal/health, access
revocation, and company deletion.
- Register the worker through the chat service and shared coordinator.
Preserve direct foreground setup calls. Refresh an existing local
Gateway through its current owner after command work.
- Preserve dirty work during drain and concurrent changes, back off
unresolved attempts, and join admitted provider work during shutdown.
- Document producer coverage, retained deadlines, supported writers,
verification failures, and remaining runtime/Cloud obligations in
`doc/idle-discord-command-registration.md` and its dated audit.

## Verification

Published head: `547fdf2139144e1e6484198278bf4bb0eade8794`. It normally
integrates master at `f8563d25077b17681ab6ef5949f7b07415a58ede`,
including landed health #17554 and company deletion #17561. Squash
ancestry caused merge conflicts; resolution applied this slice's exact
26-file owned patch from its recorded parent to pinned master. Every
other path matches master. Migration 0394 and the current lockfile are
preserved.

- **141 distinct targeted cases pass** on reconciled production source:
78 complete command-store, batching, coordinator and startup-timing
cases; 8 real automatic Discord and held-transaction Slack cases; and 55
complete startup/export and company-deletion cases. The 1,581 other chat
integration cases were deselected.
- Server typecheck and build pass. The company-deletion fixture
self-migrates an initially empty isolated database. The two Discord
fixtures use separate migrated clones; this is not a fresh-schema claim
for those fixtures. Signal-listener warnings remain recorded without
assertion failures.
- The parent integration preserves health and command notifications at
shared producer sites. It changes no provider behavior, schema or
coordinator contract. Prior producer, backlog, drain/resume,
malformed-hint, transaction, namespace, shutdown and retry regressions
remain in the complete command suite.
- [Landing compatibility
acceptance](https://github.com/paperclipai/paperclip/blob/547fdf2139144e1e6484198278bf4bb0eade8794/doc/idle-discord-command-registration.md#acceptance-after-the-company-deletion-parent-landed-october-27)
records current scope and the earlier failures. Older full-matrix or
combined-stack runs apply only to their stated revisions.
- Fresh exact-head Greptile, Ponytail if present and all CI are
required. No full local repository suite, live provider, Cloud alarm,
host suspension or savings claim is made.

## Risks

- Discord still has real five-minute verification and failure-retry
obligations. This does not prove host sleep or external alarm coverage.
- Runtime discovery, receipt-to-runtime projection, Gateway lease
renewal and persistent sockets remain separate. The remaining runtime
poll can still read command receipts; it no longer invokes the command
protocol.
- Process-local signals cover integrated writers. Independent SQL
writers and other replicas require an explicit wake contract. Existing
authorization and durable ownership checks remain authoritative.
- Refreshes coalesce with a 30-second minimum spacing. Saved work is
never discarded because a call returned without progress.
- No schema migration, new table, workflow or lockfile change is
introduced by this slice. Landed parent repairs are retained. The
receiver child must receive this integration before landing.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
reasoning. The exact serving version and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 23:32:24 -05:00
DottaandPaperclip 94a780286f fix(server): schedule tool health from committed changes and deadlines (#17554)
## Thinking Path

> - Paperclip manages AI-agent work and its connected tools.
> - Idle hosted instances need their database and network activity to
stop.
> - The maintenance tick queries tool connections even when no health
check is due.
> - Existing connection rows already contain eligibility and the last
health observation.
> - This pull request reconstructs health deadlines and refreshes them
after committed changes.
> - The shared coordinator preserves retries, drain, and shutdown
without another private timer.

## Linked Issues or Issue Description

**What happened?**

The aggregate maintenance tick scans connection health every 30 seconds.
Empty instances and connections with fresh health observations still
cause SQL activity.

**Expected behavior**

No eligible connection means no recurring health SQL. Eligible
connections retain the existing 15-minute proactive check. Configuration
changes and health observations update the schedule after commit.

**Steps to reproduce**

Start a server with heartbeat scheduling enabled and no tool
connections. Let startup finish and observe two maintenance ticks. The
old code repeats the health inventory query. The new startup regression
verifies that those ticks do not invoke the health inventory.

**Paperclip version or commit**

This branch includes merged parent #17518 and master
`96727b3cfefb61f85ffe1fbc29a9b6dd6837efe8`. The 47 changed paths belong
to the single tool-connection health duty. No database file differs from
master.

Related: #14638 covers runtime-auth exclusions. #15448, #15325, and
#11910 concern MCP failure classification. This PR preserves those
provider policies and does not replace their work.

## What Changed

- Keep the GitHub error-projection unit fixture at the health-write
boundary and verify the real transaction path in the PostgreSQL
health-worker suite.

- Add a health schedule owned by tool access. Read lean eligibility
fields and saved observation timestamps.
- Register health work with the existing coordinator. Refresh only dirty
connection/application/company scopes using bounded inventory pages,
check due connections in fair company batches, continue concrete due
backlog after one second, and retain failures with backoff.
- Notify before supported eligibility and observation writes in their
owning transactions. Cover API/gallery setup, OAuth, catalog/gateway
observations, email, voice, access removal, deletion, and settings.
- Commit connection creation with grant, binding, and runtime-slot SQL
before its wake. Keep provider calls outside the new transactions.
- Keep provider-to-transport mapping in the chat catalog. SDK-only chat
changes do not wake health work.
- Replace aggregate health scans, retain the public manual sweep, and
document the producer audit and remaining sleep limits.
- Keep metadata-only example and Smoke Lab installs scoped. Lock and
re-read application type before publishing a real eligibility change.
- Keep failed inventory reads on their own retry deadline while healthy
due batches continue.
- Apply fixture migrations before using external disposable databases in
all six producer suites.

## Verification

Published head: `8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62`. Fresh CI and
reviews are required for this exact head.

- **623 distinct targeted cases pass.** The final four complete health,
fair-batching, GitHub health recovery, and startup suites pass 80 cases
after the retry fix. Five complete producer suites pass 543 cases: tool
access 469, access 18, email 45, GitHub events four, and Smoke Lab
seven. The earlier 26 health cases are part of this total, not
additional cases.
- The six producer suites first passed against separate, initially empty
PostgreSQL 17 databases at `acbf262843`. They apply their own
migrations. The subsequent repair changes only the health worker and its
regression; its complete final consumer suites pass on that integrated
parent. Server typecheck and build pass with the final repair.
- **Installer amplification:** five real installs with 205 unrelated
connections reproduced 1,035 example rows and 1,045 Smoke Lab rows.
Repeated metadata installs now load only changed fixture connections.
The same cases restore an excluded application through each installer
and require all 205 unrelated IDs to re-enter the schedule.
- **Concurrent eligibility:** a real PostgreSQL lock-wait case requires
the installer to notify after a concurrent type change. A mutation that
retains the lock but uses the old lookup fails the settled-hint
assertion; original source was restored byte for byte.
- **Retry isolation:** the failing baseline read one unavailable company
three times while another company's 60 due checks drained. Final tests
require one initial read, at least 30 seconds between failed reads,
prompt healthy progress, eventual recovery, and SQL quiet after
settlement.
- Real-service producer coverage includes AgentMail activation/pause,
GitHub installation changes, and member access removal. Lifecycle
coverage includes outer commit/rollback, startup reconstruction,
drain/resume, failure, and shutdown.
- Failure history and exact evidence scope: [health
acceptance](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/idle-tool-connection-health.md#acceptance-follow-up-after-the-plugin-parent-merged-october-27),
[producer
audit](https://github.com/paperclipai/paperclip/blob/8c1d293d1ae2b7db753ab20f5a2a2edec1fb0a62/doc/plans/2026-10-24-tool-connection-health-sleep-audit.md).
Prior remote failures and earlier 706-case evidence remain historical.
Startup tests emitted listener-count warnings; all assertions passed. No
full local repository-suite, deployed sleep, fleet savings, or live
provider qualification claim is made.

## Risks

- Supported live writers must use the transaction signals. Independent
SQL writers and other replicas need their own wake integration.
- Configured eligible connections still have a real proactive health
obligation. This does not prove Cloud alarm coverage or host/database
suspension.
- Existing manual probes and catalog operations can overlap. This
migration does not introduce a credential-version CAS or change provider
failure policy.
- Voice remains eligible under the previous health policy. Its generic
probe can still report an unsupported transport; changing that behavior
is separate work.
- No database migration, new table, workflow, or lockfile change. This
PR includes its merged parent and current inspected master; later stack
PRs and all current-head acceptance gates remain separate.

## Model Used

OpenAI Codex, GPT-6 family, with repository tools, code execution, and
reasoning. The exact serving version and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 21:37:50 -05:00
DottaandPaperclip c3a9a8e92a perf(server): scope run identity reads to operation needs (#17931)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Tool and connection operations capture the current accepted
requester.
> - Capture locks the task and run, then reconciles pending steering.
> - Most callers use identity or task fields from the returned run.
> - The default query also transfers saved prompts, runner
configuration, results, and logs.
> - This pull request gives each production caller an explicit read
scope.
> - Narrow scopes keep their authority checks and return only selected
columns.
> - The Muse project path still reads its complete saved execution
input.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Operation-time run identity capture for native tools, gateway sessions,
project and workspace access, Slack, voice, connection management, and
connection token admission.

**Current behavior**

These callers use the default full run read. Large saved execution
documents cross the database connection even when the caller only needs
the current accepted requester or task binding.

**Proposed behavior**

Use typed identity, task, broker, and project-execution scopes. Keep
task-before-run locks, company and agent predicates, active status
validation, and pending-steering reconciliation. Return the complete
saved native input only where project admission uses it.

**Reason and benefit**

A synthetic PostgreSQL 17.11 fixture matches a wide run with a 142 KB
context and a 77 KB runner profile. Its locked run DataRow falls from
225,101 bytes to 190–605 bytes for authority reads. The project scope
returns 66,004 bytes because it must retain a complete 64 KB execution
input. These are local DataRow measurements, not a forecast of
production billing or latency.

**Breaking changes**

None to public APIs, saved data, or authority policy. Internal narrow
return types omit columns that were not selected. The default full
compatibility read and the existing GitHub-credential and authentication
scopes keep their current contracts.

**Additional context**

The public issue and PR duplicate searches found no matching open
change. This is one independent query optimization. It does not require
other performance PRs.

## What Changed

- Add typed scopes for accepted identity metadata, task authorization,
token attribution, and saved project execution input.
- Assign explicit scopes at all production `captureRunIdentity` call
sites.
- Keep the full native execution input for Muse project admission while
omitting unrelated profile, context, result, and log data.
- Narrow the broker's initial read and the existing Slack and
connection-management historical reads to the same required fields.
- Extend steering and acknowledgement regressions to all new scopes. Add
real PostgreSQL projection, malformed JSON, tenant and agent,
session-generation, and saved-input coverage.
- Add a service-generated SQL benchmark with exact raw DataRow
accounting, retained-field equality, plans, and alternating transaction
timings.
- Document the scope contracts and reproducible benchmark.

## Verification

- Baseline: `623f1dc289a14a699ebbe3f09b94179d38fc2ae2`.
- `vitest run server/src/__tests__/run-identity-projections.test.ts
server/src/__tests__/run-identity.test.ts`: 54 tests passed on an owned
disposable PostgreSQL 17.11 database.
- The tests retain task-before-run and audit foreign-key lock behavior.
Every scope rejects uncertain steering, accepts its durable
acknowledgement, and preserves requester replay ordering.
- A full-row mutation makes the narrow identity regression fail. The
restored treatment passes.
- `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-run-identity-projections.ts
--baseline-ref=623f1dc289a14a699ebbe3f09b94179d38fc2ae2` captures the
real locked SQL for both versions.
- Every retained column and selected JSON field matches the baseline.
Identity contexts match. The complete stored run remains unchanged.
Capture keeps the same five queries.

| Scope | Before DataRow bytes | After DataRow bytes | Reduction |
| --- | ---: | ---: | ---: |
| Identity | 225,101 | 190 | 99.92% |
| Task authorization | 225,101 | 410 | 99.82% |
| Token broker | 225,101 | 605 | 99.73% |
| Project execution input | 225,101 | 66,004 | 70.68% |

- DataRow bytes include row framing and raw text-format column values.
They exclude response metadata, TLS, and TCP.
- Two warmups and seven alternating samples measure the complete capture
transaction. Several timing ranges overlap. The primary claim is byte
reduction; production latency is not verified.
- Initial fixture runs exposed an empty external schema and an
incomplete synthetic conversation binding. The test setup now applies
migrations and supplies the required user/state fields. The final 54
tests passed.
- `pnpm -r typecheck` and `pnpm build` passed locally.
- Four existing skill-setup tests passed. Four assembled
connection-management/native-tool consumer cases passed with real
JWT/API-key authentication. These include per-request identity freshness
and task-scoped activity authority.
- A CI-only stale mock assertion expected two capture arguments; it now
verifies the explicit identity scope. A repository-wide capture mock
audit found no other stale argument assertions.
- Review follow-up: the projection suite uses the existing
embedded-support guard while always enabling an explicitly supplied
external database. The audit-lock regression merges URL query parameters
and retains existing PostgreSQL options.
- `vitest run server/src/__tests__/run-identity-projections.test.ts
server/src/__tests__/run-identity.test.ts
server/src/__tests__/company-skills-routes.test.ts
server/src/__tests__/project-skill-setup-identity.test.ts`: 123 tests
passed. The external PostgreSQL URL included both `application_name` and
`options`; lock tests verified `lock_timeout=1s` and retained
`statement_timeout=5s`.
- Full local tests are not run because the host's embedded PostgreSQL
setup is unavailable. Greptile reviewed
`06e1dd53a3fb24df74d5fbfb2ad397f291f1e30f` at 5/5 with zero unresolved
threads. The full final-head CI matrix passes. All 58 checks are
terminal: 55 success, one neutral, and two optional Storybook skips.
- No migration, lockfile, workflow, new polling, or cache change.

## Risks

A narrow read must retain each caller's actual fields. The caller audit
includes Slack's wake-request fallback, conversation generation, legacy
broker attribution, and the Muse parser's complete native input. JSON
selection retains value types, including malformed legacy values. The
saved-input scope preserves missing-key behavior. The default full read
still returns the complete stored row for compatibility. Locks and live
authority checks are unchanged.

## Model Used

OpenAI GPT-6, accessed through Codex. The exact runtime model ID and
context-window size were not exposed in this session. Used repository
inspection, code execution, and PostgreSQL validation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 16:31:21 -05:00
DottaandPaperclip d108aeeaaf fix(server): schedule pending tool-review expiry from committed changes (#17453)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Governed tools can wait for a human review before execution.
> - Pending reviews already save a deadline and must reject approval
after expiry.
> - The server currently scans for expired reviews on every maintenance
tick.
> - Empty scans keep the database active without advancing work.
> - This pull request schedules pending review expiry from committed
changes and saved deadlines.
> - Configured idle state and resolved history no longer cause this
expiry scan.

## Linked Issues or Issue Description

**What happened?**

The combined tool review sweep repeatedly queries pending expiry even
when no pending request has a deadline.

**What did you expect?**

Signing or resolving a request should update the expiry schedule. A
saved expiry should trigger settlement. Once no expiry remains, this
duty should stop querying.

**Steps to reproduce**

Start a claimed instance with heartbeat scheduling enabled. Leave it
idle with no pending tool reviews. Observe pending-review expiry
discovery in the maintenance sweep.

Stacked on #17450; earlier slices are #17446 and #17443. These all reuse
the delivery and transaction foundations from #15625. This PR owns
pending-review expiry only. Approved-action recovery,
interrupted-execution quarantine, and terminal interaction projection
repair remain separate migrations. No duplicate pending-review expiry
migration was found in the open PR search; credential lifetime changes
in #17221 and #17274 concern the parent credential slice.

## What Changed

- Replace repeated company review-deadline aggregates with an indexed
single-row lookup. Add and replay a concurrent covering-index migration
and generate combined snapshots throughout the stack.

- Extract pending-expiry selection and settlement into its owning
service. Keep provider execution and interaction publication with the
gateway.
- Signal deadline assignment and pending resolution inside the owning
SQL transaction. Preserve outer commit/rollback and central uncertainty
handling.
- Register company expiry deadlines with the shared scoped coordinator.
Reconstruct on startup; normal producer hints reload only their company.
Preserve writes during drain and concurrent reconciliation.
- Settle request expiry and invocation failure atomically. Keep
interaction projection recovery in the existing separate sweep.
- Remove pending-expiry scanning from the main tick. Preserve the
combined manual API and startup-only behavior when scheduling is
disabled.
- Add a partial pending-expiry index through an idempotent concurrent
migration. No queue table is added.
- Document supported producers, quiet states, remaining duties, and
external wake limits.

## Verification

- CI run `40574661575` at `f5c935d1b89877033eb82bf890f867baeb372ef3`
lost the runners for general-server shard 9/12 and E2E shard 2/8 at
20:09:03 UTC. Both job logs explicitly show a runner shutdown signal
followed by cancellation; no test assertion failure was reported by
those jobs. A single infrastructure retry is justified once the run
completes; it has not been issued yet. Both current-head reviews are
5/5, and CI must still be green before merge.

- Current master landing at `f5c935d1b89877033eb82bf890f867baeb372ef3`:
128 complete gateway expiry, startup, scoped deadline and
preparation-timing cases pass with no skips. Server types and the
production server build pass. Landed tracing changes from #17918 are
preserved. The master migration journal prefix remains semantically
identical, with only this PR's two migrations added. Fresh remote
CI/review is required before merge.

- Parent reconciliation at `17539e2a6cf8ac4922b3cbe06af6df361bfe5665`:
116 tests pass across the full tool gateway service, startup and scoped
deadline suites against a dedicated migrated PostgreSQL database. No
cases were skipped. Server typecheck passes. Fresh remote CI and review
are pending.

Current head: `ec28076f825e76a267f05517ff0c44a3e2283ce7`. This remains a
single-duty PR stacked on #17450.

- Integrated pinned master `8548173fc9efdc0fa07644fb92dcc0b6851a95e5`
with normal merges. Every branch passed generated-schema checks and its
target-branch migration-order gate. Existing SQL is unchanged; the 25
stack migrations follow master’s new migration.
- Duty-specific producer coverage, lifecycle tests, and earlier failure
records:
[2026-10-24-recurring-server-sleep-stack.md](https://github.com/paperclipai/paperclip/blob/ec28076f825e76a267f05517ff0c44a3e2283ce7/doc/plans/2026-10-24-recurring-server-sleep-stack.md).
- Combined tip `51e6cbe67979b55b340ea9f0f434c5b5c682ccd6`: **355
startup/cancellation/recovery/dispatch/preservation tests**, **75
connection-management tests**, and production server typechecking pass.
These combined checks do not mean every intermediate PR independently
ran the full repository suite.
- Real master-to-stack database upgrade passes: **379 unchanged prior
migration receipts plus 25 new migrations**, with the gateway
responsibility fixture preserved.
- Owning-branch checks: 84 runner installation/settlement cases plus
runner typechecking and generated TypeScript/Rust profile agreement; 253
GitHub review workflow cases. Stop-attribution and held-worker readiness
regressions failed before their fixes.
- Failure accounting: the initial combined run passed 349 cases and
failed six idle assertions in a reused preservation database containing
old cancelled runs. Repeating the combined command with a fresh
preservation database passed. Earlier invalid project selection, merge
import/type errors, and the interrupted full local suite are not counted
as passing checks.
- **Latest-head CI and review remain required.** Prior CI findings
include GitHub idle observation, route teardown, attention cleanup,
touch-picker E2E, workspace health, and voice-cleanup timing; local
passes do not erase those failures. No full local repository-suite,
deployed sleep, fleet cost, or live Copilot qualification claim is made.

## Risks

- All supported producers must share the database signal owner.
Independent SQL writers and other replicas need a live notification
contract; startup alone does not cover them.
- A crash after authoritative expiry and before interaction projection
still relies on the existing projection-repair duty. That duty,
approved-action recovery, and uncertain execution recovery remain
periodic in this slice.
- Unknown external outcomes remain quarantined. Provider calls stay
outside SQL transactions. Expiry is still checked before approving a
request even if physical cleanup waits during drain.
- The new index uses the existing top-level migration executor because
PostgreSQL concurrent index creation cannot run in a file-wide
transaction. Snapshot pruning follows the repository rule.
- This establishes worker quiet and reactivation, not deployed host
suspension. Existing Cloud idle-safety and external alarm coverage
remain unchanged.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning and code execution. The exact
serving model identifier and context window were not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Latest-head CI is now green after one evidence-backed retry. Greptile is
5/5 and all review threads are resolved.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 15:26:19 -05:00
DottaandPaperclip 6e781e0ab2 fix(connections): preserve removed grants when saving installs (#17906)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connections give agents access to external tools through access
profiles.
> - Operators can remove a broad app profile and keep a limited profile.
> - Saving a connection's agent list restored the broad profile to
existing installation targets.
> - Those restored grants could permit write tools that the limited
profile omitted.
> - This pull request grants the app profile only to new installation
targets.
> - The benefit is stable permission choices when an operator edits an
installation list.

## Linked Issues or Issue Description

**What happened?**

`PUT /api/tool-connections/{connectionId}/installs` attached the app
profile to every target in the submitted list. An existing target
received the profile again if an operator had removed it. Adding an
agent, removing another agent, or saving the same list could therefore
restore broad tool permissions. The activity log recorded each restored
binding as `tool_connection.install_access_extended`.

**Expected behavior**

Targets that remain installed keep their saved profile assignments. A
removed app-profile assignment stays removed. New installation targets
receive the current app-profile defaults. Removing an installation still
removes its install-derived binding.

**Steps to reproduce**

1. Install a connection with read and write tools on two agents.
2. Give one agent a read-only profile and remove its broad app-profile
assignment.
3. Save the installation list, add an agent, or remove the other agent.
4. Check the first agent's permission to use the write tool.

Before this change, the write tool becomes allowed again. After this
change, it stays denied.

Related: #17422 adds an agent-edit permission check to direct profile
bind and unbind operations. This PR preserves removed assignments during
installation-list updates. The duplicate search found no open PR for
this defect.

## What Changed

- Attach app-profile bindings only for new installation targets.
- Preserve removed grants for existing agent and company installations.
- Keep legacy connection-wide include cleanup on every non-empty
installation save, including saves with no added targets.
- Add six PostgreSQL regressions. All six use the real install HTTP
route with a local operator fixture.
- Check effective read/write decisions, saved assignments, new-agent
defaults, uninstall cleanup, legacy-rule cleanup, and access-extension
audit records.
- Document the behavior in the connector runbook and both access maps.

## Verification

- Red: all four new regressions fail before the fix. The policy returns
`allow_profile` where the tests require `deny_default`.
- Review regression: the two legacy-rule cases fail before the review
correction. The original four cases still pass.
- Green after review: all six regressions and five existing
profile-precedence tests pass after the correction.
- Command: `pnpm exec vitest run
server/src/__tests__/tool-connection-installs.test.ts
server/src/services/tool-profile-binding-precedence.test.ts`.
- The local regression used `PAPERCLIP_TOOL_ACCESS_TEST_DATABASE_URL`
with a separate, fully migrated PostgreSQL 17 test database. Embedded
PostgreSQL could not start because the host had reached its semaphore
limit. Skipped tests were not counted as a regression pass.
- `pnpm -r typecheck` and `pnpm build` passed for the initial fix.
Server `tsc --noEmit` also passes after the review correction.
- `node scripts/check-maps.mjs`, `node
scripts/check-module-boundaries.mjs`, and `git diff --check` pass.
- `pnpm test:run` was started, then stopped after 16 tests in the
native-session suite failed during embedded PostgreSQL startup. A
focused rerun of its `collect` case confirmed the database startup
error. Direct `initdb` output reported `could not create semaphores: No
space left on device`. The full local suite is incomplete. All six new
regressions executed against the separate test database and passed.
- [Initial-fix
CI](https://github.com/paperclipai/paperclip/actions/runs/40541191695)
passed at `23253ee332b90f142e48df3f553aeaaa7f6fa298`. Greptile gave that
commit 5/5. Ponytail found the legacy cleanup gap; this revision
restores that cleanup and adds both regression cases.
- [Current-commit
CI](https://github.com/paperclipai/paperclip/actions/runs/40547366181)
passes for `729c487cf2ffaae261a820de0cf9a033a0365fd4`, including type
checks, build, test shards, and both `ci / verify` and `ci / e2e`
aggregate checks.
- The first CI attempt hit a 30-second module-import timeout in the
unchanged `issue-agent-mutation-ownership-routes.test.ts` suite. All 179
tests passed locally. One [job
retry](https://github.com/paperclipai/paperclip/actions/runs/40547366181/job/122349226221)
passed without a code change. The cause of the import slowdown was not
established. The failed attempt remains recorded in CI history.
- Greptile and Ponytail both give the current commit 5/5 with no
actionable findings. Ponytail confirms the legacy cleanup finding is
fixed. The review thread is resolved, and the PR has no merge conflicts.
Ponytail's advisory check has a neutral conclusion with a 5/5
assessment.

## Risks

- New installations retain the current app-profile defaults. Removing
and then reinstalling a target is a new installation.
- A repeated installation save no longer restores a missing profile
assignment. Operators must use an explicit profile assignment when they
want to restore that grant.
- Existing unwanted grants still require a separate access review and
repair. This PR does not change stored grants, reconnect identity
behavior, or the default access model.
- No database migration or API schema change is required.

## Model Used

- OpenAI GPT-6 in Codex, with reasoning, repository tools, and code
execution. The session does not expose the exact routed model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 12:25:01 -05:00