Commit Graph
94 Commits
Author SHA1 Message Date
DottaandClaude Fable 5.2 33fac97cad feat(routines): ship the redesigned routine page and let live webhooks be tested again (#22205)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Routines start agent work on a schedule, from a webhook, or on
demand
> - The routine detail page split one routine across eight sections
behind a section sidebar (overview, triggers, variables, secrets,
delivery, runs, activity, history)
> - People had to hunt through sections to see what a routine does and
to change it, and a finished webhook could not be tested again without
starting the routine
> - This pull request makes the approved two-column routine page, one
Edit dialog and one Webhook dialog the real routed page, and lets a live
webhook go back into test mode
> - The benefit is one page that answers what the routine does, when it
runs, who runs it and what it made, with every change in one dialog

## Linked Issues or Issue Description

**Problem**
The routine detail page spreads one routine over eight sections and a
section sidebar. Changing a routine means visiting several sections,
each with its own save bar. A webhook that is already live cannot be
tested again: the API only accepts `setupPending: false`, so testing a
live webhook starts the routine.

**Proposed solution**
Ship the reviewed design (stories `routine-page-design`,
`routine-edit-dialog-design`, `routine-webhook-dialog-design`) as the
routed page in both UI shells, wired to the real API. Let `PATCH
/api/routine-triggers/:id` accept `setupPending: true` for a live
webhook so the next events are checked but never start the routine.

**Alternatives considered**
Keep the sections and restyle them. The owner rejected this: the section
sidebar should go.

## What Changed

- `/routines/:id` renders `RoutinePage` in both shells
(`RoutineDetail.production.tsx` is removed). Status switch, Repeats
rows, default agent, responsible user, project, Asks for, Secrets, Run
rules, Managed by plugin, Instructions with Show all, copy and the
comment count, and History.
- History is paged: 20 tasks per page from `GET
/companies/:id/issues?originKind=routine_execution&originId=…&sortField=created&sortDir=desc&limit=21&offset=…`,
with a real Load more. Coalesced runs show as "folded into a task".
- A failed refetch keeps the loaded page on screen (`useQueryView`); a
History failure shows `QueryErrorState` in place.
- Header: Edit, "…" (Version history, Audit log, Archive / Restore) and
Run now. In the streamlined desktop shell they sit in the breadcrumb
toolbar; elsewhere above the page.
- Version history (`/routines/:id/history`, restore and restored-webhook
keys kept) and Audit log (`/routines/:id/activity`,
`RoutineAuditActivity`) open as sub-views with a way back. Old section
addresses and `?tab=` redirect to `/routines/:id`.
- Edit dialog saves title, instructions, variables, secrets, agent,
project and responsible user in one request against the revision the
dialog opened with (save conflict shows "changed while you were
editing"). The first schedule is created, changed or removed after that.
Priority and delivery settings are never sent, so saved values stay.
- Webhooks are created, changed, tested and removed (with Undo) right
away from the Edit dialog, with the Webhook dialog nested inside it.
Closing the Webhook dialog ends setup or test mode. A webhook left in
test mode shows "Testing · events don’t run" on the page and offers
Finish testing.
- Responsibility (only with `enableResponsibilities`): the page shows
direct and through-project responsibilities from the link lookup; the
dialog links or unlinks the direct one. With the flag off nothing is
fetched or shown.
- The routines list creates routines with the same dialog in create
mode, filed in the open folder, and opens the new routine page.
- Server: `setupPending: true` on a live webhook enters test mode
(planned by a pure `planTriggerUpdate`, which also gives the activity
action) (422 for other trigger kinds; 409 for a disabled or removed
webhook or an archived routine). Test deliveries keep the last real
delivery as `lastWebhookDelivery.previous`, and finishing testing
restores it. Activity `routine.trigger_test_started`.
- Additions to the reviewed design, from review, so no capability is
lost: the schedule popover has Pause/Resume; standing-check and API rows
open a small popover with Pause/Resume and Remove; removing a trigger
from the page leaves a "… removed. Undo" line under Repeats. Stories:
`StandingCheckOpened`, `ScheduleTriggerPaused(Opened)`,
`RemovedWithUndo`.
- Save is several requests (routine, schedule, responsibility). It
changes only the schedule the form opened with, records each finished
step, and a partial failure says what was saved and that saving again
finishes it. The dialog cannot be dismissed while saving. Failed reads
show a Retry state, never empty choices.
- Removed: the routine section sidebar and contextual sidebar, the
trigger wizard, trigger cards, save bar, section components and their
tests.
- Storybook: the three routine stories moved to `stories/routines` as
current production stories, a new story mounts the routed page against a
fixture API, and stories that used the old page were updated or removed.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/routines-e2e.test.ts` (17/17) and
`src/__tests__/routines-service.test.ts` (105/105).
- `pnpm exec vitest run` on the routine page, routines list, routine
form, webhook instructions, Layout, shell navigation, audit and editor
tests: 275 tests pass.
- `cd ui && npx tsc -p tsconfig.json --noEmit`, `node
scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`, `node
scripts/storybook-catalog.mjs --check`: pass.
- `pnpm verify` exits 2 (global CI input changed); required CI supplies
the broad coverage.
- Manual: an isolated worktree instance (`pnpm paperclipai worktree init
--empty`, `pnpm dev`) with a seeded routine. Headless Playwright: an old
`/triggers` address redirects, no section sidebar, History shows 20 then
23 after Load more, the schedule popover opens, pause and resume, Edit →
Add webhook → created key → Finish, title save, Test webhook → Finish
testing, Version history, Audit log shows "Started testing a webhook",
phone width has no horizontal overflow, and Create routine from the list
opens the new page. No console errors or failed API calls.

## Risks

- Behaviour change: the section pages are gone. Old links redirect to
the single page.
- The delivery-setting editors (concurrency, catch-up, activity gate)
and priority are removed from the UI. Saved values stay, and the API
still accepts them.
- The create dialog no longer offers a folder picker or delivery
settings. A new routine goes into the folder the list is showing and
takes the default settings.
- Replace key in the Webhook dialog acts at once, as in the reviewed
design. The old confirmation is gone.
- No migration. `lastWebhookDelivery.previous` is an optional field
inside the existing jsonb column.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, headless browser) and subagents for the
server change and the Storybook move.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 13:13:15 -06:00
651749bdb4 feat(connections): AWS connector with credentials minted per connection (#22195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use outside services through connections, and the control
plane holds each connection's credentials
> - Companies want agents to "do things on AWS" with exactly the
permissions they choose, without Paperclip storing long-lived AWS keys
or rebuilding the AWS API
> - AWS hosts an MCP server that runs any AWS API call in its own
sandbox, and it accepts a one-hour bearer token that can be minted from
IAM credentials
> - This pull request adds an AWS connection that mints that token for
each connection at call time. The role method proves the connection with
the company's workload identity issuer (#22083), so no AWS key is stored
> - The benefit is that each AWS connection carries its own role and
permissions, AWS enforces them, and agents never see any AWS credential

## Linked Issues or Issue Description

Refs #22083 (workload identity issuer) and #22191 (spike findings).

**Subsystem affected**
Connections (tool access), tool gateway, shared app definitions,
instance settings.

**Problem or motivation**
Paperclip has no way to connect AWS. AWS's hosted MCP server needs a
bearer token from AWS Sign-in, minted from IAM credentials. No existing
credential path mints a header at call time from a role or key.

**Proposed solution**
Methods can declare `credentialMinter: "aws_mcp"`. The gateway and
connection health checks mint the `Authorization` header at call time,
and they share one in-memory cache per connection and grant (about 55
minutes). There are two methods:
- Role (`AssumeRoleWithWebIdentity` with a Paperclip-signed connection
token).
- Access key (the key directly, `GetFederationToken`, or `AssumeRole`).

Read-only connections add `ReadOnlyAccess` and
`AWSMCPSignInOAuthAccessPolicy` as session policies, so AWS refuses
changes. The feature is behind the `enableAwsConnector` experimental
setting.

**Alternatives considered**
- Run AWS's `mcp-proxy-for-aws` process on the control plane. Rejected:
it adds a Python runtime and processes, and gives nothing the token
exchange does not.
- Put credentials in the agent sandbox and use the AWS CLI. Deferred:
there is no per-call governance, and every sandbox would need the CLI.
- Add a new `credentialSource`. Rejected: it needs a migration, and the
`vercel_connect` database rules do not fit stored access keys.

## What Changed

- `packages/shared`:
- `credentialMinter` on connection methods. The validator requires
`mcp_remote` and `api_key`, and does not allow `keyPlacement` or another
credential source.
- The `aws` app definition (two methods, reviewed tools,
`mcpSessionRequired`, experimental key).
- The `enableAwsConnector` setting in the catalog, types and validators.
  - `aws` added to the connectable apps.
- `server/src/services/connectors/aws/mcp-credentials.ts`:
- Settings parsing and checks. The URL must be AWS's endpoint for the
saved region, and the role ARN must be valid.
- The minter: role or key session, read-only session policies, Sign-in
exchange, cache keyed on settings, key references and status, in-flight
de-duplication, eviction.
- A plain-language error mapping that never shows AWS's own text,
because that text can contain account IDs and ARNs.
- Gateway (`resolveCredentialHeadersUnrecorded`), credential access
(`resolveCredentialHeaders`), catalog discovery and health-failure
mapping:
  - The AWS branch.
- After a 401, the gateway evicts the token and mints again, then
requires an explicit retry (no replay). Discovery lists again.
- Setup:
- Minted methods may save a draft without contacting the provider. The
role must trust a connection that already exists.
- Saving an AWS role draft publishes the company's issuer, so AWS can
validate it when the customer registers it.
- `workload-identity.ts`: a draft connection can get a token, so setup
can test it. Disabled, archived or switched-off connections still get
none.
- `server/package.json`: `@aws-sdk/client-sts` and
`@aws-sdk/client-signin`. The lockfile is left to the refresh workflow.
- Artwork: AWS light and dark marks from LobeHub's MIT icon set (already
licensed in the repo). The manifest entry passes the brand checks.
- `scripts/ingest-app-definitions.mjs`: minted methods are exempt from
the `keyPlacement` requirement. The override source is
`scripts/app-definition-overrides/aws.json`. Permission reviews are
added.
- UI: the experimental settings toggle, in alphabetical position.
- Docs: `doc/connections/AWS.md`, the connections index, the issuer API
doc (draft rule), and the plan's phase table.

## Verification

- `server/src/services/connectors/aws/mcp-credentials.test.ts` (7 tests)
covers these items:
  - Both methods and the read-only policies.
  - Caching, revision, forced refresh, expiry and eviction.
  - Concurrent de-duplication.
  - The error mapping, with no AWS text leaking.
- Settings validation, including the URL and region check and the rule
that an unknown access level means read only.
- `server/src/__tests__/aws-mcp-connector.test.ts` (3 tests, real setup,
catalog and gateway code with a fake AWS and a session-requiring fake
MCP server) covers these items:
  - The setting gates setup.
  - The access key is never sent as a header or stored in config.
  - A cached token is reused.
  - After a 401, a fresh token is minted and the call is not replayed.
  - Role draft-then-resume publishes the issuer.
  - The signed token has the right `iss`, `sub` and `aud`.
  - An AWS refusal shows a plain health message without the account ID.
- Red/green:
  - Removing the gateway hook fails the access-key test.
  - Turning off `mcpSessionRequired` fails both integration tests.
- Removing draft support for minted methods fails both integration
tests.
  - Restoring each change passes again.
- Nearby suites: `tool-gateway-service`, `tool-access-service`,
`remote-mcp-connectors`, `instance-settings-service`,
`workload-identity`, plus the new files. 775 of 775 pass.
- Shared `app-definitions.test.ts` passes (42), including new
`credentialMinter` schema tests. UI `InstanceExperimentalSettings` (63)
and the brand asset tests pass. `check-app-brand-assets` validates 112
identities.
- Typecheck is clean for `packages/shared`, `server` and `ui`. The only
UI errors are in Storybook's generated catalog files, which this
worktree has not built.
- **Real AWS** (Paperclip dev account, integrated code path, issuer
behind a temporary public tunnel):
- Access key, read only: 8 tools listed. `run_script` `ListBuckets`
succeeded in 8.9 s on a cold sandbox. AWS refused `CreateBucket` ("no
session policy allows").
- Role: the draft was saved without contacting AWS, and the issuer was
published. After an IAM provider and a role trusting only that
connection were created, resuming setup listed 8 tools. `ListBuckets`
succeeded in 12.8 s. The role's own permissions refused `CreateBucket`.
- This run found two bugs the fakes did not: AWS needs MCP sessions for
calls, and the role method needs draft-first setup with the issuer
published. Both are fixed and both now have tests.
- All IAM users, keys, roles, providers and buckets were deleted, and a
listing confirmed it.
- Not run: Paperclip Cloud staging (the setting is off by default), and
`AssumeRole` from an access key with a role ARN against real AWS (unit
tests only).

## Risks

- The feature is off by default, behind `enableAwsConnector`. No
migration.
- New dependencies `@aws-sdk/client-sts` and `@aws-sdk/client-signin`
are in the existing `@aws-sdk` family (the server already uses
`client-s3`).
- Minted tokens are held only in process memory. A restart mints again.
A cached token can outlive a deactivated key by up to an hour, unless
AWS returns 401, which evicts it.
- `aws___run_script` is classified as a write, so the normal approval
and ask-first policy applies. Read-only connections rely on AWS session
policies.
- The guided setup screen and the CloudFormation template are not in
this PR. Setup uses the generic form and the documented draft and resume
steps.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, AWS SDK for JavaScript against the Paperclip dev AWS
account, a Cloudflare quick tunnel, an exploration subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 12:48:49 -06:00
DottaandClaude Opus 5.5 f0bd99ba8a fix(recovery): wait out a sandbox the provider could not start, with no Retry (#22194)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - Sometimes the provider creates the sandbox but cannot start it.
Daytona then puts the sandbox in its `error` state
> - The Daytona plugin deletes that failed sandbox and rethrows the
provider error. The error has no HTTP status and no errno, so the host
cannot tell that it is a provider-side failure
> - The run fails with `setup_failed`, and the task rests on "Retry when
ready". No agent work had started, and a fresh sandbox works, so a
person must click Retry for something the system can do itself
> - Recovery must just work, and the target for runs that need Retry is
0%
> - This pull request makes the plugin report a removed sandbox that the
provider could not start as structured evidence (the step and the
provider state). The host classifies it as a transient provider failure.
The task then uses the existing bounded retries and the provider-outage
wait from #22139
> - The benefit is that a provider start failure delays the task by
seconds instead of stranding it on Retry

## Linked Issues or Issue Description

Refs #22139 (the provider-outage wait lane that this failure now
reaches). Refs #22128 (setup failure after work; this failure happens
before any work, so that path did not apply).

**What happened?**
In a restart-chaos wave with graceful release swaps on 2026-11-18, two
tasks started a run at 14:51:15Z and 14:51:24Z. Both runs failed at
14:51:53Z with `setup_failed` and `failureCause: lease_acquire_failed`.
The cause was `Sandbox <id> failed to start with status: error, error
reason: null` from the Daytona SDK, for two different sandboxes at the
same moment. Both tasks rested on Retry. A Retry click started each task
again, and both tasks completed.

**Expected behavior**
A sandbox that the provider cannot start is a provider failure before
any agent work. Paperclip deletes it and tries again with a new sandbox,
with no Retry. If the provider keeps failing, the task waits for the
provider without a Retry, and a person gets a working Retry only after
the 30-minute ceiling.

**Steps to reproduce**
1. Make the Daytona create call fail while the new sandbox is in the
provider's `error` state.
2. Assign a task to an agent that uses that environment.
3. The run fails with `setup_failed`, and the task rests on Retry.

## What Changed

- Plugin SDK (`environment-creation-cleanup.ts`,
`environment-acquire-error.ts`, `index.ts`): new
`PluginEnvironmentAcquisitionError`. It carries the acquisition
diagnostic (step, elapsed time, budget, provider state) for a failed
allocation that the provider already removed, with no cleanup envelope.
The acquire RPC error data (`paperclip/environment-acquire-error/v1`)
now carries this diagnostic. `readEnvironmentAcquisitionDiagnostic`
reads it from that envelope. The existing cleanup envelope path is
unchanged. No provider text crosses the RPC.
- Daytona plugin (`plugin.ts`): `destroyFailedCreation` returns the
state of the sandbox that it deleted. When a create fails without an
HTTP status and the deleted sandbox was in the `error` state, the plugin
throws `PluginEnvironmentAcquisitionError` with `providerState:
"error"`. A `build_failed` sandbox (a deterministic snapshot build
failure) and any other state keep the original error.
- Server (`environment-runtime.ts`): the acquire path records the
diagnostic for this run also when the error has no cleanup envelope. A
cleanup envelope for another run, company or environment still records
nothing.
- Server (`environment-provider-unavailable-error.ts`): an acquisition
diagnostic with `providerState: "error"` is the transient reason
`provider_start_failed` ("the provider could not start the sandbox"). It
becomes `EnvironmentProviderUnavailableError`, so the run uses the two
bounded retries and then the provider-outage wait lane (one probe per
environment, 30-minute ceiling).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed SDK sources (generated by `node
tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`).

Module boundaries: the change to the shared SDK contract is additive.
The Daytona plugin owns the provider state. The host reads only the
closed diagnostic fields. A plugin that does not use the new class
behaves as before.

## Verification

- `packages/plugins/sandbox-providers/daytona`: `npx vitest run
src/plugin.test.ts -t "failed sandbox creation cleanup"` (37 passed).
New: "reports a sandbox that failed to start as a removed allocation
with its observed step" (red before the plugin change, green after) and
"keeps the create error when the removed sandbox was build_failed /
started".
- `packages/plugins/sdk`: `npx vitest run
tests/environment-acquire-error.test.ts
tests/environment-creation-cleanup.test.ts` (50 passed). New: the
diagnostic crosses the real worker RPC without provider text, and forged
or unstructured envelopes give no diagnostic.
- `server`: `npx vitest run
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts` (74 passed).
New: the reason is `provider_start_failed` with a transient provider
failure, and a diagnostic that was not recorded for this run is not
retried.
- `server`: `npx vitest run src/__tests__/environment-runtime.test.ts -t
"provider could not start|retains uncertain plugin creation"` (12
passed). New: "records the step of a sandbox the provider could not
start and removed" for a worker error (red before the runtime change,
green after) and for a forged error (not recorded).
- `server`: `npx vitest run
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/heartbeat-process-recovery.test.ts -t
"provider|acquisition|diagnostic"` (167 passed).
- Typecheck: `server` `tsc --noEmit`, `pnpm run typecheck` in
`packages/plugins/sdk` and in the Daytona plugin all pass.
- `pnpm verify` exits 2 (the SDK input registry is a global CI input).
Required CI supplies the broad coverage.

## Risks

- **Automatic retries for a deterministic start failure.** If a provider
image or configuration made every sandbox go to `error`, the task now
waits in the provider-outage lane instead of resting on Retry at once.
The lane uses one probe per environment on a 1, 2, then 5 minute
backoff, and it hands the task to a person with a working Retry after 30
minutes. The aggregate automatic-run brake is unchanged. `build_failed`
is not retried by this change.
- **Shared SDK contract.** The new field is additive. An older host
ignores it, and the run fails as before.
- No database migration. No UI change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:47:33 -06:00
DottaandClaude Opus 5.5 3c460f20cc feat(analytics): ask EU, UK and Swiss visitors before Cloud browser analytics (#22186)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud can send browser product analytics and session
replays to PostHog. Self-hosted installs never do this.
> - Today, browser analytics starts for every Cloud visitor after the
page settles.
> - Visitors in the EU, the EEA, the UK and Switzerland must agree
before analytics cookies and session recordings start.
> - This pull request adds a consent step for these visitors and a
switch in the user's profile settings.
> - The benefit is that Cloud analytics follows the owner decision: ask
in Europe, keep the current behavior elsewhere, and let every user
change the choice.

## Linked Issues or Issue Description

**Subsystem affected**
Paperclip Cloud browser product analytics (`ui/src/lib/analytics`,
`ProductAnalyticsGate`, the session route, and Profile settings).

**Problem or motivation**
Browser analytics and session replay start for every Cloud visitor
without a choice. EU, EEA, UK and Swiss visitors must consent first. The
privacy policy draft also says that users can change the choice in
settings. No such setting exists.

**Proposed solution**
Ask visitors in the EU, the EEA, the UK and Switzerland with a small
bottom banner (Accept or Decline). Do not load posthog-js before they
accept. Treat Do Not Track and Global Privacy Control as a decline for
these visitors. Keep the choice in `localStorage`. Add a "Product
analytics" switch to Profile settings for every user when browser
analytics is on.

**Alternatives considered**
A server-side country from a trusted edge header. Paperclip Cloud has no
such header today: Cloudflare records are DNS-only, so no `CF-IPCountry`
arrives, and the Railway edge adds no country header. This PR supports
an optional header (`PAPERCLIP_POSTHOG_COUNTRY_HEADER`) for a future
edge, and uses the browser time zone until then.

## What Changed

- `packages/shared`: `browserAnalytics` has a new optional
`consentRequired: boolean | null`.
- `packages/shared/src/analytics-consent.ts` (new): the one region
policy. `countryRequiresAnalyticsConsent` (EU and EEA with outermost
regions and Svalbard, GB with the Crown Dependencies and Gibraltar, CH)
and `timeZoneRequiresAnalyticsConsent` (`Europe/*`, EU, EEA and UK zones
outside Europe, and unknown zones such as `UTC` or `Etc/*`). The server
and the UI both use it.
- `server`: `resolveBrowserAnalyticsConfig` sets `consentRequired` from
the header that `PAPERCLIP_POSTHOG_COUNTRY_HEADER` names. No header or
an unknown code (`XX`) gives `null`.
- `ui/src/lib/analytics/consent.ts` (new): decides `granted`, `denied`
or `ask`. Order: for a visitor who must consent, Do Not Track or Global
Privacy Control gives `denied` (also over a stored Accept); then the
choice made on this page; then the stored choice; then the region
(server value, else browser time zone). The choice is in `localStorage`
key `paperclip.analyticsConsent`. All storage access is in `try/catch`.
A cross-tab `storage` event updates the page.
- `ProductAnalyticsGate`: one effect calls `syncProductAnalytics(config,
consent)`. Shows `AnalyticsConsentBanner` when consent is `ask`. Accept
loads posthog-js at once, without a reload.
- `product-analytics.ts`: new `syncProductAnalytics`. It starts, opts
in, opts out (`opt_out_capturing()` and `stopSessionRecording()`) or
cancels a pending load. Any consent other than `granted` stops
analytics. If consent is withdrawn while the script downloads, `init()`
does not run until a later opt-in.
- `AnalyticsConsentBanner` (new): a labelled bottom region with plain
copy, a privacy policy link, and Decline and Accept buttons (shared
`Button`). The copy says that recordings hide what you type and your
messages, documents and connected accounts, but may show page names like
task titles. It does not block the page. On mobile it sits above the
bottom navigation.
- `ProductAnalyticsSetting` (new) in Profile settings: a "Product
analytics" switch (shared `ToggleSwitch`). It shows only when browser
analytics is on for the instance. When GPC or DNT blocks analytics, it
is off and disabled and says why.
- `doc/cloud-browser-analytics.md`: new Consent section (signal,
behavior table, storage key, how to reset) and the new variable.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed for the
changed shared validator.

## Verification

- Red/green: the 7 new `ProductAnalyticsGate` consent tests fail against
the master gate (7 failed, 3 passed). The 5 review-fix tests (failed
write with working read, cross-tab change, decline during script
download, granted -> ask from another tab, granted -> ask from a session
refresh) fail on the first commit (5 failed, 16 passed). All pass now.
- `cd ui && npx vitest run src/lib/analytics
src/components/ProductAnalyticsGate.test.tsx
src/components/AnalyticsConsentBanner.test.tsx
src/components/ProductAnalyticsSetting.test.tsx
src/pages/ProfileSettings.test.tsx`: 267 tests pass.
- `cd server && npx vitest run
src/__tests__/cloud-posthog-browser.test.ts
src/__tests__/auth-routes.test.ts
src/__tests__/auth-session-route.test.ts`: 87 tests pass.
`packages/shared` `analytics-consent.test.ts` passes. `tsc --noEmit` for
`ui`, `server` and `packages/shared` shows no new errors.
- `pnpm verify` exits 2 (a doc is a global CI input). Required CI gives
the broad coverage.
- Real check in headless Chromium. I ran a local Cloud-managed instance
(external local PostgreSQL, `PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN`,
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`). `PAPERCLIP_POSTHOG_HOST` pointed
at a local capture server, not PostHog. The capture server served a
small stand-in for `array.js` that sets a `ph_*` cookie and
`localStorage` key and posts `/e/` events. I set the visitor region with
the browser time zone:

| Visitor | Result |
| --- | --- |
| `Europe/Berlin`, no choice, 18 s wait | Banner shows. 0 `/_ph`
requests. No `ph_*` cookie or storage key. |
| Then Accept (keyboard Enter) | Banner closes. `GET
/_ph/static/array.js` and `POST /_ph/e/` in 3 s. Key is `granted`. After
reload: no banner, events continue. |
| `Europe/London`, Decline, wait, reload, wait | No banner. 0 `/_ph`
requests. No `ph_*` cookie or storage key. Key is `denied`. |
| `Europe/Zurich` with Global Privacy Control | No banner. 0 `/_ph`
requests. No `ph_*` cookie or storage key. |
| `America/New_York` | No banner. posthog-js loads after the page
settles, as before. |
| Profile settings switch (US) | Shows on. Off: client is opted out at
once. On: opt-in, events continue. |

- I ran the browser check again after the review fixes, with the same
results.
- Screenshots (banner light, banner dark, mobile 375 px light and dark,
settings switch): see the PR comment.
- Cold load (local dev server, fresh browser context per run, median of
7, ms):

| Build | Time zone | FCP | DOMContentLoaded | load |
| --- | --- | --- | --- | --- |
| master | New York | 728 / 724 | 687 / 684 | 689 / 686 |
| this PR | New York | 748 / 724 | 708 / 684 | 710 / 686 |
| master | Berlin | 740 | 694 | 697 |
| this PR | Berlin | 756 / 724 | 716 / 684 | 718 / 686 |

The second round gives the same medians for master and this PR. The
first-round difference is noise. The consent code adds no request and no
load-time wait.

## Risks

- Region by time zone is approximate. It asks some visitors outside the
EU (for example `Europe/Istanbul`, or `UTC` browsers). This is the safe
direction.
- A forged country header (only when `PAPERCLIP_POSTHOG_COUNTRY_HEADER`
is set) changes only the default for that visitor's own browser.
- The choice is per browser. A user on a new browser sees the banner
again (in consent regions).
- Visitors in consent regions now send no browser analytics until they
accept. Expect fewer EU/UK sessions in PostHog.
- Self-hosted installs: no change. They get no `browserAnalytics`
config, so no banner, no switch and no load.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, headless browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:52:20 -06:00
DottaandClaude Opus 5.5 2a43fe3e34 feat(telemetry): add company_id to Cloud direct events from background work (#22183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud sends server telemetry straight to PostHog
(`server/src/services/posthog-direct/`)
> - Request events get `company_id` from the request context. Background
events have no request.
> - So `agent.task_run`, `routine.run`, `connection.invoked`, and other
background events arrive with `stack_id` but without `company_id`. On
staging, 66 of 66 `agent.task_run` events had a null `company_id`.
> - Without `company_id`, product analytics cannot split agent and
routine activity by company.
> - This pull request lets each background emit site name the company of
the entity that it reports on. Only the direct PostHog transport reads
that company.
> - The benefit is that Cloud background events carry `company_id`, and
the self-hosted collector output does not change.

## Linked Issues or Issue Description

Refs #22006

**Subsystem affected**
Server telemetry: Cloud direct delivery to PostHog
(`server/src/services/posthog-direct/`) and the server emit sites for
background events.

**Problem or motivation**
Cloud direct events from background work have no `company_id`. Heartbeat
runs, scheduled routines, connection invocations, and interaction
updates do not run inside an HTTP request, so the request context gives
no company. On paperclip.staging, all 66 sampled `agent.task_run` events
had `company_id` null.

**Proposed solution**
Add a small AsyncLocalStorage scope, `withTelemetryCompany(companyId,
fn)`, in `server/src/telemetry-company-scope.ts`. Each background emit
site wraps its typed `track*` call with the company id of the row that
it already has. The direct transport reads the scope at track time,
through the existing `captureEventContext()` seam. A live request
company still wins. The collector path never reads the scope.

**Alternatives considered**
1. An optional context argument on the shared `track*` helpers. This
changes the public helpers in `@paperclipai/shared` for a Cloud-only
value, so I did not use it.
2. A scope around each whole work unit (heartbeat run, routine run,
invocation). The emit calls are fire-and-forget and are spread across
many modules, so it is easy to miss one or to leak a company into other
work. A scope at the emit site is narrow and explicit.
3. Let the transport look up the company in the database. The transport
must not query the database.

**Roadmap alignment**
This completes the Cloud direct PostHog work from #22006. It does not
change the telemetry contract.

## What Changed

- Add `server/src/telemetry-company-scope.ts`. It has
`withTelemetryCompany(companyId, fn)` and `scopedTelemetryCompanyId()`.
With no company id, it calls `fn` directly.
- `capturePostHogEventContext()` uses the scoped company when no live
request gives a company. The request company, user, and session behave
as before.
- Wrap these emit sites with the company of their entity:
- `agent.task_run` (`emitAgentTaskRun`, `emitAgentTaskRunById`;
`companyId` is now part of the run that the function reads). This covers
heartbeat, native runtime, recovery, run-dispatch, and watchdog callers.
- `routine.run` (`routineService` dispatch, which covers scheduler,
webhook, and manual runs)
  - `agent.first_heartbeat` (heartbeat agent status)
  - `connection.created`, `connection.updated`, `connection.invoked`
  - `interaction.created`, `interaction.resolved`
- `agent.task_completed` (issue route; the issue company is the fallback
when the request gives no company)
- Request-only events (`project.created`, `goal.created`,
`agent.created`, `routine.created`, `skill.imported`) keep the request
context. `server.started` and `error_handler.crash` are not
company-scoped. Dynamic plugin and `question_response.delivery` events
are not in the contract, so the direct path drops them.
- Update the "Cloud Direct Delivery" section of
`packages/shared/src/telemetry/README.md`.

Design note on module boundaries: the scope lives beside
`server/src/telemetry.ts`, not inside `posthog-direct/`. Emit sites do
not import the transport, and the transport does not know which services
exist. The shared telemetry package does not change.

## Verification

- Red/green: with the source changes removed, these 4 new tests fail.
With the changes, they pass.
- `agent-task-run-telemetry.test.ts`: `emitAgentTaskRun` and
`emitAgentTaskRunById` track in the run company scope.
- `routine-run-telemetry.test.ts`: a routine run tracks in the routine
company scope.
- `posthog-direct-telemetry.test.ts`: on Cloud, background
`agent.task_run` and `routine.run` (also after an `await`) carry
`company_id`, and an unscoped event does not.
- `posthog-direct-telemetry.test.ts`: a live request company wins over
the scope, and the scope fills the gap when the request has no company.
- Red/green per emit family: the tests for `connection.created`,
`connection.updated`, `connection.invoked`, `interaction.created`,
`interaction.resolved`, `agent.first_heartbeat`, and
`agent.task_completed` assert the company scope at track time. With the
emit-site wrappers removed, all 7 assertions fail.
- New guard test: on self-hosted, a scoped `agent.task_run` and
`routine.run` produce a byte-identical collector envelope with no
`company` text.
- `cd server && npx vitest run` on 16 telemetry, PostHog, routine,
heartbeat status, interaction, and connector test files: 408 passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Local check with a real `posthog-node` client sent to a local capture
server (never real PostHog), embedded Postgres, and a real
`routineService.tickScheduledTriggers()` tick plus `emitAgentTaskRun`:
- Cloud mode: the captured `routine.run` (`source: schedule`) and
`agent.task_run` both had the seeded `company_id`, plus `stack_id` and
`environment`. `server.started` had no `company_id`.
- Self-hosted mode: the collector body from this branch and from
`origin/master` are identical after I normalize random ids and
timestamps.
- I did not boot a full Cloud-managed server, because it needs S3 and an
external Postgres. The check above uses the same server modules and the
real PostHog client.

## Risks

- Privacy review: this is a Telemetry change under
`rules/data-paths.mdc`, so I request a privacy review. Background Cloud
events now send the real Paperclip `company_id` to PostHog. #22006
already approved `company_id` as a Cloud direct property for request
events. This PR only fills it for background events. It adds no new
property, dimension, or event, so the generated contract does not
change. `packages/shared/src/telemetry/README.md` is updated in this PR.
- Low runtime risk. The self-hosted collector path does not read the
scope, and a test proves its output does not change.
- On Cloud, more events now have `company_id`. No event loses a field.
- If a request with a company tracks an event for an entity in a
different company, the request company wins. This is the same as before
this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:34:39 -06:00
DottaandClaude Opus 5.5 1c556d5340 fix(native): never rest a task blocked on the agent's own blocker (#22140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report. A `paperclip_block`
report names a blocker, its owner and the unblock action
> - The status arbiter binds a task-wide blocker. When the owner is not
a person, it binds the blocker to the same agent and wakes that agent
> - The dispatcher refuses those self-wakes
(`chat_control_recovery_proof_unresolved`), because only continuation
and retry effects are proof for a native wake. The task then rests
blocked with no owner, no wake and no Retry
> - #21959 continues a system blocker in a fresh session, but only for
the reason codes `environment_unavailable` and `tool_unavailable`. A
restart-chaos rerun showed the same dead end with the reason code
`tool_output_failure`
> - This pull request makes sure that a blocker that the agent reports
for itself always has a next owner: the automatic environment lane, one
bounded self-continuation, or a person with a Retry that works
> - The benefit is that no task rests blocked with nobody to act, and no
path wakes the agent in a loop

## Linked Issues or Issue Description

Refs #21959

**What happened?**

A native OpenCode run (DeepSeek) said that every tool call returned the
text "cryptic". It reported `paperclip_block` with owner `{kind:
system}`, scope `task_wide`, reason code `tool_output_failure`, and an
unblock action about the tool-output failure. The run succeeded with
disposition `blocked`. The arbiter bound the blocker to the same agent
(`task_wide_blocker_bound`). The `notify-owner` wake coalesced, and the
`blocker-owner` wake was skipped as
`chat_control_recovery_proof_unresolved`. The task rested blocked
(`needs_attention`, 0 blockers), with no Retry and no wake. An earlier
rerun showed the same shape with reason code `environment_unavailable`,
which #21959 fixed for those two codes only.

**Expected behavior**

A blocker that the agent reports for its own tools or environment is a
harness problem. Paperclip continues it automatically, with a bound. A
blocker that the agent says it owns gets one fresh session. A blocker
that needs a person goes to a person, with a clear prompt and a Retry
that starts the task again. Nothing is left with nobody.

**Steps to reproduce**

1. Run a native task. Make the agent call `paperclip_block` with owner
kind `system`, scope `task_wide` and a reason code other than
`environment_unavailable` or `tool_unavailable` (for example
`tool_output_failure`).
2. See the decision `task_wide_blocker_bound` with the agent as the
unblock owner.
3. See both owner wakes refused, and the task blocked with no Retry.

**Paperclip version or commit**

`master` at 9e1a85fb46.

## What Changed

- `native-environment-blocker.ts`: a system-owned blocker now uses the
bounded fresh-session environment lane for any reason code, except the
person-only codes `permission_denied`, `credential_missing`,
`policy_restricted` and `budget_exhausted`. An agent-owned blocker with
`environment_unavailable` or `tool_unavailable` also uses it. The lane
is unchanged: at most 2 retries on the shared failure counter, then a
board blocker.
- New `native-agent-blocker.ts` and a new arbiter branch for a task-wide
blocker that is not person-owned, from the current owner's own succeeded
run (not a chat, not a review):
- Owner kind `agent`: one fresh-session continuation
(`native_agent_blocker_continuation`). It uses the shared failure
counter with a maximum of 1, so it never runs after another automatic
retry in the same chain. A paused task does not get it.
- All other cases, and a repeated blocker: the blocker goes to the board
with the agent's unblock action (`native_agent_blocker_needs_person`).
There is no self-wake.
- `native-provider-capacity-retry.ts`, `status-decision-committer.ts`,
`native-run-finalizer.ts`: materialize the continuation through the
existing fresh-session retry lane (`forceFreshSession`, `retryOfRunId`,
30 s delay, refuses a stopped run). Both new reason codes need the
provider-failure owner recheck under the issue lock. The finalizer reads
the pause hold for every blocked result.
- `heartbeat.ts`, `run-dispatch/adapters/postgres.ts`: the continuation
gets the same dispatch ownership checks as the environment retry.
- `task-interruption-presentation.ts`, shared
`TaskInterruptionPresentation.userAction`: a task that is blocked by a
`native_agent_blocker_needs_person` decision on its latest run shows
`userAction: "unblock"` with `canRetry: true`. `ExecutionBlockerNotice`
shows "The agent stopped and needs your help to continue. Read its last
comment, do what it asks, then Retry." with a Retry button. The inbox
label is "The agent needs your help". The assigned-work invariant reads
it as `assignee_declared_block`.
- `blocked-question-answer.ts`: an answer to the agent's open question
still wakes it after the new hand-off.
- Runner `completion-result.ts`: the "Paperclip continues this blocker
automatically" message now matches the server's environment lane.
- I did not change `readChatControlNativeParent`. The fix removes the
self-bound blocker wakes instead of accepting them as proof, so a
self-wake loop cannot form.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-agent-blocker.integration.test.ts`
(new, embedded Postgres, real executor, finalizer and committer). Red
before the fix (3 of 3 fail with `task_wide_blocker_bound`), green
after:
- The t205 shape (system owner, `tool_output_failure`) gets one
fresh-session retry, no owner wakes, and the retry run that reports done
moves the task to `done`.
- An agent-owned blocker continues once. The same blocker in the
continuation goes to the board, with no further retry and no owner
wakes.
- A system blocker with `credential_missing` goes to the board with no
automatic retry.
- An external-owned blocker goes to the board. The task shows
`userAction: "unblock"` and `canRetry: true`. `requestTaskRetry` accepts
the Retry and `resumeTaskRetry` starts exactly one new run of the task.
- `npx vitest run src/services/native-runtime/status-arbiter.test.ts`:
new case for the once-only continuation, the hand-off, the pause hold,
the chat exclusion and the non-owner exclusion.
- `cd ui && npx vitest run
src/components/ExecutionBlockerNotice.test.tsx`: new case for the help
prompt and its Retry.
- `cd packages/paperclip-runner && npx vitest run
src/contracts/completion-result.test.ts`: new cases for the conflict
message.
- Related suites pass (11 files, 523 tests): arbiter corpus,
task-interruption presentation and query, assigned-work invariant,
blocked-question answer, liveness escalation, run-dispatch adapter,
failed-turn continuation, tool-receipt limit and provider capacity.
- `tsc --noEmit` passes for `server`, `packages/shared`,
`packages/paperclip-runner` and `ui` (only the existing generated
Storybook catalog errors). `scripts/check-token-gates.mjs` is clean.
- `pnpm verify` exits 2: the source SDK pin refresh is a global CI
input, so required CI supplies the broad coverage.

## Risks

- More automatic runs: an agent-owned blocker can add 1 run per chain,
and a system blocker with a new reason code can add up to 2 (the
existing lane). Both share the failure-retry counter, so a chain never
gets more than the existing maximum. The company-wide automatic-run
brake admits these runs like any other recovery run. Judge the change by
company-wide automatic run starts per hour and their success rate.
- A blocker that only a person can resolve, but that the agent labels as
its own, now takes one extra run (about 30 s later) before it reaches a
person. This is intended.
- Board-owned blockers now show an attention card and a Retry, where
before they named the agent and showed nothing. A Retry starts a new run
of the task; the person decides when.
- Previous release: no persisted shape changes. The new reason codes are
new values in existing text columns. Decisions saved by the previous
release keep their saved effects.
- Chats keep their current behavior, because the person in the chat
answers directly.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:43:28 -06:00
43b2d8be22 feat(dashboard): add an experimental areas and responsibilities dashboard (#22069)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The dashboard is the first page a person sees in a company
> - The classic dashboard shows agents, runs, costs and recent tasks
> - It does not show what the person is responsible for, or if that work
goes well
> - People think in areas, such as Fundraising or Customers, and in the
things they own in each area
> - This pull request adds a dashboard built on areas and
responsibilities, behind an experimental setting that is off by default
> - The benefit is that a person sees what they own, how it is going,
and what agents do for it, on one page

## Linked Issues or Issue Description

No public issue exists for this work. The description follows the
feature request template.

**Subsystem affected**

Dashboard, navigation, the task conversation, routines, and the instance
experimental settings.

**Problem or motivation**

The classic dashboard counts agents and tasks. It does not say what a
person owns or if it goes well. People who run a company with agents
want to see their areas, the responsibilities in each area, the bar each
one must meet, and the agents and routines that serve it.

**Proposed solution**

A person owns responsibilities. Each responsibility belongs to an area
and has a bar, a number to reach. Projects, tasks and routines link to a
responsibility, and the page shows its status from those links. An agent
can interview the person and draft the dashboard. The person accepts or
declines each change, and any change can be undone from a saved version.

**Alternatives considered**

Add the same data to the classic dashboard. That page is a fixed set of
cards for company activity, so a second, person-centred page is clearer.
The setting lets a person pick one of the two.

**Roadmap alignment**

This is an experimental feature behind `enableResponsibilities`. It is
off by default and changes nothing for a person who leaves it off.

## What Changed

- Add the setting "Areas & Responsibilities dashboard"
(`enableResponsibilities`, off by default). On, Dashboard in the sidebar
opens the new page, and the classic dashboard link is not shown in the
sidebars or the command palette. Off, nothing changes.
- Add tables and migrations 0492 to 0497 for areas, responsibilities,
links to work, metrics, status reports, saved dashboard versions, and
model-call records for summaries.
- Add the API: areas, responsibilities, links, bars, metrics, status
reports, summaries, runs, automation, and the dashboard conversation
with versions and undo. Each route checks company access and writes to
the activity log.
- Add four task conversations with an agent: about a bar, about an
automation, about a routine, and about the whole dashboard. The agent
proposes a change. Nothing applies until a person accepts it.
- Add standing checks: routines that an agent carries out on its
heartbeat and reports.
- Add the Dashboard page: search, counts, areas that open and close, a
primary New area button, and a message box at the bottom of the page.
Add the responsibility page with its bar, metrics, work and routines.
- Add the Paperclip skill sections and runner API entries that let an
agent use these routes. The runner skill inventory is regenerated (166
to 174 rows).
- Keep the new runner API tests in their own file so
`runner-api.test.ts` stays at 985 lines.
- Onboarding is not changed. The onboarding changes that build on this
work are a separate pull request.

## Verification

- Type checks pass for `packages/db`, `packages/shared`, `server` (all
layers) and `ui`.
- `pnpm --filter @paperclipai/db check:migrations` passes. A new
`drizzle-kit generate` run produces no further change.
- Tests for every changed file pass: 26 server test files (3,009 tests),
116 UI test files (1,991 tests) and 28 shared test files (432 tests).
- The runner capability inventory checks pass.
- I started the app on a local database that already held earlier data.
Master's new migrations and these migrations applied without error. With
the setting on, the Dashboard and a responsibility page loaded with no
console or server errors. The More menu and the command palette show one
Dashboard.
- I did not run `pnpm verify`, the full local suite, or the browser
suites. Required CI covers them.
- The agent conversations are tested with simulated agents. No real
model has run them.

## Risks

- Medium. The change is large, and the setting is off by default. With
it off, the new routes refuse requests and the UI is unchanged.
- Migrations only add tables and columns. Two replace a CHECK
constraint, and one builds an index `CONCURRENTLY`. These two have the
`online_compatible` override with a reason in the file, and the second
one is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.
- Master can add migrations before this merges. The migrations then need
new numbers.
- `responsibilities.ts` (1,663 lines), `responsibility-summaries.ts`
(1,239 lines) and one Storybook fixture file (1,305 lines) are new files
over 1,000 lines. They each keep one cohesive unit together: the service
for one table family, the model-call path for summaries, and the
fixtures that the stories share. No existing file crosses 1,000 lines.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude: Claude Fable 5.2 (`claude-fable-5-2`) wrote most of
the change, with Claude Sonnet 5.5 (`claude-sonnet-5-5`) for the final
steps. Both used tool use, code execution and browser checks.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-17 20:45:44 -08:00
DottaandClaude Opus 5.5 4afcbfc0c7 fix(native): retry transient provider refusals while reattaching a handed-off run (#22118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap hands live native runs from the outgoing server to
the incoming server. The incoming server reattaches each run to its
original sandbox.
> - Reattachment first resumes the exact sandbox lease through the
sandbox provider plugin.
> - In a release-swap wave, one reattach resume got an HTTP 502 from the
provider gateway. The host did not retry it. The run failed with the
generic `setup_failed` code after 13 minutes of agent work.
> - The task then stayed blocked with a Retry banner for about 10
minutes, until the fresh-continuation fallback reached its bound and
started a new run.
> - This pull request retries a transient provider refusal during
reattachment, with a bounded budget. It also reads the provider status
from the plugin RPC error envelope, so the host sees the 502.
> - The benefit is that a brief gateway failure during a swap is
invisible to the user. The same run continues and no Retry banner shows.

## Linked Issues or Issue Description

**What happened?**
During a graceful release swap, the incoming server reattached a live
native run to its retained sandbox. The `environmentResumeLease` plugin
call failed with an HTML "502 Bad Gateway" body 280 ms after the
reattach started. The other 18 handed-off runs in the same swap
reattached in about 8 seconds. The host treated the 502 as an unknown
error, so the run failed with `setup_failed`. The task stayed `blocked`
with `canRetry: true` until the fresh-continuation fallback created a
new run 10 minutes later.

**Expected behavior**
A transient provider refusal (HTTP 408/425/429/5xx, a connection errno,
or a plugin RPC deadline) during reattachment is retried within a
bounded budget. The same run continues hands-off. A definite refusal
(for example HTTP 409) or a confirmed missing sandbox still fails at
once with its own named reason.

**Steps to reproduce**
1. Start a remote native run on a reusable sandbox.
2. Send SIGTERM to the server so it hands the run to the next server.
3. Make the first `environmentResumeLease` call on the incoming server
fail with HTTP 502.
4. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**
`master` at ca34aca777 (release-swap wave) and ea154ce55e (this branch
base).

Root causes:
- Why setup failed: `environmentRunOrchestrator.acquireForRun` called
`resumeRunLease` once for a reattachment. Acquisition already retries
transient resume refusals, but reattachment did not. Also,
`transientSandboxResumeFailure` read `status` and `statusCode` only from
the thrown error. A plugin refusal arrives as `JsonRpcCallError`, which
carries the provider status only in the validated
`paperclip/environment-resume-error/v1` data envelope. So a plugin 502
was not transient even on the acquisition path.
- Why recovery took 10 minutes: `setup_failed` on a native run with
provider work is not a restart-class code, so the same-run reattach lane
does not take it. Failed-workspace preservation recorded a restore hold.
The fresh-continuation fallback waits for that hold up to
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes) when it sees no
progress. The continuation started at 23:23:40, which is 10 minutes and
2 seconds after the run finished at 23:13:38.
- Why a Retry banner showed: no fresh-continuation check row existed yet
and no system recovery was active, so the presentation fell through to
`failed` with `canRetry: true`. The banner did not stay after the task
was done. The issue rests at `done` with `canRetry: false`. The grader
reported 10482 s because it closed the banner at its final read hours
later. Its last sample that saw the banner was 272 s after the first.

## What Changed

- `environment-run-orchestrator.ts`: the exact-lease resume during
reattachment goes through `retryTransientResumeCall`. The first call
keeps the provider's configured start deadline. A first call that used
up that deadline (plugin RPC TIMEOUT) is not repeated. After a transient
refusal, a 90 s budget (`REUSABLE_SANDBOX_RESUME_BUDGET_MS`) bounds each
later wait and passes the remaining budget as `timeoutMs`. A confirmed
missing sandbox and a definite refusal are not retried.
- `packages/plugins/sdk/src/environment-resume-error.ts`: owns one retry
policy for provider failures: `isRetryableEnvironmentHttpStatus` (408,
425, 429, 500, 502-504 and gateway/edge 520-527, 529, 598, 599; never
501 or 505) and `isRetryableEnvironmentErrorCode(code, scope)`.
ABORT_ERR is never retried. ENOTFOUND is retried only in the
`known_host` scope (an exact reattach to a host this run already
reached, the DNS-outage case). Lease acquisition keeps its canonical
semantics in the `first_acquisition` scope.
- `environment-provider-unavailable-error.ts`: lease acquisition uses
the SDK policy instead of its own sets.
- `reusable-sandbox-resume-replacement.ts`: owns the resume retry helper
and `transientSandboxResumeFailure(error, scope)`, which reads only
structured evidence (status, envelope errno, plugin RPC deadline,
sandbox state transition) and a few transport phrases. The acquisition
resume loop uses the same classifier and backoff helpers, and
`resumeFailureTimedOut` (label or envelope ETIMEDOUT) for its idle-start
extension.
- `environment-runtime.ts`: `EnvironmentDriverResumeInput.timeoutMs`
caps the sandbox plugin driver's resume RPC. Other drivers keep their
own deadline.
- Cloudflare bridge client: a bridge request deadline throws
`CloudflareBridgeTimeoutError` with code `ETIMEDOUT`. Daytona:
`SandboxLivenessTimeoutError` carries code `ETIMEDOUT`. Both reach the
host as a structured errno in the resume envelope.
- `tests/restart-chaos/staging-mix.ts`: `bannerShownMs` counts a visible
Retry banner at most 2 minutes past the last sample that saw it. Before,
it counted the whole unsampled gap until the final read.

Previous-release story: this changes no persisted state, launch
configuration, or pack identity. A run saved by the previous release
reattaches the same way. Only the in-process retry around the resume
call is new.

## Verification

- `pnpm verify` (selected Vitest files), and focused runs of the four
changed server test files and the two restart-chaos test files.
- Red/green: `server/src/__tests__/environment-run-orchestrator.test.ts`
- "retries a provider gateway refusal during reattachment and keeps the
original allocation" fails on master (1 call, rejects with the 502) and
passes with the fix.
- "treats a provider HTTP %i refusal during reattachment as transient"
(500, 502, 520, 599).
- "gives the first reattach resume the provider deadline and bounds
every retry by the budget".
- "does not repeat a reattach resume that used up the provider's start
deadline", "does not retry a definite provider refusal during
reattachment", and "preserves a retained sandbox startup failure without
acquiring a replacement" (default budget, one call).
- `server/src/services/reusable-sandbox-resume-retry.test.ts`: a table
over every classifier branch, and the helper's deadline rules.
- `server/src/__tests__/environment-runtime.test.ts`: "caps the exact
resume call at the caller's remaining budget".
- `server/src/services/environment-provider-unavailable-error.test.ts`:
scope table and the extended status lists. Orchestrator reattach cases:
ENOTFOUND retried, ABORT_ERR and 501 not.
-
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`:
"applies the idle-start extension to a provider liveness deadline named
only by its errno" (red/green).
- Cloudflare `bridge-client.test.ts` and Daytona `plugin.test.ts`: the
deadline errors carry ETIMEDOUT in the resume envelope.
- Red/green integration:
`server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts`
"keeps reattaching through a gateway refusal during a graceful hand-off
and continues the same run". The run's controller is lost, restart
recovery claims it to reattach, and the first resume fails with the 502
envelope. On master the run fails before the runner step (1 resume
call). With the fix the resume is retried, the same run reaches the
runner step at controller generation 2, no replacement allocation is
acquired, and no second run is created.
- `tests/restart-chaos/staging-mix.test.ts`: banner tests for a late
hide sample and for a banner still open at report time.

## Risks

- The acquisition resume loop no longer retries a refusal whose only
evidence is "timeout" or "timed out" text. Cloudflare and Daytona now
name their own deadlines ETIMEDOUT, and the plugin RPC deadline is
structured, so those stay retried. A provider that reports a timeout
only as text fails at once.
- Lease acquisition now also retries gateway statuses 520-527, 529, 598
and 599.
- A reattach resume can now take the provider deadline plus up to 90 s
when it gets transient refusals. Each retry is the same exact-lease
lifecycle resume. It cannot acquire or replace a sandbox, so a retry
cannot create a second executor.
- The aggregate recovery brake is unchanged. This change adds no new run
starts and no work before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:11:04 -06:00
Devin FoleyandPaperclip ea154ce55e Add bulk keyed issue document reads for native helpers (#22105)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Native helpers read task documents through authorized API tools.
> - A scan of 146 cases needs one saved response and one download per
case.
> - The document service can read the same key across authorized tasks
with bounded queries.
> - This pull request adds a bounded bulk read with the existing task
access policy.
> - Helpers can transfer one batch and check current revisions before
fetching changed bodies.

## Linked Issues or Issue Description

**Subsystem affected**

Task document REST API, shared query contracts, and native API
discovery.

**Problem or motivation**

Per-task document reads make native sync scans cost hundreds of tool
calls. Task list timestamps do not prove that document content is
unchanged.

**Proposed solution**

Add `GET
/api/companies/{companyId}/issue-documents?key=notes&issueIds=...`. Read
up to 200 task UUIDs under current company and task permissions. Return
current bodies and revision metadata, or metadata alone with
`includeBody=false`.

**Alternatives considered**

Generic request batching would add tool execution and partial failure
rules. One bulk resource read removes the per-case relay cost with the
existing tools. Do not skip documents based only on task-row timestamps.
Keep managed skill files immutable; cleanup of copied skill test
fixtures belongs to those tests.

**Roadmap alignment**

This improves existing task documents and native API access. It adds no
new workflow or UI. The roadmap has no duplicate bulk document read
work. No duplicate open pull request was found.

## What Changed

- Add a bulk keyed document route and a company-scoped document reader.
Reuse the current batched task-read policy and completed-task read
receipts.
- Preserve request order and deduplicate IDs. Return `document: null`
for readable tasks without the key. Omit unreadable, hidden, foreign and
missing tasks without revealing which exist.
- Bound requests to 200 IDs and responses to 8 MiB of JSON bytes. Check
encoded body sizes in SQL before fetching bodies in the same read
snapshot. Build JSON within the byte budget. Oversized responses fail
with 413 and are never saved as partial artifacts.
- Admit at most two active reads per actor and eight per server
instance. Extra requests receive 429 with Retry-After. Keep the slot
until both the database work and response settle.
- Batch completed-handoff lookups and group child receipts by parent.
Extend the existing event batch writer for ordinary non-waking receipts
with atomic sequence allocation. Stored receipt shapes stay unchanged
for existing consumers.
- Expose revision-only reads without selecting document bodies. Document
native discovery, one save plus one download, and complete-inventory
checks.
- Add real HTTP, database and artifact tests for 146 open and completed
cases, 200 maximum-size bodies, metadata overhead, access failures,
query limits and JSON expansion. Cover admission limits, failure
cleanup, and concurrent event writers. Refresh the shared SDK source
pins.

## Verification

Final revision: `93dc8b029600211d79a050650f3dff82e5dad52f`.

- [CI run 44212220917, attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44212220917)
passed. This includes typecheck, build, general tests, Runner checks,
serialized server suites, all e2e shards, and the final verify gate.
-
[Build](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506305267),
[e2e](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309117),
and
[verify](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134510062263)
are green.
- [Linux CI job
134506309291](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309291)
passed all 16 bulk document integration tests. This includes the native
transfer of all 146 documents with the saved size and hash,
completed-case receipt batching, and size-only rejection of 200
maximum-size documents.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22105#issuecomment-6965449032)
is 5/5 on the final revision. All review threads are resolved. Security
checks pass. GitHub reports CLEAN and MERGEABLE.
- Local server typecheck passed on macOS with Node 25.6.1. The focused
run below passed 177 tests; the Linux-only transfer test was skipped
locally and then passed in CI.

```sh
pnpm --filter @paperclipai/server typecheck
cd server
pnpm exec vitest run \
  src/routes/issue-document-batch.test.ts \
  src/services/native-runtime/issue-document-batch.integration.test.ts \
  src/__tests__/documents-service.test.ts \
  src/__tests__/openapi-routes.test.ts \
  src/__tests__/heartbeat-run-event-atomic-append.test.ts \
  src/__tests__/chat-completion-delivery.test.ts \
  src/__tests__/native-child-task-read-routes.test.ts
```

- Shared typecheck and build passed. From `packages/shared/`, `pnpm exec
vitest run src/validators/issue.test.ts` passed 61 tests before the
rebase; the batch query contract did not change in the rebase.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
passed. Runtime and lockfile pins are unchanged.
- `pnpm verify --explain` returned exit 2 because the API docs and
shared contracts require broader CI. The successful CI run above
supplies that coverage.

Failure and recovery record:

- Both performance findings reproduced before the fixes. Scale tests now
verify a size-only query for 200 large documents and one handoff lookup
plus two receipt-write statements for 146 completed cases. Concurrent
writers retain contiguous unique sequences and decoded payloads.
- Initial local runs found missing shared exports and OpenAPI route
coverage. These are fixed. Fixtures now respect Linux-only installation,
thrown cross-company denials, and the ES2022 TypeScript target. Rebase
conflicts preserved the onboarding coverage from master.
- CI attempt 1 interrupted Build, e2e shard 7 and six general-test
shards with runner shutdown signals. Attempt 2 recovered the tests but
its [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134499680425)
also received a shutdown signal and exited 143. No compiler or test
assertion failure was found in those interrupted job logs. The reason
the runners stopped remains unknown. Attempt 3 recovered Build and its
dependent verify gate on the same revision. Earlier interruptions are
not counted as passes.

## Risks

- Additive API only. No migration or existing endpoint response change.
- Helpers must adopt the batch response to gain the tool-call reduction.
They must verify that every requested task is present before treating
the scan as complete. Unreadable tasks are intentionally omitted.
- Split large document batches when the route returns 413. Metadata
remains available without bodies. Concurrency limits are per server
instance and can refuse simultaneous reads with 429; callers retry after
an active read finishes.
- This does not change company-owned helper inventory expiry or cleanup
tests, installed skill permissions, or generic tool batching. It does
not deploy to a tenant.

## Model Used

OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning,
tool use, and code execution. Context window size was not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 17:42:18 -08:00
ce27dde855 Start onboarding in chat and defer execution until the user chooses work (#22077)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Onboarding connects an agent and gives the user a place to discuss
work.
> - The previous flow created an execution task and project before the
user chose a scope.
> - A standing chat can gather the goal and clear exit criteria first.
> - This pull request opens chat after onboarding and starts a live
welcome after the chat is visible.
> - The agent proposes work for acceptance, or creates a task when the
user explicitly requests one.
> - Subscription setup and shared question state must work for this
first conversation to succeed.

## Linked Issues or Issue Description

**Subsystem affected**
Company onboarding, agent chat, local subscription setup, task
numbering, and shared interaction views.

**Problem or motivation**
The first session creates work before the user has agreed to a goal.
Standing chats consume task numbers. An answered chat question can
remain pending in the inbox. Claude code entry requires an extra click,
and switching from Claude to Codex can leave an incompatible model
selected.

**Proposed solution**
Open an owner-bound chat for every new company. Start a live welcome
after the chat paints in a visible tab. Gather scope in conversation and
use the existing task acceptance flow. Keep chat records out of the task
number sequence. Publish question responses to shared caches and cancel
stale reads. Submit pasted Claude codes once and wait for verified
sign-in before connecting.

**Alternatives considered**
A fixed welcome would not use the onboarding context. A hidden execution
task would still create work and consume a task number. Keep the
existing task proposal and acceptance contract instead.

**Roadmap alignment**
This improves the shipped Agent Chat and Personal & Shared AI Accounts
flows in ROADMAP.md. It does not add a separate execution or approval
system.

Related prior work: Refs #19643 and #19512. Related route handoff work:
Refs #12557. This change replaces the automatic first task with chat; it
also repairs the subscription and shared question state paths used
during that journey.

## What Changed

- Finish onboarding in agent chat for default and team-template
companies. Create no execution task or project.
- Queue one live welcome after visible chat content paints. Pass company
goals and the initial idea as context. Preserve access, budget, and
hiring controls.
- Instruct the agent to gather a goal and exit criteria, offer an
interview, and propose tasks for acceptance. Allow direct creation on an
explicit user request.
- Leave new standing chats unnumbered. The first execution task receives
number 1. Preserve existing identifiers.
- Synchronize answered question cards across chat and inbox, including
concurrent requests and inactive cached views.
- Submit a pasted Claude authorization code once. Show connection
progress and hide the restart link while sign-in is still preparing.
Detect rejected OAuth codes promptly. Clear incompatible models when the
provider changes.
- Use `Work with [agent name]` as the default chat composer placeholder.
- Add regression coverage and update onboarding documentation and API
notes.

## Verification

- Completed a real Claude subscription journey on macOS with an isolated
local database. Authorized the account, pasted the real code, and
reached the review step without clicking Submit code or Connect. Get
started opened chat and the live welcome run succeeded. The company had
zero execution tasks, zero projects, and task counter 0.
- On rebased commit c664949858, 6 browser cases passed for
paste-to-connect-to-chat, task-free onboarding, existing-company
routing, the native connection probe gate, desktop/mobile planning, and
two-tab chat/inbox question synchronization.
- On the rebased head, 419 focused UI tests passed. The focused server
and adapter run passed 160 tests and hit one `socket hang up` in the
existing authenticated chat route test. The full 72-test conversation
suite passed on one repeat. The cause of that first socket failure
remains unconfirmed; the passing repeat does not establish its cause.
- UI typecheck and server declaration typecheck pass after the rebase.
The first server check found a missing local dependency from master; a
frozen-lockfile install repaired it. No dependency manifest or lockfile
change was needed.
- The first Greptile review found two welcome edge cases. Four new tests
reproduced both before the fix. On commit b380f14ef9, all 114 onboarding
and queue tests pass, including timeout retries and queued welcomes
replaced by a user message or chat reset. Server source and focused test
type checks pass on this revision.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22077#issuecomment-6961533641)
gives rebased commit 2dbad34445 a score of
5/5. The current-head check passed. Both welcome review threads are
resolved, with no new actionable findings.
- CI serialized shard 8/9 found a missing OpenAPI inventory entry for
the new route file. The exact coverage test reproduced this locally.
Commit 17a72ec8b7 adds explicit mounted paths and documents both
board-only operations. All 65 OpenAPI and onboarding route tests and
server source typecheck pass after the fix. The final CI run passes on
this revision.
- Rebased onto master 23ce68a8dc and regenerated the SDK source registry
from committed inputs. Worktree preflight, SDK source validation, and
all 65 API/onboarding tests pass on final head
2dbad34445.
- On final head 2dbad34445, all 419
affected UI tests pass. UI package typecheck and server declaration
typecheck also pass. Commands: `pnpm --filter @paperclipai/ui
typecheck`; `node server/scripts/typecheck-layers.mjs
--declarations-only`; focused Vitest suites for the wizard, subscription
login, agent chat, shared interaction cache, and OpenAPI/onboarding
routes.
- [Final-head CI
run](https://github.com/paperclipai/paperclip/actions/runs/44208652564)
completed successfully on commit
2dbad34445. All required typecheck, build,
test, Runner, browser, serialized-server, and canary gates pass,
including the [final verify
job](https://github.com/paperclipai/paperclip/actions/runs/44208652564/job/134487749343).
The fresh routing gate selected the trusted AWS fleet after the
protected owner policy in #22098 merged. No source rebase was needed:
the gate reads that policy from protected master.
- The repaired OpenAPI shard passed in the earlier same-head CI run.
General server shard 8/12 hit a 20-second step timeout in the existing
`issue-recovery-live-run-agent-census` suite. This PR does not change
that suite or its fixture. All four tests pass locally on the exact
current commit in 84 seconds. The cause of the CI timeout remains
unconfirmed. Every other test job passed. A single failed-shard retry
was accepted but remained queued on its saved GitHub-hosted selection.
It was cancelled before test execution after #22098 merged. The fresh
full-CI run passed on the unchanged head under the approved AWS policy,
including [server shard
8/12](https://github.com/paperclipai/paperclip/actions/runs/44208652564/job/134481388730).
This passing execution does not establish the cause of the earlier
timeout.
- Worktree preflight, committed SDK source input validation, and diff
whitespace checks pass after the rebase.
- `pnpm verify` reports exit 2 because the new subsystem documentation
requires broader CI coverage. The final successful CI run supplies that
broader coverage. This is not a full local suite pass.

## Risks

- The initial task is now an explicit user decision. Existing onboarding
tasks and numbered records remain readable. No schema migration is
required.
- A welcome still depends on a working provider connection and the
normal run queue. Failed starts can be retried without duplicate
successful welcomes.
- Claude paste acceptance was verified with a real subscription on
macOS. This change does not claim a new live Codex or Windows acceptance
test.

## Model Used

OpenAI Codex, based on GPT-6, with tool-assisted reasoning, code
execution, terminal checks, and browser verification. The client does
not expose the exact routed build or context limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:52:35 -08:00
DottaandClaude Opus 5.5 f295e496db fix(daytona): retry lease acquisition after a provider connection failure (#22087)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs on Daytona start with a lease acquisition in the Daytona
plugin
> - When the host cannot reach Daytona, the Daytona SDK raises a
connection error that has no HTTP status and no errno
> - The host only retries a lease failure that carries structured
evidence, so this failure became `setup_failed` and the task waited for
a person to press Retry
> - This pull request makes the plugin report the SDK connection error
as a closed transport code that the host already retries
> - The benefit is that a short network blip during sandbox startup no
longer needs a Retry click

## Linked Issues or Issue Description

**What happened?**

In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host
could not resolve or reach Daytona for about 12 minutes. One task needed
a Retry click. Its only run failed with `setup_failed`
(`lease_acquire_failed`) and the message `Failed to acquire lease for
environment "Daytona warm ..." (sandbox): read EADDRNOTAVAIL`. Automatic
recovery did not continue the task because `setup_failed` is not a
transient setup code. The task stayed failed for about 100 minutes until
the grader pressed Retry.

**Expected behavior**

A lease acquisition that fails because the provider could not be reached
is retried automatically with the bounded bootstrap budget, as
`environment_provider_unavailable`. A person never needs to press Retry
for a network blip.

**Steps to reproduce**

1. Use the Daytona provider.
2. Make a Daytona API call during lease setup fail at the transport
level (no HTTP response), for example with `read EADDRNOTAVAIL` or
`getaddrinfo ENOTFOUND`.
3. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**

master `ca34aca777` (wave) and `02ee538be9`.

Root cause: the pinned `@daytonaio/sdk` converts an Axios transport
error into `DaytonaConnectionError` (or
`DaytonaConnectionTimeoutError`). It keeps neither the errno nor the
cause. The worker RPC therefore sent no provider diagnostic, and
`readTransientLeaseAcquisitionFailure` returned `null`.

## What Changed

- The Daytona plugin wraps a `DaytonaConnectionError` from lease
acquisition in an error with code `CONNECTION_FAILED`, and a
`DaytonaConnectionTimeoutError` with code `ETIMEDOUT`. The message is
unchanged. It reads the SDK class name only, never message text.
- The plugin SDK closed diagnostic set accepts `CONNECTION_FAILED`, so
the code crosses the worker RPC.
- The host treats `CONNECTION_FAILED` as a transient connection failure.
The run becomes `environment_provider_unavailable` and uses the existing
bounded bootstrap retry (30 s, 60 s) and the automatic-run brake.
- Refreshed `tests/runner-e2e/source-sdk-public-inputs.json` for the
changed SDK file.

## Verification

- Red/green: with the source change removed, the new plugin tests
(`reports a DaytonaConnectionError during setup as a transient transport
failure`, timeout variant) and the new host tests (`treats errno
CONNECTION_FAILED as transient`, `retries a provider connection failure
reported across the worker RPC`) fail. With the change, they pass.
- `cd packages/plugins/sandbox-providers/daytona && npx vitest run` (637
passed).
- `npx vitest run
server/src/services/environment-provider-unavailable-error.test.ts
packages/plugins/sdk/tests/environment-acquire-error.test.ts` (66
passed).
- `pnpm typecheck` in the Daytona plugin passes.
- `pnpm verify` asks for broad CI coverage because the SDK input
registry changed. CI supplies that coverage.

## Risks

- Low risk. A wrong Daytona API URL also produces
`DaytonaConnectionError`. It now gets two bounded retries (about 90 s)
before the same Retry state as before.
- The retries count toward the automatic-run brake, so a long outage
cannot multiply run starts across the company.
- A create that timed out on the client keeps its existing
creation-cleanup envelope. The retry is a new creation attempt with its
own identity.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:55:47 -06:00
DottaandClaude Opus 5.5 23ce68a8dc feat(connections): end Google access when a Paperclip-managed Google connection is removed (#22065)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Connectors let agents use Google Workspace through a Google sign-in
that Paperclip Cloud manages for the user
> - When a user removes a Paperclip-managed Google connection, Paperclip
deletes the tokens locally, but the grant stays live at Google
> - Google's OAuth review expects that a disconnect ends the app's
access, and users expect it too
> - Google's revoke ends the user's whole grant to the shared client, so
only the Paperclip Cloud broker can decide when it is safe to revoke
> - This pull request asks the broker to revoke each removed grant,
keeps local removal unconditional, and tells the user the outcome in
plain language
> - The benefit is that removal ends Google access when no other
Paperclip connection needs it, and the user knows what to do when it did
not

## Linked Issues or Issue Description

**Subsystem affected**
Connectors: Paperclip-managed Google Workspace and the legacy
per-service Google connections (server removal, shared contracts,
connection UI).

**Problem or motivation**
Removing a Paperclip-managed Google connection or grant was local only.
The refresh token was deleted in Paperclip, but Google still listed
Paperclip as having access to the account. The docs said so ("Removal is
local for managed grants").

**Proposed solution**
On removal, send each removed grant's refresh token to the broker's
signed `POST /v1/connector/revoke`. The broker revokes at Google only
when no other live binding could share the grant. Paperclip records the
broker's outcome code and shows one of three plain messages. Local
removal never waits on, or fails because of, that call.

**Alternatives considered**
Call Google's revoke endpoint from the app. This ends the user's whole
grant to the shared client and can break their other Google connections,
so the broker owns the decision.

**Additional context**
Older brokers answer 409 `REVOCATION_NOT_AVAILABLE`. The app treats
that, a timeout, and a network error as "could not end access at Google"
and still removes the connection.

## What Changed

- `paperclip-cloud-connector.ts`: `revoke()` takes an optional
`timeoutMs`, returns the broker's `outcome` from an allowlist, and keeps
`REVOCATION_NOT_AVAILABLE` as an allowed rejection reason.
- New `tool-access/connections/google-access-removal.ts`. For a managed
Google connection, `prepare()` reads each removed grant's refresh token
and returns a `send()` step. `send()` calls the broker for each distinct
token, one after another, under one 5-second deadline. Every result and
every error (including a Paperclip Cloud client that cannot be created)
becomes an outcome code. It never rejects and never logs a token. An
unexpected error is logged with the error name only and reported as
`unexpected_error`.
- Order for removing a whole connection: read the tokens, run the whole
local teardown (grants, installs, app profile, gateway tokens, runtime
slots, catalog, OAuth state, archive the connection), call the broker,
delete the sent refresh-token secrets whatever the outcome, then delete
the remaining secrets. A failure anywhere in the teardown makes no
broker call. Order for removing one grant: read the token, commit the
revocation, call the broker, delete the grant's refresh-token secret.
`send()` owns the call and the deletion through an injected
secret-removal function.
- Owed revocations: a revoked managed Google grant whose refresh-token
secret still exists (and that nothing live uses) is sent by every later
removal, then the secret is deleted. This covers a process stop after
the teardown and before the call, with no new column. Once the secret is
deleted, nothing is sent again.
- Shared tokens: the shared (organization) sign-in writes the same
secret refs to the organization grant and the connection. So revoking an
organization grant alone neither calls the broker nor deletes the
secret, because the connection still uses that token; removing the
connection does both. The same rule applies to any refresh token still
used by another live grant or connection.
- Audit history: once the broker call settles, a
`connection.google_access_removal` audit event records `reasonCode` =
result and `providerAccess` (plus `grantId` for a single grant). If that
write fails, the removal still succeeds and the failure is logged.
- Revoke signs with the subject that consented to the grant
(`providerTenant.externalId`, saved by the managed callback from the
claim). Grants saved without it fall back to the refresh rule. Refresh
is unchanged; see Risks.
- `removal.ts` and `grants.ts`: `providerAccess: { provider, result,
outcomes }` goes to the responses and to the `tool_connection.archived`
and `tool_connection.grant_revoked` activity entries. The tool access
audit field `providerRevocation` keeps its Vercel meaning only.
- `packages/shared`: `ManagedGoogleAccessRemoval`,
`ManagedGoogleRevocationOutcome`, `ConnectionGrantRevocationResult`.
- UI: one helper, `connectionRemovalToast`, builds the removal toast
(base sentence plus the Google notice, tone and action) for Connectors,
Connections and the app page. No codes are shown.
- API: `docs/api/tool-connections.md` documents both DELETE responses
and `providerAccess`. The OpenAPI registration describes the optional
field and its four results.
- Docs: `GOOGLE-WORKSPACE.md` (order, owed revocations, decision table),
`GMAIL.md`, the Google connector memory note, and the help page on
deleting connection data.

Decision table, judged over the distinct tokens sent in this removal:

| Outcomes | `result` | User message |
| --- | --- | --- |
| Every token `revoked` | `ended` | "Paperclip's access to this Google
account was also ended at Google." |
| Any failure-type code: `provider_revocation_failed`,
`binding_not_known` (the broker had no record, so Google was not
called), `provider_revocation_disabled`, `revocation_in_progress`,
`revocation_not_available`, `broker_unavailable`, `broker_error`,
`credential_unavailable`, `unexpected_error` | `not_ended` | "Paperclip
removed the connection but couldn't end its access at Google. You can
remove Paperclip from your Google account's third-party connections." +
"Open Google account connections" |
| Only `other_connections_remain` | `other_connections_remain` | "Other
Paperclip connections still use this Google account, so Google keeps
Paperclip's access until you remove those too." |
| At least one `revoked`, the rest `other_connections_remain` |
`ended_unconfirmed` | "Paperclip's access was ended at Google. If more
than one Google account was connected here, check your Google account's
third-party connections to be sure." + "Open Google account connections"
|

Grants record no Google account identity: the managed callback saves
`externalId` (the Paperclip subject), `name` (the provider name), and
OAuth timing and scopes, and none of the 14 scopes returns an account
identifier. So the last row is always `ended_unconfirmed`. This includes
two tokens of one user and several agent grants on one Google account.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/google-managed-connection-removal.test.ts
src/services/tool-access
src/services/paperclip-cloud-connector-revoke.test.ts
src/services/paperclip-cloud-connector.test.ts
src/__tests__/tool-connection-removal.test.ts
src/__tests__/tool-access-service.test.ts
src/__tests__/openapi-routes.test.ts` (714 passed)
- `pnpm --filter @paperclipai/ui exec vitest run
src/features/connections/googleAccessRemoval.test.ts src/pages/apps`
(passed)
- Server test typecheck (`node server/scripts/typecheck-layers.mjs`)
passes.
- Red then green: the server tests failed on `master` sources (21), then
each review round's new tests failed on the previous head and pass now.
They cover: the order on both paths, local failure before the grant
commit making no call and the retry sending, cleanup failure after the
call making no second call, a process stop between commit and call being
sent on retry with the secret deleted after, a Paperclip Cloud client
that cannot be created, the consenting subject for an organization
grant, sequential sends under one deadline, each row of the decision
table (per-agent and same-user cases give `ended_unconfirmed`, a mixed
failure gives `not_ended`), the unexpected-error log without a token, no
token in any log call, the delete-stored-data option, and a toast test
per surface.
- Added time, measured locally with a stubbed broker that answers at
once: median 22.1 ms against 20.3 ms without the broker step (12 runs
each), about 2 ms. With a real broker, one round trip per removed token,
capped at 5 seconds in total.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Not verified against a live broker or a real Google account.

## Risks

- A removal of a managed Google connection or grant can take up to 5
seconds longer when the broker is slow.
- Refresh subject is not changed in this PR. In the broker
(`recordProviderGrant`), a ledger row's subject is fixed when the row is
first written, and a refresh under another subject leaves that row
untouched. So moving refresh to a new subject could stop updating the
row's last-seen time. After 200 days the row would then count as not
live, and a later removal elsewhere could revoke at Google while this
connection still works. That risk cannot be ruled out from code for rows
written before this release, so both refresh paths keep their old rules.
The two refresh paths already use different subject orders today: the
gateway prefers `connectorSubjectAgentId` and then
`connectorSubjectUserId`, while background refresh prefers the grant's
own subject. That is pre-existing, and this PR leaves it unchanged.
Revoke uses the consenting subject, which matches the claim-time ledger
row. When the row was first written by a refresh under a different
subject, the broker answers `binding_not_known` and the user gets the
warning.
- Repeated removals can send an already-removed token again; the broker
returns its recorded outcome and changes nothing at Google.
- Against an older broker, users see the "could not end access at
Google" message, which is accurate.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:52:20 -06:00
github-actions[bot]andlockfile-bot b43a08201c chore(lockfile): refresh pnpm lockfiles (#22020)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-17 12:56:23 -06:00
DottaandClaude Opus 5.5 f4cb69205e Let plugin workers serve connector tools (plugin transport) (#22019)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types to the Connectors catalog
(#22004), but only for transports the host already runs (remote MCP,
local stdio)
> - A plugin cannot yet serve a connector's tools itself, and legacy
plugin tools (top-level manifest `tools`) still reach every agent
without a connection or install
> - A plugin should be able to ship a full connector: settings,
credentials, tools, grants, policies and audit, like a built-in one
> - This pull request adds the `plugin` transport, the SDK API to serve
it, and moves legacy plugin tools under the same activation rules
without changing today's access
> - The benefit is that any connector can live in a plugin, and agents
get plugin tools only when an operator enables them

## Linked Issues or Issue Description

Refs #22004

**Subsystem affected**
Plugins (SDK, worker protocol, manifest), Connectors (tool access,
catalog discovery), tool gateway, database.

**Problem or motivation**
A plugin connector can only describe an existing transport. There is no
way for a plugin worker to serve a connector's tools with per-connection
settings and credentials. Legacy plugin tools are visible to every agent
as soon as the plugin is enabled, which is the opposite of the connector
activation rules.

**Proposed solution**
Add a `plugin` transport. The gateway keeps every decision (grant,
policy, approval, rate limit, audit) and calls the plugin worker with
the connection, its settings and a per-call credential. Legacy top-level
tools become one implicit connector per plugin. Existing installations
keep exactly their current access through a one-time migration.

**Alternatives considered**
Move legacy tools to connection tool names (`app.<connection>:<tool>`).
Rejected: names change, so existing tool policies and agent instructions
that name `plugin:tool` would stop matching (a possible widening or
loss). Legacy tools keep their names and handlers; only their visibility
is now gated by an install of the implicit connector.

## What Changed

- `packages/db`: migration `0491` widens
`tool_connections_transport_check` with `plugin` (`NOT VALID`, annotated
`online_compatible`), and adds `plugin_connector_legacy_access`, seeded
with one row per installed legacy-tools plugin and company.
- `packages/shared`: method schema gains `settingsSchema` and `tools`
(`{ kind: "static", list: [{ name, displayName?, description?, risk,
parametersSchema }] } | { kind: "dynamic" }`), only for `transport:
"plugin"`; tool names must match `[a-z0-9_]`; plugin-served methods use
`none` or `api_key`. The connect API accepts `settings`.
- `packages/plugins/sdk`: `ToolRunContext.connection` (`id`, `appSlug`,
`methodKey`, `grantId`, `settings`, `credential?`);
`ctx.connectors.register(appSlug, toolName, fn)` and
`ctx.connectors.listTools(appSlug, fn)`; new `listConnectorTools` RPC;
`executeTool` routes connector calls by `appSlug`. A worker refuses to
start when a declared static tool (or dynamic lister) has no handler.
The test harness gains `executeConnectorTool` and `listConnectorTools`.
- `server/src/services/plugin-connector-runtime.ts`: routes one call to
the plugin's worker; a stopped worker is a
`PluginWorkerUnavailableError`.
- Connect flow (`tool-access.ts`): a `plugin` method creates a
`paperclip_plugin` application and a `plugin` connection with `config =
{ sourceTemplateKey, connectionMethodKey, settings }`, validated against
`settingsSchema`. Credentials, grants, credential policy, installs and
profiles use the existing code.
- Catalog: static tools come from the definition, dynamic tools from
`listConnectorTools`, with the existing version hashing and retirement.
Tool risk comes from the declaration.
- Gateway: plugin connections join connected-tool discovery and both
execution paths (direct and approved actions). For each call the gateway
resolves the grant and, for `api_key`, the credential for that call
only. The run context comes only from the session. A stopped worker
returns `503 plugin_connector_unavailable` with `retryable: true`. The
token broker refuses to export plugin connector credentials.
- Legacy tools: a plugin with top-level `tools` and no `apps` gets an
implicit connector (slug `default`, named after the plugin, no
authentication). Its tools keep their names, handlers and today's access
decision, and an agent sees them only when a connection of that
connector is installed for it or for the company. The connector's saved
action rules (Off, Ask first) can only restrict them.
`completeLegacyPluginToolAccess` turns each migration row into such a
connection with a company install, at boot and when a disabled plugin is
enabled again; it is single-flight across servers, resumes a draft left
by an interrupted attempt, and never overrides an operator's later
changes.
- Catalog: tools a dynamic connector stops offering are retired from
discovery; `listConnectorTools` carries the connection's `companyId`,
and uses the acting person's grant for personal connections.
- Board: the setup flow renders `settingsSchema` with the shared JSON
Schema form, sends `settings`, and restores saved settings on resume and
reconnect.
- `create-paperclip-plugin --template connector` now scaffolds a plugin
connector (`apps` + `ctx.connectors.register` + a harness test). SDK
README and `PLUGIN_SPEC.md` describe connectors; top-level `tools` stay
documented as legacy.
- `inferToolRisk` moves to `tool-risk-inference.ts` so the catalog can
reuse it.

Design notes where the spec left room:
- OAuth for `plugin`-transport methods is not supported in this PR
(`none` and `api_key` only); OAuth plugin connectors can use
`mcp_remote` today. Noted as follow-up.
- Board "Test" calls on a plugin connection pass an empty
`agentId`/`runId`, because there is no agent run.
- Companies created after the upgrade get no automatic legacy
connection; they follow the normal activation rules.

## Verification

- `server`: `npx vitest run
src/__tests__/plugin-connectors-plugin-transport.test.ts` (new):
- rules 1-3: a ready plugin and configured accounts give no tools; an
install for agent A gives only A the tools; agent B is denied on
invocation;
- the worker receives `runContext.connection` with the right id, method,
grant, settings and the call's credential; spoofed arguments do not
change it; each account calls with its own key; results do not contain
the key;
- invalid settings are rejected (422); a stopped worker is `503
plugin_connector_unavailable` (retryable); a disabled plugin hides the
tools;
- dynamic tools are listed through the worker and a tool it stops
offering is retired;
- rule 5: a new installation of a legacy-tools plugin gives no tools; an
installation recorded by the migration keeps the same tool names for
every agent, through one company-wide connection (two concurrent
carry-overs make one; an interrupted draft is finished; a retry keeps an
operator's restriction), and is not also exposed as connection tools;
Ask first and Off on the implicit connector restrict the legacy tools;
  - the token broker refuses to export a plugin connection's credential.
- Existing gateway plugin-tool tests (`tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`) now set up the
legacy grant an existing installation has, and pass unchanged otherwise.
- Regression: `plugin-connectors-circleback-parity`, `tool-gateway`,
`tool-gateway-service`, `prompt-injection-gateway`,
`tool-access-service` (846 passed), plus `connector-tool-policy`,
`plugin-lifecycle-restart`, `remote-mcp-connectors`,
`ai-connection-router` in an earlier run.
- UI: `AppsConnect.test.tsx` covers sending plugin settings from the
settings schema.
- SDK: `tests/worker-rpc-host.test.ts` (start refusal without handlers,
connector routing, tool listing). Scaffold: `entrypoints.test.ts`.
- `packages/db`: `pnpm typecheck` including `check:migrations`; server
layered typecheck (sources and tests); `tsc --noEmit` for shared, SDK,
scaffold and UI.

## Risks

- Behavior change for new installations: legacy plugin tools no longer
reach agents until an operator connects the plugin's connector.
Installations that exist at upgrade time keep their tools through the
migration rows.
- The migration widens a check constraint (`NOT VALID`) and copies
plugin × company rows into a new table. Both are fast; the previous
release never writes `plugin`.
- Calls depend on the plugin worker; when it is down the agent gets a
retryable error.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 18:09:42 +00:00
DottaandClaude Opus 5.5 904c01d1a3 feat(mcp): graduate Assistant connections (MCP) out of experimental (#22015)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Assistant connections (MCP) let an outside assistant, or another
Paperclip instance, sign in as a person and use the `/mcp/paperclip`
endpoint
> - The feature was behind the experimental `enablePublicMcp` setting.
The setting was off by default and in practice off on Cloud stacks
> - Instance-to-instance work (one person's agent files a task on a
colleague's instance) fails when the receiving instance has the setting
off
> - This pull request moves the feature out of Experimental. It is now
always on, and a stored or Cloud-managed "off" value is ignored
> - The benefit is that every authenticated instance with a public URL,
including Cloud workspaces, accepts assistant connections with no setup
step

## Linked Issues or Issue Description

**Subsystem affected**
Public MCP (`server/src/services/public-mcp/`), instance settings,
Connectors → Assistant Connection (MCP) setup page, Experimental
settings page.

**Problem or motivation**
A remote MCP connector on one instance cannot use another instance's
`/mcp/paperclip` unless an admin on the other instance finds and turns
on an experimental toggle. On Cloud, the toggle is off and is managed by
the fleet. Instance-to-instance delegation therefore does not work by
default.

**Proposed solution**
Remove the gate. Keep `enablePublicMcp` only as a compatibility key, the
same way `enableApps` was graduated: old stored rows, PATCH bodies and
Cloud managed configs still parse, but the effective value is always
`true`.

**Alternatives considered**
An instance-admin kill switch that defaults ON. I did not keep one. Each
person already controls access: a connection exists only after that
person approves OAuth consent for one company, and the person or an
admin can revoke it. Deployment policy (authenticated mode and a valid
public URL) still decides whether the endpoint exists. A second switch
would add a state that can silently break instance-to-instance work
again. A schema migration was also not needed: the value lives in the
`experimental` JSON column, and the stored key becomes inert.

**Roadmap alignment**
Requested by the maintainer to enable instance-to-instance
communication.

## What Changed

- `packages/shared`: `enablePublicMcp` is a deprecated compatibility
key. The schema default is `true`. The feature catalog entry is
"Assistant connections (compatibility)" with both defaults `true`.
- `server` instance settings: normalization always returns
`enablePublicMcp: true`. The Cloud managed overlay skips the key, so a
fleet value of `false` cannot disable it. Removed the key from the Cloud
staging experimental-key list. Removed the client-mirror policy step
that ran only when the toggle changed.
- `server` public MCP: personal discovery, OAuth, tools, setup and event
delivery no longer read instance settings (`isEnabled()` returns `true`
for the personal endpoint without a DB read). Dot, Grok and Muse agent
connections keep their own experimental settings.
- `server` Cloud paths: the client-mirror sync, the hosted event
authority, the ingress-handoff expiry worker and the idle sleep proof
treat the front door as enabled. The warm-standby `503
workspace_unclaimed` behavior is unchanged.
- `server` Dot/Grok gates: they no longer also require
`enablePublicMcp`. Error messages no longer tell people to enable
Assistant connections.
- `ui`: removed the Experimental toggle, the "Enable assistant
connections / Open Experimental settings" block on the Assistant
Connection page, and the "Enable Assistant connections (MCP)" hint in
Dot setup. Updated the Dot and Grok copy.
- Storybook: deleted the Experimental-setting story and removed the
"enable the experiment" step from the journey and presenter.
- Docs: `doc/public-mcp.md` ("Enable an instance", "Start from
Connections", `GET /api/mcp/setup`), `doc/openai-dot-runner.md`,
`doc/idle-grok-dispatch.md`, and the agent-access map.
- Tests: new red/green tests for "enabled with no setting stored" and "a
stored or managed off value is ignored". Removed the tests for the
"disabled" behavior, which no longer exists.

## Verification

- Red/green: with the old source and the new tests,
`instance-settings-readonly.test.ts`,
`instance-settings-service.test.ts` and `public-mcp.test.ts` fail (4
failures, and the public-MCP suite fails in `beforeAll` because
`isEnabled()` was `false` with no setting stored). With the new source
they pass.
- `server`: `public-mcp.test.ts`, instance-settings (readonly, service,
managed overlay), `managed-config`, `cloud-event-authority`,
`events-work`, `dot-runner`, `dot-oauth-onboarding` and
`agent-runtime-assignments`. 354 passed.
- Cloud client-mirror lifecycle
(`client-mirror-lifecycle.postgres.test.ts`, skipped in CI) passed
against a local Postgres 17 with
`PAPERCLIP_MCP_SLEEP_TEST_DATABASE_URL`. 6/6, plus `events-work` 28/28.
- Idle and sleep: `idle-incoming-wake` (service and route),
`idle-deadline-proof`, `idle-incoming-wake-capability`,
`idle-sleep-proof`, `client-mirror-ack`, `recovery-deadline-acceptance`
and `idle-completed-history` pass. The warm-standby test now also
asserts that discovery after the claim does no SQL read.
- `ui`: `InstanceExperimentalSettings`, `AssistantConnection`,
`NewAgentDialog`, `NewAgent` and `ExternalAgentInviteDialog.muse` pass.
267 passed.
- Typecheck: `packages/shared`, `server` and `ui` pass. `node
scripts/check-maps.mjs` passes.
- `pnpm verify`: see the CI checks on this PR.
- After the deploy: confirm
`/.well-known/oauth-protected-resource/mcp/paperclip` and `/mcp/setup`
return 200 on paperclip.staging without changing any setting.

## Risks

- Behavior change: instances that had the setting off now serve
discovery, OAuth and `/mcp/paperclip`. Security invariants are
unchanged: the endpoint is still mounted only in authenticated mode with
a valid public URL, OAuth still needs a real signed-in user's consent,
local implicit board authority cannot approve, and revocation works the
same way.
- Cloud sleep: on a managed stack, the MCP sleep proof blocks sleep
until the client mirror publishes the tenant front door to Cloud. The
client-mirror sync runs at boot and does this once. Stacks that already
had the flag on use this path today. A stack whose Cloud connector
cannot publish the mirror would stay awake. Watch fleet sleep after the
deploy.
- Personal public-MCP requests now read instance settings one time less.
- Fleet admin: the Cloud fleet catalog can still show and set
`enablePublicMcp`. The value is ignored. Removing it from the fleet
catalog is a separate change in the Cloud repo.
- No migration. The `0354` trigger still compares the stored key. That
causes at most one extra connector-runtime invalidation if an old client
writes the key.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:42:36 -06:00
DottaandClaude Opus 5.5 b1efe32c60 feat(connections): name and guide connections to another Paperclip instance (#22018)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each Paperclip instance serves an assistant connection (MCP) at
`/mcp/paperclip`, and any instance can connect a remote MCP server by
its address
> - So one instance's agents can file and follow tasks on another
instance, for example a colleague's
> - But the other instance's tools look like the agent's own Paperclip
tools: same product, similar names, and the connection prefix was a
generic internal link key
> - An agent can then file local work on someone else's instance, or the
reverse
> - This pull request recognizes a Paperclip address, names the
connection and tool prefix after that instance, adds agent instructions,
and refuses a connection to the same instance
> - The benefit is that agents can tell their own Paperclip from a
connected one

## Linked Issues or Issue Description

**Subsystem affected**

Connections: "Connect your own MCP server" (generic remote MCP) and the
tool gateway names that agents see.

**Problem or motivation**

When instance A connects instance B through `/mcp/paperclip`, A's agents
get B's `paperclip_create_task` and other tools. The tool prefix came
from an internal key (`app-gallery🔗<uuid>`), so native runners saw
`app_mcp_app_gallery_link_paperclip_create_task_…`. Nothing told the
agent that the tools act on a different instance. An agent could send
internal work to another organization by mistake.

**Proposed solution**

Recognize an address whose path is `/mcp/paperclip`. Use only the typed
address, never data from the remote server. Name the connection after
the host (`acme Paperclip`). Use that name as the tool prefix for these
connections only. Add default agent instructions. Refuse the instance's
own address with `paperclip_self_connection` (422).

**Alternatives considered**

Read the company name from the remote `paperclip_connection` tool. This
needs an extra authenticated call during setup, and it puts
remote-controlled text in names and instructions. The host is enough to
identify the stack. Require approval for each write. We rejected this:
actions start on, the same as other connections.

**Roadmap alignment**

Supports instance-to-instance work through existing MCP connections. It
adds no new transport.

## What Changed

- `packages/shared/src/paperclip-peer.ts`: detect a Paperclip MCP
address, derive a label and name, compare with own origins (loopback
hosts on the same port are equal), and build default agent instructions.
- `server/src/services/tool-access.ts`: default name and default agent
instructions for a new Paperclip connection. Instructions that the
person supplies are kept.
- `server/src/routes/tool-access.ts`: refuse the instance's own address
before any network request.
- `server/src/services/tool-gateway.ts`: use the prefix recorded on a
Paperclip connection (`paperclipPeerToolNamespace`, derived from its
address at creation). Renames never change it. Other connections, and
Paperclip connections made before this change, keep their current tool
names.
- `ui/src/pages/apps/generic-mcp-connect.ts`: same default name in the
setup flow.
- `doc/connections/GENERIC-REMOTE-MCP.md`: new section "Connecting
another Paperclip instance".

## Verification

- `packages/shared`: `npx vitest run src/paperclip-peer.test.ts`, 20
passed.
- `server`: `npx vitest run
src/__tests__/generic-mcp-connection.test.ts`, 98 passed. The naming,
instructions, self-connection and tool-prefix tests fail on the base
revision and pass with this change (red/green).
- `ui`: `npx vitest run src/pages/apps/generic-mcp-connect.test.ts`, 29
passed.
- `tsc --noEmit` passes for `packages/shared` and `server`.
- Live check with two local instances. Instance B was in authenticated
mode on port 3201 with assistant connections on. Instance A was on port
3202.
- A connected to `http://localhost:3201/mcp/paperclip`. The name was
`localhost-3201 Paperclip`. A registered with B by dynamic client
registration and asked for read, write, configure and offline access.
- B's consent page showed "Connect Paperclip localhost-3202 to
Paperclip". After approval, A had all 37 tools active, writes included.
- From A, `paperclip_connection` returned B's company.
`paperclip_create_task` created `ACM-1` on B.
- An agent session on A listed
`mcp.localhost-3201-paperclip-…:paperclip-create-task` (runner name
`app_mcp_localhost_3201_p_paperclip_create_task_…`). Before the gateway
change, the name was `app_mcp_app_gallery_link_paperclip_connection_…`.
The run received the connection instructions.
- Connecting A to `http://localhost:3202/mcp/paperclip` returned 422
`paperclip_self_connection`.

## Risks

- Low risk. Only connections whose address path is exactly
`/mcp/paperclip` change. Other connections keep their names,
instructions and tool names.
- A Paperclip connection created before this change keeps its original
tool names, because only connections that record
`paperclipPeerToolNamespace` at creation get the new prefix. Saved exact
tool-name rules and pending approvals for those connections keep
matching. They also get no default instructions; people can add them.
- Native runners keep only 20 characters of the namespace (`mcp_` uses
4), so long host labels are cut. The tool part stays complete.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser pane).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:41:13 -06:00
DottaandPaperclip 7f9867895c fix(ui): keep automated work out of personal recents (#22024)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The sidebar helps each person return to their own work.
> - A task visit previously added any readable task to personal recents.
> - Automated updates also moved old tasks above newer personal work.
> - The Mine inbox already defines personal participation.
> - This pull request applies that rule to recent task candidates and
preserves their order during background updates.
> - Users can see current task status without a stream of unrelated
automated work.

## Linked Issues or Issue Description

**What happened?**

Routine tasks and imported GitHub review tasks appeared in personal
recents after a read. Later agent updates moved these tasks up the list.
A connected account or stored creator ID could make these tasks look
like personal work.

**Expected behavior**

A task must pass the viewer's existing Mine participation rule before it
appears in personal recents. Routine ownership, connected-account
provenance and reads alone do not qualify. A viewer can qualify through
manual creation, assignment, a direct comment, an allowed user action or
a live request addressed to them. Inbox archive state does not remove
personal history. Passive reads and machine updates must not advance its
order.

**Steps to reproduce**

1. Open an automated routine task or imported review task without
participating in it.
2. Check the sidebar recents.
3. Let an agent update an older task while newer personal work is in the
list.
4. Before this fix, the automated task appears and background updates
can move older tasks up.

## What Changed

- Add a viewer participation field to bounded navigation summaries.
Reuse the canonical Mine query and existing access checks. The field is
a read projection; no database migration is needed.
- Show only confirmed personal tasks. Remove old nonparticipants after a
successful read. Do not trust participation from local storage or full
task caches.
- Keep status and title snapshots current without changing personal
order. Explicit local edits and comments can advance recency.
- Migrate browser history once to a new storage version. Older tabs
cannot restore machine-driven ordering.
- Keep confirmed rows visible if a new batch request fails. Keep these
facts within the current account and company. Hidden candidates do not
bypass the one-second reorder delay.
- Record sidebar renames as local activity. Place a first passive visit
below existing history.
- Use the saved comment time for comments sent with reassignment or
custom run settings. Pending sends do not advance recency.
- Keep readable subtasks as context beneath qualifying parents. Keep
inbox archive independent of recents.
- Update the design rules, UI behavior notes, regression tests and
Runner source contract pin.

## Verification

- Current head: `92ece0e61859a6c54b714b3611f062ff9b5633bb`. The PR
merges cleanly with current master.
- Latest red/green: both configured-comment cases failed before the fix
and now pass. Reassignment and custom run settings advance recency only
after the comment saves, using its saved time.
- Latest local checks: 339 tests passed across IssueDetail and recent
storage, hooks and sidebar. UI typecheck and diff checks passed.
- Earlier local coverage: 592 targeted API/UI tests passed. After the
master conflict resolution, all 29 overlapping Gmail tests and the
548-input source registry check passed.
- Both Greptile and Paperclip Review report 5/5 on the current head. All
four review threads are resolved.
- Current-head [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44132164473)
passed, including the final verify gate, build, typecheck, browser
shards, serialized tests, Runner checks and all general-test shards. The
workspace-a (1/2) retry passed after the first attempt hit an
asynchronous jsdom/Radix event error in NewAgentDialog.
- `pnpm verify` requires broader CI because DESIGN.md changed. No full
local suite or deployed-browser acceptance was claimed. This change is
not deployed.

## Risks

- Unknown eligibility stays hidden until the summary succeeds. Failed
requests preserve saved candidates and previously confirmed state within
the current account and company, including after batch changes.
- This intentionally removes automated tasks from old personal history
when the viewer has no qualifying participation.
- Readable subtasks under a personal parent remain context. They do not
become independent recent entries from machine activity.
- No task ownership, routine definition, inbox membership or permission
rule changes.

## Model Used

OpenAI GPT-6, running as Codex, with reasoning, repository tools and
focused test execution. No additional model workers were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 11:25:58 -06:00
DottaandClaude Opus 5.5 8172d27089 feat(cloud): browser product analytics through a same-origin PostHog proxy (#22012)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Cloud runs the same app for hosted customers, and the same
UI ships in the npm package.
> - The PostHog project for Cloud gets only server events today. It has
no page views, sessions, replays or people.
> - So the Product and Web analytics pages are empty, and we cannot see
how Cloud users move through the board.
> - This pull request adds browser analytics for Cloud only. It is off
until an operator sets two switches on a Cloud stack.
> - Self-hosted installs do not change: they never download posthog-js
and never serve the PostHog proxy.
> - The benefit is real page views, sessions, heatmaps, Web Vitals and
masked session replays for Cloud, linked to the user and the workspace.

## Linked Issues or Issue Description

**Subsystem affected**
Board UI (analytics bootstrap and privacy markers), server session
payload and a new same-origin proxy route, shared validators.

**Problem or motivation**
Cloud sends only server events to PostHog. There is no browser SDK, so
PostHog has no page views, sessions, replays or identified people for
Cloud. We cannot see where Cloud users get stuck.

**Proposed solution**
The signed-in session gives the UI a PostHog config only on a
Cloud-managed stack with `PAPERCLIP_POSTHOG_PROJECT_TOKEN` and
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`. The UI loads posthog-js after the
page settles. Events go through a strict same-origin proxy at `/_ph`, so
ad blockers do not drop them. Replay masks all text and all inputs by
default.

**Alternatives considered**
Bake a key into the UI build: refused, because the same build ships to
self-hosted users. A generic reverse proxy: refused, because it would
forward any path. Mask only content areas: refused, because a missed
surface would leak; masking is deny by default.

**Additional context**
This is the browser part of the Cloud direct PostHog plan. The server
direct sender is already on master (#22006). Fleet configuration is a
separate change. PostHog person deletion on account deletion is a
follow-up.

## What Changed

- Uses the shared Cloud PostHog config (`resolveCloudPostHogConfig` in
`server/src/services/cloud-posthog-config.ts`, already on master from
the server direct-sending change). This PR does not change it.
- `server/src/services/cloud-posthog-browser.ts`:
`resolveBrowserAnalyticsConfig` returns `{ token, apiHost: "/_ph",
uiHost, environment, stackId, replayEnabled }` only when
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`. `environment` is always set: the
explicit variable, else `staging` for a `*.staging.paperclip.app` host,
else `production`.
- `GET /api/auth/get-session` carries the new optional
`browserAnalytics` field (the existing signed-in bootstrap request, so
no new request on the cold path). `packages/shared` has
`browserAnalyticsConfigSchema` and the session schema field.
- `server/src/routes/posthog-proxy.ts`: a strict proxy, mounted before
the body parsers and only when browser analytics is on. It forwards only
these requests and answers 404 to all others without contacting PostHog:
- `POST /e/`, `/i/v0/e/`, `/batch/`, `/s/`, `/flags/`, `/decide/` to the
ingest host.
- `GET /array/<token>/config(.js)` and `GET
/static/[<version>/]<name>.js` to the assets host (derived from the
ingest host).
- `stream.pipeline` joins the streams, so an error on either side
(PostHog closes the connection partway through a response, or the
browser goes away) closes both and never goes unhandled. It builds a new
header set from an allowlist plus `X-Forwarded-For`. It never forwards
the inbound Host, Cookie or Authorization headers. It drops `Set-Cookie`
from responses. Bodies stream; nothing is buffered. It refuses bodies
over 64 MiB (413, also for chunked bodies that grow past the limit) and
limits requests in flight to 16 per client address and 256 in total
(429).
- `ui/src/lib/analytics/product-analytics.ts` (the only module that
talks to posthog-js) and `ui/src/components/ProductAnalyticsGate.tsx`:
- posthog-js loads after the page settles (or at a 15 s deadline), only
when the session has a config. It loads as PostHog's browser script
`/_ph/static/array.js` through the proxy, the same way posthog-js itself
loads its extensions (recorder, web vitals). It is not an npm dependency
and is in no bundle chunk.
- Init: page views on history change, page leave, autocapture, Web
Vitals, heatmaps, `person_profiles: "identified_only"`,
`tracing_headers` for our own host only, replay only when
`replayEnabled`. Surveys, product tours, web experiments and
conversations are off.
- Super properties on every event: `deployment_source: "cloud"` and
`environment`.
- After sign-in: `identify(session.userId, { email, name })` (person
properties only). `session.userId` is the authenticated actor's user id
verbatim (for example `paperclip-id:...`): `GET /api/auth/get-session`
copies `req.actor.userId` into it, and server events use that same actor
id as `distinct_id`, so browser and server events land on one person.
Tests: `auth-routes.test.ts` (the session returns the actor id verbatim)
and `ProductAnalyticsGate.test.tsx` (identify uses it) and
`group("workspace", stackId)`. On sign-out: `reset()`, then the super
properties again. `useSignOut` calls it before it leaves the page,
because Cloud sign-out navigates away while the session query is still
populated.
- Privacy (`ui/src/lib/analytics/privacy.ts`), deny by default:
- Replay masks every text node (`maskTextSelector: "*"` with
`maskTextFn`) and every input value (`maskAllInputs`).
- Only elements with `ANALYTICS_UNMASK_PROPS` show their text: sidebar
items whose label is a fixed string (`analyticsChrome`, added at the 99
call sites with literal labels), and a shared `Button`, `Label`,
`TabsTrigger` or `EmptyState` only when its caller passes
`analyticsChrome`. None of them opts in by default, because they also
render customer text (agent names, document titles). Sidebar items with
dynamic labels (recent tasks, agents, projects) stay masked.
- Replay masks attribute values too (`maskAttributeFn`, deny by
default). Only layout attributes (`class`, `style`, `role`,
`data-state`, SVG geometry), rrweb metadata and stylesheet links keep
their values. `title`, `aria-label`, `alt`, `placeholder`, `src`,
`href`, `data-*` and element references (`id`, `for`, `aria-labelledby`,
`aria-describedby`, `aria-controls`, which often embed names or slugs)
are always masked, even on unmasked chrome. So the organization
switcher's `aria-label`, task titles in `title` attributes, image
sources and download links with file paths never reach a replay.
- `before_send` (`scrubCapturedEvent`) drops the `title` property from
every event, because posthog-js copies `document.title` (task and
organization names) into page views. Every URL string in every event,
including replay meta events, keeps only its origin and path: query and
hash values never leave (they can hold tokens, `code`/`state`, file
paths or search text), and the token segment of `/board-claim/<token>`
and `/invite/<token>` is redacted. It recurses only into plain objects
and arrays, so the `Date` timestamp stays intact.
- `ANALYTICS_MASK_CLASS` keeps customer content masked even inside
unmasked chrome: `MarkdownBody` (task and issue bodies, comments, chat
messages, documents and artifact markdown), `MarkdownEditor`, the task
chat thread, the issue chat thread, the comment timeline, the file
viewer (source, markdown and CSV views) and the organization name in the
organization switcher. Text in any `contenteditable` editor is always
masked.
- `ANALYTICS_NO_CAPTURE_CLASS` (`ph-no-capture`, a blank box in replay)
on every password-type `Input` (about 40 secret, token and API key
fields), environment variable values and new secret values.
- Autocapture never records element text or attribute values
(`mask_all_text`, `mask_all_element_attributes`).
- Connected-account content (Google user data and Slack data: email
bodies, calendar events, Slack messages in tasks, chat, tool results,
previews) never appears unmasked in a replay. Replay masks all text by
default; nothing that renders connected-account content is in the unmask
allowlist; autocapture records no element text anywhere
(`mask_all_text`). `ui/src/lib/analytics/unmask-allowlist.test.ts` pins
the allowlist: only 7 files may spread the unmask attribute, sidebar
opt-ins need a literal label, and any `analyticsChrome` opt-in on
`Button`/`Label`/`TabsTrigger`/`EmptyState` fails until it is added to a
reviewed list (empty today). The same test renders Gmail, Calendar and
Slack-style tool results next to unmasked navigation and checks they are
masked.
- Guard test: UI code must never call `capture()` with a
`PAPERCLIP_EVENTS` name, so each event has one sender.
- Docs: `doc/cloud-browser-analytics.md` (switches, per-stack opt-out,
proxy allowlist, identity, privacy) and a link from
`doc/observability.md`.
- No new npm dependency and no lockfile change: posthog-js comes from
PostHog through the proxy at run time. (An earlier revision added
`posthog-js` to `ui/package.json`; the Runner support suite requires
every `ui` dependency in the reviewed lockfile, which a pull request
cannot update, so the script approach replaced it.)

## Verification

- `pnpm verify`: see the result below.
- Focused tests:
- `server/src/__tests__/cloud-posthog-browser.test.ts`: exposure matrix
(self-hosted with every switch set, Cloud without token, without stack
ID, without the browser switch, switch not `1`, enabled, replay off),
environment fallback, proxy mount only when enabled, allowed routes, 404
for disallowed methods and paths (including `//host` and `..` paths)
with no upstream contact, Host/Cookie/Authorization stripped,
`Set-Cookie` dropped, 502 when PostHog is down, and no unhandled error
when PostHog closes a response partway (red on the old `pipe()` code,
green now).
- `server/src/__tests__/auth-routes.test.ts`: the session carries
`browserAnalytics` only on Cloud with the token and the switch.
- `ui/src/lib/analytics/*.test.ts`,
`ui/src/components/ProductAnalyticsGate.test.tsx`: no import of
posthog-js without a config or when signed out; load after the page
settles, once; identify, group, reset with a mocked posthog; email and
name never become event properties; deny-by-default replay masking;
guard test.
- Real run (local, headless Chromium, production UI build, never the
real PostHog project):
- Self-hosted server, no env: 0 requests for PostHog's script and 0
requests to `/_ph` in 9 cold loads and in a 40 s session with
navigation. `browserAnalytics` is `null`. `/_ph/e/` falls through to the
UI.
- Cloud config: a full Cloud boot needs S3, a managed database and the
control plane. So a small local stand-in served the same production UI
build, mounted the real proxy from `server/src/routes/posthog-proxy.ts`,
added the real `resolveBrowserAnalyticsConfig()` output to the session,
and forwarded other API calls to the local server.
`PAPERCLIP_POSTHOG_HOST` pointed at a local capture server.
- Captured through `/_ph` only (0 requests to any posthog.com host):
`$pageview` on each client navigation (5), `$identify` with `$set: {
email, name }`, `$groupidentify`, `$groups: { workspace: <stack id> }`
on events, `environment` and `deployment_source` on every event,
`$web_vitals`, `$autocapture` with no `$el_text`, `$$heatmap`, and 10
`$snapshot` replay events. A search of every decoded `/e/` and `/s/`
payload (events, replay text and replay attributes) found the email only
in `$identify`'s `$set`, and found none of: the task body, the typed
comment, the task title, the organization name. No Cookie header reached
the capture server. 96 of 113 API requests carried
`X-POSTHOG-SESSION-ID` (the others ran before posthog-js loaded).
- Client-side visits to `/invite/INVITESECRET123` and
`/cli-auth/abc?token=CLISECRET456` reached PostHog as `/invite/REDACTED`
and `/cli-auth/abc` (no query); neither secret was in any payload.
- Disallowed proxy requests (`PUT /_ph/e/`, `GET /_ph/api/projects/`,
`POST /_ph/static/x.js`) answered 404.
- Sample replay payload (decoded rrweb text nodes from a captured `/s/`
request). The task body, the typed comment, the task title, the
organization name and the email were not in any replay payload:

```json
{"text_nodes": [
  {"parentTag": "title", "textContent": "****** ** * *********"},
  {"parentTag": "a",     "textContent": "**** ** **** *******"},
  {"parentTag": "span",  "textContent": "****** **"},
  {"parentTag": "span",  "textContent": "*** ****"},
  {"parentTag": "span",  "textContent": "Tasks"},
  {"parentTag": "span",  "textContent": "Inbox"},
  {"parentTag": "h2",    "textContent": "******* *******"},
  {"parentTag": "p",     "textContent": "****** *** * ******* ** ******"}
]}
```

- Cold load, 9 fresh-profile loads each, median, same stand-in with
analytics off and on:

| | DOMContentLoaded | Board content visible |
| --- | --- | --- |
| Analytics off | 128 ms | 336 ms |
| Analytics on | 126 ms | 334 ms |

Final head. The difference is inside run-to-run noise (earlier pairs:
332/335 ms and 335/348 ms). PostHog's script loads only after the
content is visible.

## Risks

- Inert by default. Without a Cloud-managed stack, a project token, a
stack ID and `PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`, the session field is
`null`, the proxy is not mounted and no analytics code runs. The OSS
package does not change behavior.
- Per-stack opt-out: `PAPERCLIP_POSTHOG_REPLAY_ENABLED=0` turns replay
off; unsetting `PAPERCLIP_POSTHOG_BROWSER_ENABLED` turns all browser
analytics off. Both apply on the next restart.
- Replay text shown in the clear comes only from fixed sidebar labels
and from shared components whose caller opts in. Today no `Button`,
`Label`, `TabsTrigger` or `EmptyState` call site opts in, so replays
show layout, clicks and navigation with most copy masked. Opting in
fixed-label call sites is a follow-up. Review sample replays on staging
before production.
- Image `src` values are masked, so images record as empty boxes.
Sandboxed iframes (HTML artifacts, MCP apps) also record as empty boxes.
Replays therefore show layout and product copy, not customer media.
- PostHog's script is not version-pinned: the assets host serves its
current browser build, as for any PostHog snippet install, and
posthog-js already loads its extensions from that host at run time. It
runs only on Cloud stacks with the browser switch on.
- The proxy needs no sign-in, like PostHog's documented proxies:
posthog-js sends some events while a page unloads, and the proxy can
only reach the two PostHog hosts, on the listed paths, with the public
project token. Body size and in-flight limits bound the load one client
can add.
- posthog-js rejects bots (including headless browsers) on its own,
which is the wanted production behavior.
- `SidebarNavItem.production.tsx`, `Sidebar*.tsx` and the shared `ui/*`
components changed by one attribute each; no visual change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, headless browser runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 11:23:03 -06:00
DottaandClaude Opus 5.5 7f1f0eef67 fix(google-workspace): request free/busy with read & write Calendar, and update connector docs (#22026)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector gives agents Gmail, Drive
and Calendar through one connection, behind an experimental setting
> - The user picks each service as read only or read & write. Paperclip
then requests only scopes from a fixed list of 14
> - Read & write Calendar requested `calendar.calendarlist.readonly` and
`calendar.events`. Our code assumed `calendar.events` also covers
free/busy lookups
> - Google's REST `freeBusy.query` does not accept `calendar.events`. So
the REST `suggest_time` tool cannot read free/busy on a read & write
grant, and falls back to reading each attendee's events
> - This pull request adds `calendar.events.freebusy` (already one of
the 14) to the read & write Calendar selection of the unified
connection, and stops treating `calendar.events` as implying free/busy
> - The benefit is that `suggest_time` on REST can see coworkers who
share only free/busy, and the stored grant no longer claims a permission
that Google did not give. It also brings three Google connector planning
docs up to date

## Linked Issues or Issue Description

Refs #22016, #22017, #22010, #22014

**What happened?**

The read & write Calendar selection of the unified Google Workspace
connection requests `calendar.calendarlist.readonly` and
`calendar.events`. `toolScopes.calendar.suggest_time` lists
`calendar.events` as sufficient, and `googleWorkspaceEffectiveScopes`
treats `calendar.events` as implying `calendar.events.freebusy`.

Google's two references differ:

- REST
[`freeBusy.query`](https://developers.google.com/workspace/calendar/api/v3/reference/freebusy/query),
checked 2026-11-17: "This request allows authorization with at least one
of the following scopes:" `calendar.readonly`, `calendar`,
`calendar.events.freebusy`, `calendar.freebusy`. `calendar.events` is
not in the list.
- Google's hosted Calendar MCP
[`suggest_time`](https://developers.google.com/workspace/calendar/api/v3/reference/mcp/tools_list/suggest_time),
checked 2026-11-17: "Requires one of the following OAuth scopes:"
`calendar`, `calendar.readonly`, `calendar.events`,
`calendar.events.freebusy`, `calendar.freebusy`. Here `calendar.events`
is accepted.

So `calendar.events` is enough for Google's MCP tool but not for the
REST endpoint that our REST `suggest_time` uses.

**Expected behavior**

A user who picks read & write Calendar gets a grant that allows
`freeBusy.query`. A grant that does not have the free/busy permission is
not recorded as having it. Grants made before this change keep working
as read & write.

**Steps to reproduce**

1. Call `googleWorkspaceSelectedScopes({ calendar: "write" })`. It
returns two scopes, without `calendar.events.freebusy`.
2. Call `googleWorkspaceEffectiveScopes(readOnlyCalendarScopes,
[calendar.events])`. It returns `calendar.events.freebusy`, which Google
did not grant.
3. With Calendar on the REST transport, `suggest_time` on a read & write
grant gets an insufficient-scope answer from `freeBusy.query` and uses
the events fallback.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`:
- `googleWorkspaceSelectedScopes` adds selection-level scopes from a new
`SELECTION_ADDED_SCOPES` map. Read & write Calendar adds
`calendar.events.freebusy`. The per-service `calendar.write` profile
(`SERVICE_PROFILES["calendar.write"]`) is unchanged, because the Cloud
broker matches legacy profiles by exact scope set.
- `googleWorkspaceEffectiveScopes` no longer expands `calendar.events`
to `calendar.events.freebusy`. `calendar.events` still implies
`calendar.events.readonly`. `calendar.readonly` still implies free/busy,
which Google's reference allows.
- `googleWorkspaceSelectionForScopes` is unchanged. It needs the legacy
write profile's scopes, so a two-scope grant and a three-scope grant
both read back as read & write.
- `toolScopes.calendar.suggest_time` is unchanged. It still accepts
`calendar.events`: Google's MCP tool accepts it, and on REST the tool
has an events-based fallback that names attendees it cannot read.
- Counts: default all-read-and-write selection is now 10 scopes (was 9);
all read-only stays 8; the union of all 27 selections stays exactly the
14.
- Tests (written first and seen failing on `master`):
- `packages/shared/src/google-workspace-connectors.test.ts`: scopes for
read and for read & write Calendar; the legacy `calendar.write` profile
keeps two scopes; default count 10 and read-only count 8; old two-scope
grant, new three-scope grant and partial consent (user declines
free/busy) all round-trip as read & write and keep `list_calendars`,
`list_events`, `suggest_time`, `create_event` and `delete_event`; no
recorded free/busy that Google did not grant. The existing exhaustive
test still checks that no selection requests a scope outside the 14.
- `server/src/__tests__/tool-access-service.test.ts`: customer-owned
Workspace OAuth with read & write Calendar asks Google for all three
scopes. With and without free/busy in Google's answer, the connection is
active, the Calendar tools are listed, the grant stores exactly what
Google granted, and it reads back as read & write.
- Stale tests on `master` fixed: `#22014` made Gmail 9 of 9 in the REST
tool set while `#22016` still expected 3 Gmail REST tools.
`google-workspace-rest-dispatch.test.ts` and
`google-workspace-rest-transport.test.ts` now derive the Gmail list from
the allowlist and use Chat as the service with no REST tools.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md` and
`doc/connections/tool-method-permission-reviews.json` say ten scopes and
explain the Calendar change.
- Docs: `doc/memory/google-connector-memory.md` (shipped PRs #22010,
#22014, #22016, #22017; three Nov 17 decisions; "Where Google content is
kept" replaced with #22010's inventory, what deletion keeps and its
limits; Scopes section; REST tools status table; still-open list),
`doc/plans/2026-11-17-google-workspace-production-checklist.md` (help
page and retention audit ticked with #22010; REST tool progress),
`doc/plans/2026-11-17-google-workspace-code-fixes.md` (fix 6 done with
limits; section 9 status table; fix 10 note).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

Not changed:

- `server/src/google-workspace-rest-tools/**`. Another change owns that
folder. Its README open question about this scope is now resolved and
can be removed there.
- The picker disclosure (`googleWorkspaceDisclosure.ts`). Read & write
Calendar already says agents "can see your calendars, events and
free/busy times". With this change that line is accurate on REST too.

Behavior for existing grants: a saved read & write grant without
free/busy still reads back as read & write, keeps every Calendar tool,
and refreshes without change. Nothing prompts the user to reconnect. A
user who chooses read & write Calendar in the picker is asked for the
new scope. An account-page reconnect without a new choice still restores
the exact saved request, as before.

Cloud broker and Google Console: no change needed.
`calendar.events.freebusy` is one of the 14 in `workspace.all`, the
broker accepts any nonempty subset of `workspace.all`
(`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`), and the legacy
`calendar.write` profile's exact scope set did not change.

## Verification

- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts src/app-definitions.test.ts`: 55
passed. On `master` the new shared tests fail (3 failed).
- `cd server && npx vitest run src/__tests__/tool-access-service.test.ts
-t "free/busy"`: 2 passed. With `master`'s
`google-workspace-connectors.ts`: 2 failed.
- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts
src/services/google-workspace-rest-dispatch.test.ts`: 29 passed (4
failed on `master` before this change).
- `cd ui && npx vitest run
src/features/connections/GoogleWorkspaceServices.test.tsx
src/pages/apps/AppDetail.test.tsx`: 115 passed.
- `cd packages/shared && npx tsc --noEmit -p .`: clean.
- `pnpm verify` exits 2 (a global CI input changed), so required CI
supplies the broad coverage.
- Docs facts checked against `gh pr view` for #22010, #22014, #22016,
#22017, the code, and paperclipai/paperclip-website PR 197 (still a
draft; does not mention OpenRouter).

## Risks

- Low risk. The submitted set of 14 does not change, and the broker
contract does not change.
- New read & write Calendar grants record `calendar.events.freebusy`
next to `calendar.events`. Code that reads stored scopes sees one more
scope. All consumers (callback, refresh, discovery, picker pre-fill) are
covered by the tests above.
- A read-only Calendar request on an account that Google reports only
`calendar.events` for no longer records free/busy. That matches what
Google granted; `suggest_time` is then not offered on that grant.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use, web fetch and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:40:58 +00:00
DottaandClaude Opus 5.5 01451fae5b Let plugins add connector types to the Connectors catalog (#22004)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use external services through connectors: a catalog of
connector types, connections (accounts) per type, and per-agent
installs, policies and audit in the tool gateway
> - Today only core can add a connector type. Every type is a JSON
definition compiled into `@paperclipai/shared`
> - Plugins can add agent tools, but those tools skip connections,
grants, per-agent installs and the Connectors page
> - This pull request lets a plugin add connector types with the same
definition schema, and makes them behave exactly like built-in
connectors
> - The benefit is that a connector can ship as a plugin, and an
operator controls it with the same Connectors page and the same access
rules

## Linked Issues or Issue Description

**Subsystem affected**
Plugins, Connectors (tool access), tool gateway

**Problem or motivation**
A plugin cannot add a connector type to the Connectors catalog. Plugin
tools (`tools` in the manifest) do not use connections, credentials per
account, agent installs, policies or the Connectors page.

**Proposed solution**
Add a manifest field `apps` (gated by a new `connectors.provide`
capability). Each entry uses the built-in app-definition schema. The
host namespaces the slug by plugin key and runs setup, OAuth, discovery,
grants, policies and audit through the existing code. This is the first
of two pull requests. The second adds a `plugin` transport so a plugin
worker can serve the tools itself.

**Activation rules (product requirements)**
1. Installing or enabling a plugin grants no tools. It only adds
connector types to the catalog.
2. Connecting an account grants no tools.
3. An agent gets the tools only from an agent install or a company
install of an active connection. Discovery and invocation both enforce
this.
4. A disabled or uninstalled plugin (or an upgrade that removes the app)
hides its connector types and connections from the Connectors APIs and
they reach no agent. Saved setup survives disable and comes back on
re-enable. Uninstall keeps rows archived for audit and revokes
credentials.


**Alternatives considered**
Keep plugin tools (`tools`) as the only plugin path. Rejected: they
bypass connections, per-account credentials, per-agent installs and the
Connectors page.

## What Changed

- `packages/shared`: new capability `connectors.provide`; manifest
`apps` validated with `appDefinitionSchema`. The validator rejects apps
without the capability, duplicate slugs, host-derived fields (`plugin`,
`aiConnectionRouter`, `experimentalFeatureKey`, `availability`), and
methods that need provider code in the host (REST adapters, chat
channels, AI runtimes).
- `packages/shared/src/plugin-apps.ts`: reversible namespaced slugs
(`plugin-<hex plugin key>-<slug>`), the catalog entry builder, and
`reviewedToolRisk`.
- `packages/shared/src/app-definitions.ts`: catalog lookups
(`getConnectableAppDefinition`, `getAppStoreDefinition`, ...) also
resolve the current plugin apps. The set is replaced on each read of
plugin state, so a built-in app is never shadowed.
- App definitions can declare `reviewedTools` (reviewed
read/write/destructive names and the risk for unlisted tools).
Circleback's reviewed risk moves from
`server/src/services/connectors/circleback/risk.ts` into
`circleback.json`, with the same classification. A copy of the
definition now classifies tools the same way.
- `validators/json-schema.ts`: `jsonSchemaSchema` moves to its own
module to break an import cycle between the plugin and app-definition
validators (still re-exported from `validators/plugin.ts`).
- `server/src/services/plugin-app-catalog.ts`: reads ready plugins from
the `plugins` table and answers "is this connection's connector
available". It does not trust in-memory lifecycle events, because
several lifecycle managers and processes can share one database.
- Gateway: connected-tool discovery and `resolveConnectionGrant` drop
connections from an unavailable plugin connector. Built-in connections
do not read the plugins table.
- Connectors APIs: the gallery lists ready plugin apps (with the same
Cloud connector profile filter as core); `listConnections` and
`listApplications` hide unavailable plugin connections; setup entry
points refresh plugin apps first.
- Plugin uninstall archives the plugin's connections through the
standard connection removal (grants revoked, owned credentials deleted,
rows kept).
- UI: the gallery response registers plugin apps for the setup screens;
`/apps/connect?source=plugin-…` loads the gallery first and shows an
unavailable state when the plugin is disabled.
- Docs: `doc/plugins/PLUGIN_SPEC.md` section 11A and the capability
list.

Design choices where the spec left room (picked to match "a plugin
connector behaves like a built-in connector"):
- Only `ready` plugins provide connectors. `error` and `upgrade_pending`
hide them, the same as the AI connection router plugins.
- `rest_api` is rejected for plugin apps in this PR. Core REST
connectors need reviewed adapter code in the host, so a definition alone
cannot work. The `plugin` transport in the next PR covers that case.
- Disabled plugin connections are hidden from list endpoints. Reading
one connection by id still works, so audit links keep resolving.

## Verification

- `packages/shared`: `npx vitest run src/plugin-apps.test.ts
src/validators/ src/app-definitions.test.ts` (416 passed). New tests:
manifest validation, slug namespacing, no shadowing of built-in apps,
reviewed risk.
- `server`: `npx vitest run
src/__tests__/plugin-connectors-circleback-parity.test.ts` (new). A
test-only plugin
(`server/src/__tests__/fixtures/plugin-connectors/circleback-parity/`)
ships an unmodified copy of `circleback.json`. The same journey runs
against built-in Circleback and the plugin copy with a recorded DCR
OAuth + MCP fixture, and must give identical catalog risk/status,
suggested defaults and per-agent tool sets. It covers each rule:
- Rule 1: the gallery adds only the namespaced type; no connections
exist; no agent tool changes.
  - Rule 2: two signed-in accounts, no agent sees a tool.
- Rule 3: an install for agent A gives only A the tools of that
connection; agent B is denied on invocation; each account calls the
provider with its own credential.
- Rule 4: disable hides the type, both connections and applications,
removes tools from discovery and invocation (404) and blocks new setup,
while built-in Circleback is unchanged; installs and grants are
byte-identical; re-enable restores tools with the same names; an upgrade
that removes the app hides it; uninstall archives both connections,
revokes grants and deletes owned secrets.
- Red/green: removing the gateway gate or the declared review makes the
test fail.
- An env-gated live check (`PAPERCLIP_CIRCLEBACK_LIVE=1`) starts the
real Circleback sign-in for both connectors and compares the requests.
It never runs in CI. I did not run it for this PR.
- Regression: `ai-connection-router`, `connector-tool-policy`,
`mcp-apps`, `plugin-lifecycle-restart`, `tool-access-service`,
`braintrust-connection`, `remote-mcp-connectors`,
`connection-intents-service`, `tool-gateway` (796 passed).
- `ui`: `app-connect-policy.test.ts`, `AppsConnect.test.tsx` (new plugin
cases) pass.
- `tsc --noEmit` for shared, server, ui (only pre-existing ungenerated
Storybook catalog errors), plugin SDK and CLI.
- `pnpm verify --explain` selects a broad plan because a doc under
`doc/plugins` changed; CI supplies that coverage.

## Risks

- No migration. Built-in connectors only gain one string check per
discovery; the plugins table is read only when a plugin connection is
present.
- Circleback's reviewed risk is now data. The behavior is the same, and
the existing policy tests and new tests cover it.
- Connection intents (agent connection requests in chat) still search
only built-in apps. Plugin connectors can be set up from the Connectors
page. This is a follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file editing, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 15:51:39 +00:00
DottaandClaude Opus 5.5 c652c813bc feat(google-workspace): per-service transport switch for REST-backed tools (off by default) (#22016)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use Google Workspace through one built-in `google-workspace`
connection and the tool gateway
> - Today every Google tool call goes to Google's hosted Workspace MCP
servers, which are a Developer Preview and cannot be offered to the
public
> - #22000 added Paperclip's own Gmail tools over Google's REST APIs in
`server/src/google-workspace-rest-tools/`, but nothing calls them yet
> - This pull request adds one per-service transport switch and connects
the gateway to the REST tools behind it. Every service stays on Google's
MCP servers, so production behavior does not change
> - The benefit is that a service moves to the REST tools with a
one-line, reviewed change once all its tools exist, and moves back the
same way

## Linked Issues or Issue Description

Refs #22000

**Subsystem affected**
Tool gateway and catalog discovery for the built-in Google Workspace
connection.

**Problem or motivation**
Google's Workspace MCP servers are a Developer Preview. Paperclip
launches on its own tools over Google's REST APIs. The REST tools exist
(#22000), but the gateway cannot call them, and there is no way to
choose them per service.

**Proposed solution**
Add a per-service transport setting (`google_mcp` or `paperclip_rest`)
in the shared Google Workspace module. Add one server adapter that is
the only importer of the REST folder. When a service is set to
`paperclip_rest`, the gateway call path and discovery use the adapter in
process. All gateway controls stay the same.

**Alternatives considered**
A loopback or hosted MCP endpoint for the REST tools. It adds a
connection, JSON-RPC framing and the protocol probe to every call, and
authenticated public deployments refuse loopback addresses. An
environment variable or instance setting. A plain constant keeps every
flip a reviewed code change.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: adds
`GOOGLE_WORKSPACE_TOOL_TRANSPORT` (every service is `google_mcp`),
`transport` on the result of `googleWorkspaceToolTarget`, and
`googleWorkspaceServiceToolNames`.
- New `server/src/services/google-workspace-rest-dispatch.ts`: the only
importer of the REST folder. It keeps only the bearer token from the
gateway's credential headers, passes the gateway's egress-guarded fetch
and cancellation signal, and returns either the MCP result or a token
problem. `googleWorkspaceRestTransportGaps()` lists allowlisted tools
that a `paperclip_rest` service does not implement. Today Calendar is
eligible (9 of 9) and Gmail is not (3 of 9); nothing is flipped.
- `server/src/services/tool-gateway.ts` (`executeRemoteHttpTool`): for a
`paperclip_rest` tool, the gateway rechecks the current grant, calls the
adapter, and returns through the same result handling as the MCP path.
Google's insufficient-scope answer maps to `oauth_insufficient_scope`. A
rejected token forces a refresh and returns
`oauth_refreshed_retry_required` without a replay. A passed deadline
returns `tool_timeout`. The audit record names the protocol `Google
Workspace REST`. The grant recheck is now one shared helper for both
paths. A timed-out Google Workspace write, on either transport, now
returns `tool_timeout` with a warning that the change may or may not
have been made; a timed-out read keeps the plain timeout message.
- `server/src/services/tool-access/catalog/discovery.ts`: for a
`paperclip_rest` service, discovery and health run one REST probe with
the caller's token, then list the REST tools under the same names and
scope filter. A 401 refreshes and probes again. A second 401 on a
managed grant marks it `needs_reauthorization`. Any other probe failure
makes only that service unavailable.
- `server/src/google-workspace-rest-tools/`: adds `probe.ts` and its
export. Each service has an ordered list of cheap reads with the scopes
that allow each (Gmail: `users.getProfile`; Calendar: one calendar list
entry, one primary-calendar event, or one minute of free/busy). The
probe uses the first read the grant allows. A grant that allows none is
listed without a probe, and an insufficient-scope answer never drops a
service. It also and lets `boundary.test.ts` allow exactly the adapter
as the importer. The README now describes the implemented seam, how to
flip a service, and the removal procedure.
- `doc/connections/GOOGLE-WORKSPACE.md`: describes the transport
setting.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared files.

## Verification

- `cd server && npx vitest run
src/__tests__/google-workspace-rest-transport.test.ts`: 15 tests. They
were red before the gateway and discovery changes (10 failed), and green
after. With Gmail set to `paperclip_rest` through a test-only change of
the map, the tests use the real managed setup, discovery, gateway,
policy and screening with a fake Google fetch. They prove: discovery
lists the same tool names filtered by scopes after one probe; a read
runs with the caller's token and is audited; a draft waits for approval
and then runs; a tool outside the grant and a narrowed grant are refused
before any Google request; Google's insufficient scope maps to
`oauth_insufficient_scope`; a 401 refreshes the token, is not replayed,
and the retry uses the new token; an injected email is blocked; a passed
deadline aborts the Google request and returns `tool_timeout`; the
experimental-feature gate stops calls and discovery; a failed Gmail
probe leaves Docs usable and Gmail comes back; a lasting 401 asks for
reconnection. A timed-out draft carries the uncertain-write warning in
the error, the invocation record and the audit entry on both transports,
and an approved REST draft whose request was lost returns the warning.
With the default map, a call goes to `gmailmcp.googleapis.com` and the
adapter is never called.
- `cd server && npx vitest run
src/services/google-workspace-rest-dispatch.test.ts
src/google-workspace-rest-tools`: eligibility (no partial flip),
bearer-only forwarding, signal and fetch pass-through, token outcome
mapping, probe, boundary.
- `cd packages/shared && npx vitest run
src/google-workspace-connectors.test.ts`: every service defaults to
`google_mcp`.
- Existing Google tests are unchanged and pass:
`google-workspace-injection-screening.test.ts`, the Google cases in
`tool-gateway-service.test.ts` and `tool-access-service.test.ts` (142
Workspace/Google/Gmail cases).
- `npx tsc --noEmit` for `packages/shared` and `server`: clean.
- `pnpm verify`: exit 2 (a doc under `doc/` is a global CI input), so
required CI supplies broad coverage.
- Speed, stubbed fetch, 300 calls after warm-up: the adapter adds about
15 µs per call. A full `gmail__get_message` through `executeTool` took a
median of 18.2 ms on REST and 18.3 ms on MCP (p90 21.6 ms and 22.3 ms).
The REST path adds no waits, no extra round trips and no dynamic
imports.

## Risks

- Low risk for production: every service stays on `google_mcp`, and the
default-map test proves the adapter is not called.
- The server now loads the REST folder at start. It uses only modules
the server already loads (MCP SDK, zod).
- A later flip must add a probe and the API host for each new REST
service. The `Record` type of the probe table makes typecheck fail when
a REST service has no probe.
- On `paperclip_rest`, MCP resources, MCP Tasks and elicitation do not
exist for that service.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 09:49:49 -06:00
DottaandClaude Opus 5.5 d1b5cfe038 feat(connections): let users delete data read through a connection (#22010)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call connected apps, such as Google Workspace, through
Paperclip's tool gateway
> - Paperclip saves part of each tool result, approval previews, app
views and the full run transcripts that contain tool output
> - No user action deleted this data. Removing the connection left it in
place
> - Google's restricted-scope review requires that users can delete the
data an app obtained, and that the privacy policy says how
> - This pull request adds "Delete data read through this connection"
for any connection, on the connection's page and as an option when you
remove it
> - The benefit is that the picker text ("They stay there until you
delete them") is now true, and the privacy policy can link to a help
page

## Linked Issues or Issue Description

**Subsystem affected**

Connections (tool gateway audit tables, run logs, run events) and the
Connectors UI.

**Problem or motivation**

Paperclip stores content from a connection's tool calls in several
places and gives users no way to delete it:

- `tool_invocations.arguments_summary/result_summary` and
`tool_call_events.arguments_summary/request_summary/result_summary` keep
the first ~4,000 characters of each redacted value
(`summarizeToolValue`, `tool-access-policy.ts` `redactArguments`). No
retention limit.
- `tool_action_requests` keeps `canonical_arguments_summary`,
`preview_markdown` and `signed_arguments`. The linked approval card
(`issue_thread_interactions`) keeps copies in `payload.detailsMarkdown`,
`payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson` and
`result.toolAction.resultSummary`.
- `tool_mcp_app_views` keeps `tool_input` and `tool_result` for MCP app
views.
- Run transcripts hold full tool output: the run log (`local_file` with
an optional S3 mirror, or `s3_segments`/`s3_archive` object journals),
`heartbeat_run_events.message/payload`, and
`heartbeat_runs.stdout_excerpt/stderr_excerpt`. Neither run-log store
had a delete path.
- Connection removal (`removal.ts`) archives the connection and revokes
credentials. It deletes no content.

**Proposed solution**

A durable, resumable, company-scoped deletion request per connection.
See "Design" below.

**Alternatives considered**

- Delete synchronously in the request. Rejected: an interrupted request
leaves partial state that needs a person to retry it.
- Delete only the run events that mention the connection. Rejected: tool
output also reaches the model's later messages, so the safe unit is the
whole run transcript.
- Delete the audit rows. Rejected: counts, outcomes and hashes feed the
activity history and usage counts.

## Design

**Rule.** The job never drops a reference to content it has not
confirmed deleted. It is marked complete, and writes the
`tool_connection.content_deleted` activity entry, only after every step
succeeded. A step that cannot confirm its work throws. The request then
stays pending, the UI says "Paperclip keeps trying until it finishes",
and the worker retries it.

**Store inventory.** Every place that stores tool arguments, results or
previews. I found them from `summarizeToolValue`, `redactArguments`,
`writeAudit`, and every column or JSON field named like summary,
preview, arguments, result, toolInput or toolResult.

| Store | Content | Cleared by this PR | Proof |
| --- | --- | --- | --- |
| `tool_invocations.arguments_summary`, `result_summary`, `progress` |
Redacted request/result preview, provider progress | Yes: `summary`
becomes `""` with `contentDeletedAt`; `progress` null; size and sha256
kept | content-deletion test 1 |
| `tool_call_events.arguments_summary`, `request_summary`,
`result_summary` | Same previews | Yes, same | test 1 |
| `tool_access_audit_events.details.argumentsSummary` (`writeAudit`) |
Argument preview | Yes, same; decision, ids, size, hash kept | test 1 |
| `tool_action_requests.canonical_arguments_summary`,
`preview_markdown`, `signed_arguments` | Approval preview and arguments
| Yes, for settled requests | test 1, card retry test |
| Approval card in `issue_thread_interactions`:
`payload.detailsMarkdown`, `payload.toolAction.previewMarkdown`,
`payload.toolAction.argumentsSummaryJson`,
`result.toolAction.resultSummary` | Approval preview, arguments,
executed result | Yes. Cards are found from every settled request of a
batch, in the same transaction | test 1, card retry test |
| `heartbeat_runs.context_snapshot` and `agent_wakeup_requests.payload`
(`toolAction(s)[].resultSummary`,
`paperclipAgentMessage.untrustedToolResults[].resultSummary`) | Approved
action result delivered to the agent | Yes, for settled runs and settled
wakes | test 1 |
| `tool_mcp_app_views.tool_input`, `tool_result`, `model_context` | App
view input, result, shared context | Yes: rows deleted | test 1 |
| Run log: object journal, archive journal, original `local_file` log
and its legacy S3 mirror | Full transcript | Yes, every copy, for each
settled run that called the connection or received an approved result |
run-log-store tests, test 1 |
| `heartbeat_run_events.message`, `payload` | Transcript events | Yes,
except structural events (lifecycle, terminal, presentation, usage,
compaction, session/turn start, `native.*`) | test 1 |
| `heartbeat_runs.stdout_excerpt`, `stderr_excerpt`, and every string
`summary`, `result`, `message`, `text`, `output`, `content` or `body` at
any depth of `result_json` (including `nativeResult`, `acceptedResult`,
`semanticResult` and their `continuation`) | Output excerpts and final
answer | Yes; schema and control fields kept | test 1, canary |
| `approvals.payload` (formal board approval) | Tool name, risk and
hashes only | Not needed: no content | — |
| `tool_mcp_tasks` | Request hash and sealed provider task id; no result
| Not needed: no content | — |
| `tool_mcp_elicitations.questions` and their question cards | Questions
the provider asked a person | No. These are provider prompts, not tool
results, and the answered question stays in the task. Follow-up if
wanted | — |
| `tool_mcp_skills`, `tool_mcp_skill_files`, `tool_catalog_entries` |
Skills and tool descriptions the provider publishes | No. These are
provider definitions, not data read through tool calls | — |
| Transcripts of later runs that read an app view's `model_context` |
Shared app context | No. Nothing records which runs read it. The help
page says so. Follow-up | — |
| `issue_comments.body`, issue documents (including continuation
summaries), files, chat messages | Task content written by agents or
Paperclip | No, by design. The user removes it by editing or deleting it
or the task. The confirmation says in bold that anything agents wrote
into tasks is not removed, and so does the help page. The canary
excludes `issue_comments.body` by name for this reason | canary asserts
it is kept |
| Raw provider traces | Full provider traffic | No. Admin opt-in,
encrypted, expire within 24 hours | — |
| Feedback exports already sent | Run logs the user chose to share | No.
Already outside Paperclip | — |
| Content of runs of a deleted task or company | Same stores | No.
Follow-up | — |

**Durable request (schema change).** I added one table,
`tool_connection_content_deletions`, and migration `0490`. I followed
`doc/DATABASE.md`: edited the schema, exported it from
`packages/db/src/schema/index.ts`, ran `pnpm db:generate`, and ran the
migration drift and compatibility tests. The classifier marks it
`online_compatible` (a new table and its indexes only). No existing
table can hold a durable per-connection request with a cutoff time,
cursor, counts and retry state. Without it, an interrupted deletion
could resume only when a person pressed the button again.

**Resume after restart.** The worker uses the shared
`registerDeadlineWork` lifecycle on a new
`tool-connection-content-deletion` queue. The request insert signals the
queue in the same transaction. At server start the worker runs once,
`next()` finds any pending row, and processing continues from the saved
phase and cursor. Nobody presses anything. `next()` returns a deadline
only while a pending row exists, so an idle instance sends no recurring
query. A failure records `attempts` and `last_error` and retries after
30 seconds.

**One run selection.** `runsWithStoredContentSql` selects the runs whose
transcripts are deleted: runs that made a tool call or tool-call event
through the connection, and runs whose context lists one of the
connection's approved action requests, limited to runs that still hold a
log, an excerpt, result text or a content event. The dialog count, its
in-progress warning and the job's `runs` phase all use this one
function.

**Phases.** `invocations`, `events`, `auditEvents`, `appViews`,
`deliveredResults` (50 action requests per page; matching runs and wakes
are locked with `FOR UPDATE` before their documents are rewritten),
`runs` (the shared selection, 50 run ids per page, then one run at a
time), `done`.

**Canary test.** `tool-connection-content-deletion-canary.test.ts` puts
a unique marker in a tool call's arguments and result, an approved
action with a preview, a result delivered to a later run and its wake, a
native run that quotes the marker in its answer and events, a run log
migrated from a local file with an S3 mirror to an object journal, an
MCP App view and the audit tables. It checks that the marker is first
present in the database, the object store, the legacy mirror and the log
files. After the deletion it scans every `text`, `varchar`, `json` and
`jsonb` column of every table found through `information_schema` (no
hand-written table list), every object in both stores and every file
under the log directory. Any hit fails with the table and column or the
key. It runs three times: a complete deletion, a deletion interrupted
twice by storage failures and resumed, and two deletions for different
connections on one task at the same time. The only exclusion is
`issue_comments.body`, named with its reason.

**Batching and time budget.** Tool rows are read in pages of 200 with
keyset pagination on the existing `(company_id, created_at)` indexes,
bounded by the request time. Each page's writes and the saved progress
commit in one transaction. Counts are therefore exact, and a crash
repeats at most one page, idempotently. Runs advance one at a time from
the shared selection. Each 1,000-event page is its own step. Log removal
is its own step and runs before its progress transaction, so object
storage calls never run inside a database transaction. The 10-second
budget is checked after every step. Nothing changes on the agent boot or
tool-call path.

**In-flight work.** A run counts as settled only when its status is
terminal, `finished_at` is set, cost accounting is done, and no native
finalization is open. Calls, approvals, events, app views, delivered
results and transcripts tied to other runs or wakes are skipped and
counted. The confirmation says "N runs are still working and will be
skipped. Delete again after they finish."

**Permissions.** Board users with configure access (connection manager
or creator) can delete, like `DELETE /tool-connections/:id`. Agents,
viewers and other companies are refused. The activity entries
`tool_connection.content_deletion_requested` and
`tool_connection.content_deleted` hold counts only.

**UI reads.** The dialog loads counts once when it opens. While a
deletion runs, it polls `GET
/tool-connections/:id/stored-content/deletion`, a single indexed row
read, every 2 seconds. Polling stops when the job completes or the
dialog closes. The Delete button is not gated on counts.

## What Changed

- `packages/db`: new `tool_connection_content_deletions` table and
migration `0490` (online compatible: new table and its indexes only).
- `packages/shared`: `ToolConnectionStoredContent`,
`ToolConnectionContentDeletion` and counts types; optional
`contentDeletedAt` on `ToolRedactedValueSummary`.
- `server`: `RunLogStore.remove`, which deletes every copy of a run's
log (`composeRunLogStore`); `content-deletion.ts` service; deadline
worker wired into startup and pre-drain; `GET
/tool-connections/:id/stored-content`, `POST` and `GET
/tool-connections/:id/stored-content/deletion`, and `DELETE
/tool-connections/:id?deleteStoredContent=true`; OpenAPI entries.
- `ui`: "Delete data" row on the connection's Permissions tab with a
confirmation that shows counts, skipped runs, and "This cannot be
undone"; "Also delete data read through this connection" checkbox in the
Connectors removal dialog.
- Docs: user help
`docs/guides/board-operator/deleting-connection-data.md` (for the
privacy policy link) and an API note in
`doc/connections/CONNECTOR-PLAYBOOK.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json` refreshed for the
shared type change.

`server/src/routes/tool-access.ts` (3,010 lines) and
`ui/src/pages/apps/Browse.tsx` (1,722 lines) grow by about 40 and 20
lines. The routes reuse that file's connection access checks; the dialog
reuses the existing removal confirmation. The new logic lives in its own
files.

## Verification

- `server/src/__tests__/tool-connection-content-deletion.test.ts`
(embedded PostgreSQL): content is gone from every "Yes" store in the
inventory; rows, hashes and sizes stay; another connection and another
company are untouched; a running run is skipped and is deleted after it
settles; a storage failure leaves the request pending with `retrying`,
and the next pass completes it; an approval-card update that fails once
is cleared on the retry; an unrecognised log backend keeps the pointer,
keeps the job pending and writes no completion entry; a zero time budget
yields after every step and still finishes; a repeat request returns the
pending one; agents and outsiders are refused; removal with
`deleteStoredContent=true` queues the deletion.
- `run-log-store.test.ts` and `object-run-log-store.test.ts`: `remove`
deletes local and mirrored logs and object journals, removes a migrated
run's original local file and mirror with its journal, refuses an object
log when object storage is not configured, and is idempotent.
- `ui/.../StoredContentDeletion.test.tsx`: confirmation text, counts,
skipped-run warning, and the request.
- Red: these tests fail on `master` because `remove`, the service and
the routes do not exist. Green on this branch.
- Ran: the tests above, `openapi-routes`,
`server-startup-feedback-export`, `tool-connection-removal`,
`packages/db` migration drift and compatibility tests, all
`ui/src/pages/apps` tests (950), `pnpm check:token-gates`, server and UI
`tsc --noEmit`. `pnpm verify` exits 2 (global CI input changed), so
required CI supplies broad coverage.

## Risks

- Deleting is permanent by design. The confirmation says so.
- A `local_file` log written on another replica's local disk stays on
that pod's disk until the pod is replaced; its S3 mirror is deleted. New
logs on S3-configured instances use object journals, which are deleted.
- On versioned buckets, earlier object versions remain until the
bucket's lifecycle policy removes them (the existing journal delete path
does not remove versions).
- Not in this PR (follow-ups): the "No" rows of the inventory (except
agent-written task content, which is out of scope by design), mainly the
same deletion when a task or a company is deleted, and later runs that
read an app view's shared context.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:55:35 -06:00
DottaandClaude Opus 5.5 55b692d532 feat(telemetry): send Cloud server telemetry directly to PostHog (#22006)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends first-party telemetry events through a collector.
The collector anonymizes the events and copies them into PostHog.
> - Paperclip Cloud is our own service. It needs real IDs (workspace,
user, agent, task) so that a person can go from a PostHog event to the
exact workspace and task.
> - The collector path hashes these IDs. It also cannot link a server
event to the browser session that caused it.
> - This pull request adds a server-owned direct sender for Cloud only.
Self-hosted installs keep the collector path with no change.
> - The benefit is real IDs and session links on Cloud, and no change to
self-hosted privacy.

## Linked Issues or Issue Description

**Subsystem affected**

Server telemetry (`packages/shared/src/telemetry`,
`server/src/telemetry.ts`).

**Problem or motivation**

All Cloud telemetry goes through the collector. The collector hashes the
install ID, agent IDs and task IDs. So, on Cloud, you cannot find the
workspace, user or task of an event. Server events also cannot show in a
browser session replay.

**Proposed solution**

On a Cloud-managed server that has `PAPERCLIP_POSTHOG_PROJECT_TOKEN`,
`PAPERCLIP_CLOUD_STACK_ID` and `PAPERCLIP_POSTHOG_ENVIRONMENT`, send
each telemetry batch straight to PostHog with `posthog-node`. Use the
same event shape that the telemetry backend loader makes today, and add
the real IDs. Keep every other install on the collector.

**Alternatives considered**

Send real IDs through the collector. We did not do this, because the
collector exists to anonymize self-hosted data.

**Roadmap alignment**

This is Cloud analytics infrastructure. It does not change a core
product feature.

## What Changed

- Add `server/src/services/cloud-posthog-config.ts`.
`resolveCloudPostHogConfig()` returns `null` unless the instance is
Cloud-managed and has a token and a stack ID.
`isPostHogDirectDeliveryEnabled()` also requires an environment and the
kill switch on (`PAPERCLIP_POSTHOG_DIRECT` is not `0`).
- Add a narrow `TelemetryTransport` seam to `TelemetryClient` in
`packages/shared`. With a transport, each flushed batch goes to the
transport and never to the collector. Without one, the client does what
it did before. A transport can capture per-event context at track time.
The context does not go into the collector envelope.
- Typed helpers hash `task_id` unless the transport sets
`hashesPrivateRefs: false`. Private `source_ref` values (for example a
private repository name) are customer content, so they stay hashed on
every path, Cloud included, with `source_ref_hashed: true`.
- Add `server/src/services/posthog-direct/`:
- `event-shape.ts` builds PostHog messages with the loader's event
names, metadata keys and key order, the deterministic event uuid
formula, and the unknown-enum fallback. It drops events that the backend
quarantines (unregistered names).
- `enum-fallbacks.generated.ts` holds the backend fallback rules. Enum
values come from the shared generated contract.
- `transport.ts` loads `posthog-node` with dynamic `import()` only on
the first Cloud send.
- `request-context.ts` is Cloud-only Express middleware. It reads only
`X-POSTHOG-SESSION-ID` and `X-POSTHOG-WINDOW-ID`. It ignores
`X-POSTHOG-DISTINCT-ID`. The acting user comes only from the
authenticated request actor. Context stops when the response finishes,
so later background work does not use that user.
- `server/src/telemetry.ts` selects exactly one sender per process. On
start, the Cloud sender sends `$groupidentify` (group `workspace`, key =
stack ID) and one `cloud.workspace_started` event with `install_hash`
(sha256 of the raw install ID). Both go straight to `posthog-node` and
are not in `PAPERCLIP_EVENTS`.
- `distinct_id` is the acting user ID, else `workspace:<stackId>`.
Events without an acting user keep `$process_person_profile: false`.
GeoIP stays off on all server events.
- Shutdown flushes the client and then shuts down `posthog-node`.
- Add `scripts/generate-posthog-direct-fixtures.mjs`. It runs the
telemetry backend's own ingest and loader on one envelope and writes the
golden fixture and the fallback table. The script header tells how to
regenerate them.
- Update `packages/shared/src/telemetry/README.md` with a "Cloud Direct
Delivery" section about the privacy boundary.
- Add `posthog-node@5.38.2` to `server/package.json`. This PR does not
change `pnpm-lock.yaml`, because the lockfile refresh workflow owns it.

Intended differences from the loader output on Cloud: `distinct_id`
(user or workspace, not the HMAC install ID), raw `agent_id`, raw
`task_id` (private `source_ref` stays hashed), and these added
properties: `stack_id`, `company_id` and `user_id` when known,
`environment`, `$groups`, `$session_id` and `$window_id`. The uuid
formula is the same. The uuid is different only when the dimensions are
different (raw `task_id`).

## Verification

- `pnpm verify` returned exit 2 (a global CI input changed: the new
script). I ran focused checks. Required CI supplies the broad coverage.
- Focused tests pass: `packages/shared` `src/telemetry` (6 files, 42
tests). Server: `posthog-direct-telemetry`,
`posthog-direct-event-shape`, `cloud-posthog-config`,
`shared-telemetry-events`, `heartbeat-status-preserving-telemetry`,
`issue-thread-interactions-telemetry`, `native-run-finalizer-telemetry`,
`agent-task-run-telemetry`, `error-handler` (9 files, 115 tests), and
`server-startup-feedback-export` (133 tests).
- `tsc --noEmit` passes for `packages/shared`, `server` and `cli`.
- New tests:
- Config matrix: not Cloud, no token, no stack ID, no environment, kill
switch, browser and replay flags.
- Self-hosted: `posthog-node` is never loaded, the collector gets the
batch, and the envelope bytes are the same as before (with `task_id`
hashed). Tracing headers are ignored.
- Cloud: a private `source_ref` stays hashed and the raw repository name
never reaches the sender.
- Cloud: one batch goes only to PostHog. Real IDs, `$groups`,
`environment` on every event. `cloud.workspace_started` reaches the
PostHog sender with `install_hash`. No other event has `install_hash`.
- Golden: our builder output equals the backend loader output, except
the intended differences above. Key order is the same.
- `distinct_id`: user, agent and anonymous requests. A spoofed
`X-POSTHOG-DISTINCT-ID` never changes `distinct_id` or `user_id`. A bad
session header is dropped. Work after the response ends does not keep
the user (red without the response-end guard, green with it).
- Kill switch and a missing environment both fall back to the collector.
- Local run with a real server and a local capture server (no real
PostHog project):
- Cloud mode (authenticated, local Postgres, `PAPERCLIP_POSTHOG_HOST`
set to the capture server): the capture server got `$groupidentify`,
`cloud.workspace_started`, `server.started` and `goal.created` on
`/batch/`. `goal.created` from a Cloud tenant user had `distinct_id` =
that user, `user_id`, `company_id`, `$session_id`, `$window_id`,
`$groups` and no `$process_person_profile`. The spoofed distinct ID
header was not used. Nothing went to `/ingest`. `SIGTERM` ran the
`telemetry_flush` step and sent the queued events.
- Self-hosted mode (same token variables but no Cloud signal): the
capture server got only `/ingest` with the normal envelope. Nothing went
to `/batch/`.
- Runtime dependency note: `posthog-node` loads with dynamic `import()`
only on Cloud with direct delivery on. The load happens after startup,
outside the boot path. The unit test mocks the package to count loads.
The local run used the real package.

## Risks

- Medium. This is a telemetry change, so a privacy review is requested.
Self-hosted output does not change. The direct path is off unless all
three Cloud variables are set, so a merge does nothing until fleet
config sets them.
- Cloud events with an acting user create PostHog person profiles keyed
by the Paperclip user ID. This is intended under the Cloud terms.
- The fallback table can drift if the backend adds a new normalization
rule. A test fails when a rule does not match the shared enum contract.
Enum values come from the shared contract, so new enum values do not
drift.
- `posthog-node` keeps network failures in its own queue and drops
events that PostHog rejects. The client does not resend these, so events
are not sent twice.
- `pnpm-lock.yaml` is not in this PR. The lockfile refresh workflow adds
`posthog-node` after merge.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, local server runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 08:54:32 -06:00
Nicky Leach bc0a236b95 feat(runner): accept a resume-compatible build for a relaunched released Runner (#21976) 2026-11-17 06:15:01 -08:00
DottaandClaude Opus 5.5 66ac5d7aa0 fix(google): keep Paperclip's Google client inside the 14 submitted scopes (#22001)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The built-in Google Workspace connector uses Paperclip's own Google
OAuth client through the Paperclip Cloud broker, or a customer-owned
client
> - Google is reviewing that client for exactly 14 scopes (Gmail
read/compose, Drive read/file, Docs, Sheets, Slides, Calendar)
> - The code still accepted the old 21-scope union for the unified
connection, and other Paperclip-managed Google methods (Gmail archive,
Chat, People, Workspace Search) could ask the same client for scopes
outside the submission
> - A consent request for an undeclared scope through the verified
client breaks the review
> - This pull request makes the 14 an explicit constant, enforces it on
every unified-connection request, callback and refresh, and stops new
consent for managed Google profiles outside it
> - The benefit is that Paperclip's Google client cannot request a scope
that Google has not reviewed, while saved connections keep working

## Linked Issues or Issue Description

Refs the plan in `doc/plans/2026-11-17-google-workspace-code-fixes.md`
(items 2, 3 and part of 10).

**What happened?**
The unified `workspace.all` profile was the 21-scope legacy union, and
`isGoogleWorkspaceScopeGrant` accepted any subset of it. The Gmail
`paperclip-archive` method asked for `gmail.modify`. The Chat, People
and Workspace Search cards had Paperclip-managed methods with Chat,
Contacts, Directory, profile and `calendar.readonly` scopes. None of
these are in the Google submission.

**Expected behavior**
Paperclip's own Google client requests, stores and refreshes only the 14
submitted scopes. New managed setup is not offered for anything else.

**Steps to reproduce**
1. Advertise `gmail.archive` (or `chat.read`) from the broker.
2. Open Apps. The managed archive/Chat method is offered, and starting
it signs a broker session for scopes outside the 14.
3. Refresh a unified grant: the signed binding lists all 21 legacy
scopes.

## What Changed

- `packages/shared/src/google-workspace-connectors.ts`: add
`GOOGLE_WORKSPACE_SUBMITTED_SCOPES` (the 14, written out by hand).
`workspace.all` now uses it. `isGoogleWorkspaceScopeGrant` checks
against it. Add `googleWorkspaceProfileWithinSubmittedScopes`.
- `server/src/services/paperclip-cloud-connector.ts`: add
`paperclipCloudConnectorConsentAllowed`. `startAuthorization` refuses a
Google profile outside the 14 before any broker call.
`appWithPaperclipCloudConnectorAvailability` does not offer those
methods for new setup, even when the broker advertises them.
- `authorization-start.ts`: a saved connection on such a profile cannot
restart consent (`paperclip_cloud_connector_scope_not_offered`, 422).
- Capping contract, the same on managed and customer-owned OAuth
(`paperclip-cloud-connector.ts` `openCredentials`, `cloud-callback.ts`,
`direct-callback.ts`, `grant-refresh.ts`): Paperclip caps Google's
reported scopes to the 14 (`googleWorkspaceSubmittedScopesIn`) and to
the saved request, then continues. A scope outside the 14 is ignored and
never enables an action. Only missing scope evidence or an empty result
marks the grant for reconnecting. Reconnecting with a new service choice
always works. What Paperclip *requests* stays fail-closed: an outgoing
unified authorization request with any scope outside the 14 is rejected.
- Docs: `doc/connections/GOOGLE-WORKSPACE.md`,
`doc/connections/GOOGLE-WORKSPACE-SERVICE-CONSENT.md`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed shared file.

### Investigation

**Scope flow, unified connection.** Both paths start in
`authorization-start.ts`. The requested set comes from the picker
(`googleWorkspaceSelectedScopes`), a saved draft, or the saved grant,
and must be inside the method's `scopesHint` (the 14 since the connector
was added in #15545). Managed: `cloudConnector.startAuthorization` signs
`scp` = the request; the broker validates it as a subset of its
registry; the claim is unsealed with the request as AAD;
`cloud-callback.ts` caps Google's grant to the request. Customer-owned:
`direct-callback.ts` checks Google's raw `scope`, then caps it. Refresh:
managed refresh signs `scp` = the profile definition (previously 21) and
unseals with that set as AAD; customer refresh checks the raw `scope`.

**What depends on the 21-scope set.** Only the managed refresh binding
(`scp` and envelope AAD) and tests. The broker (`paperclip-cloud`
origin/master, `src/connections/providers.ts`, `broker.ts`) accepts any
nonempty, duplicate-free subset of its `workspace.all` union for
session, claim and refresh, and seals with the signed `scp`. So
narrowing the app's `workspace.all` to 14 keeps the contract. I narrowed
the definition itself instead of adding a parallel check. A saved
unified grant can still report one of the 7 extra scopes: older app
versions signed the 21-scope `workspace.all` binding on refresh, and
Google can return scopes the same client was granted before. This PR
therefore caps those scopes instead of failing the grant.

**Managed Google methods.** See the table below. The broker's
origin/master registry has no `gmail.archive` profile, so
`paperclip-archive` could not reach the managed client today.

| App / method | Profile | Scopes | Before | After |
| --- | --- | --- | --- | --- |
| google-workspace / paperclip-workspace | workspace.all | subset of the
14 | 21 accepted | 14 only |
| gmail / paperclip-read, paperclip-draft | gmail.read, gmail.draft |
gmail.readonly, gmail.compose | offered | offered |
| drive, docs, sheets, slides, calendar / paperclip-read,
paperclip-write | per service | inside the 14 | offered | offered |
| gmail / paperclip-archive | gmail.archive | + gmail.modify | offered
if advertised | saved only |
| google-chat / paperclip-read, paperclip-write | chat.read, chat.write
| chat.* | offered if advertised | saved only |
| google-people / paperclip-read | people.read | directory, contacts,
userinfo.profile | offered if advertised | saved only |
| google-workspace-search / paperclip-read | workspace-search.read | +
calendar.readonly, chat.messages.readonly | offered if advertised |
saved only |

"Saved only": the method stays in the app definition so saved
connections resolve their method, refresh and run. New setup and new
consent are refused. Customer-owned methods are unchanged.

### Broker changes needed in paperclip-cloud (not in this PR)

1. Narrow the `workspace.all` request allowlist for `session` (and the
provider URL) to the 14. On refresh and claim, cap Google's reported
scopes to the allowlist (or ignore extras) instead of rejecting with
`provider_rejected`, and keep accepting a signed `scp` binding of the
old 21 from older app versions until they are retired.
2. Drop or disable `chat.read`, `chat.write`, `people.read`,
`workspace-search.read` for new sessions on the verified client. Keep
refresh for saved grants until the owner retires them.
3. Do not merge the `gmail-archive-oauth-profile` branch for the
Paperclip-managed client.

## Verification

- Red first, then green: `pnpm exec vitest run
packages/shared/src/google-workspace-connectors.test.ts
server/src/services/paperclip-cloud-connector.test.ts` (13 new or
changed tests failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "refuses new managed
setup|saved Paperclip-managed Gmail archive|fails closed on invalid
refreshed|pilot"` (8 failed before the change).
- Red first, then green: `pnpm exec vitest run
server/src/__tests__/tool-access-service.test.ts -t "caps a|keeps a .*
grant on its submitted|reconnects a"` (4 capping tests failed on the
earlier fail-closed revision; both reconnect tests pass).
- `pnpm exec vitest run server/src/__tests__/tool-access-service.test.ts
-t "combined Workspace|Workspace|Gmail|managed"`: 161 passed. Full
`tool-access-service` and `connection-management.integration` files: 599
passed on the earlier revision.
- `pnpm exec vitest run packages/shared/src/app-definitions.test.ts
server/src/services/gmail-tool-governance.test.ts
server/src/services/gmail-archive-tool-policy.test.ts`: pass.
- `pnpm --filter @paperclipai/shared typecheck`, `pnpm --filter
@paperclipai/server typecheck`: pass.
- `pnpm verify` returned exit 2 (a global CI input changed). Required CI
supplies the broad coverage.

## Risks

- Rollout order is app first, then broker. Refresh now signs the 14. If
the broker narrows `workspace.all` first, refresh from app versions that
still sign 21 (including self-hosted instances on the broker) fails
until they upgrade.
- A saved unified grant whose provider-reported scopes include a scope
outside the 14 keeps working on its in-set scopes; the extra scopes no
longer enable any action. A grant with only out-of-set scopes needs
reconnecting, and a reconnect with a new service choice succeeds.
- When the broker narrows `workspace.all` to the 14, it must cap or
ignore extra scopes on refresh and claim (it re-validates them today),
not reject saved grants.
- Saved Chat, People, Workspace Search and Gmail archive managed
connections keep working, but their owners cannot reconnect them through
Paperclip's Google client. They see a 422 with a plain message. The
owner must decide whether to retire them.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:41:31 -06:00
DottaandClaude Opus 5.5 d258f478fc test(ci): stop the live-run census hanging server shards, and name any stalled test file (#21941)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - PR CI splits the server tests across 12 shards; `ci / verify` fails
when any shard fails
> - Since 2026-11-16 the server shards were cancelled at their 20-minute
job limit with no output (shard 2/12 five times, plus shards 1, 3 and
9), which blocked unrelated pull requests until reruns
> - Turbo prints a task's output only when it ends, so a cancelled shard
never names the stuck file
> - Recomputing each cancelled shard's file list at its own merge commit
put `issue-recovery-live-run-agent-census.test.ts` in 12 of 14 of them;
it is also the slowest file in passing shards (235-300 s)
> - That census started 9-14 real agent runs per file while `setTimeout`
was faked, waited on them with no bound, and had 30-minute test
timeouts, so a stuck wait could never fail inside the job
> - This pull request stops the census from starting real runs, bounds
every wait, and adds a CI reporter that fails a run naming any file that
stops making progress
> - The benefit is that this file cannot hang a shard, and the next
stuck file fails in minutes with its name and step in the log

## Linked Issues or Issue Description

Refs #21867 (the same fix for the sibling agent-row census), #21831 (the
behavior this census guards)

**What happened?**
`ci / General tests (server (2/12))` was cancelled at 20m16s in actions
runs 43865258195, 43926799428, 43932238377, 43934278368, 43944290677 and
43935722497 (both attempts), and other server shards hit the same limit
(for example 43901003914 shard 1, 43945768035 shard 3, 43845343831 shard
9). The hung jobs show CPU near 0 for the last 9 minutes and printed
nothing after `Running //#test:run:general`.

Shard membership changes with every merge commit. To compare the shards,
I recomputed each job's file list at its exact merge commit with
`GITHUB_WORKFLOW=PR` (the PR workflow removes the native-runner suites
from these shards, so a local dry run without it gives a different
list).
`server/src/__tests__/issue-recovery-live-run-agent-census.test.ts` was
in 12 of the 14 hung shards.

Why that file:
- During its 10-minute fake-timer warm-up, the "live" task was
`in_progress` with no run. That is real assigned work, so recovery woke
its agent and a real `process`-adapter run started while `setTimeout`
was faked. The run lost its lease ("Legacy controller lease lost"),
recovery retried, and a local run of the file showed 13 real run
executions.
- No wait in the file had a bound, and each test had a 30-minute
timeout, so a delivery pass waiting on a faked timer or a run holding a
lock never failed inside the 20-minute job.
- Under local CPU contention, 3 of 4 copies of the old file failed the
census assertion (39 settled checks, caused by the stray runs), and each
copy took about 270 s with 9-14 real runs.

Fixing the runs exposed a second problem in the fixture: the seeded
tasks were stamped "now". The first 5-minute backstop after the live
run's first hint re-checks rows changed within 5 minutes before the last
full pass, so it re-checked all 860 seeded tasks. The old file passed
only because the stray runs happened to trigger that backstop during the
warm-up.

**Expected behavior**
The census never starts a real run, every wait has a bound that names
what it waited for, and a CI test file that stops making progress fails
the job quickly with its name.

## What Changed

- `issue-recovery-live-run-agent-census.test.ts`
- The live task is a settled review until its run starts; `startLiveRun`
now writes the checkout (`in_progress` + `executionRunId`) the way
execution start does.
- Population agents and the held agent take no on-demand wakes
(`wakeOnDemand: false`), so a recovery check cannot dispatch a run. The
census counts queue checks, so this does not change what it measures.
- The seeded population is backdated 2 hours ("settled long before the
window", as the file's comment already said).
- Each step (seed, worker start, run start, run-event tick, fake-timer
advance, pass settlement, direct writes) has a real 20-second bound
captured before the timers are faked, with a message naming the step.
Test timeout is 4 minutes instead of 30.
- New `scripts/ci/stalled-file-reporter.mjs`, enabled in GitHub Actions
from the root `vitest.config.ts`: when a running file starts no test or
hook and reports no result for 8 minutes
(`PAPERCLIP_VITEST_FILE_STALL_MS` overrides), it prints a `::error
file=...` annotation naming the file and the step it was in, cancels the
run and exits 1. The slowest healthy server file runs about 5 minutes in
total, so 8 minutes without any progress means a stuck file. Tests:
`scripts/ci/stalled-file-reporter.test.mjs` (added to
`test:release-registry`).

## Verification

- Census, red: the old file runs 13 real agent runs locally (`heartbeat
execution setup failed`, `LegacyControllerLeaseLostError`); with 4
copies under CPU contention, 3 failed with 39 settled checks at about
270 s each.
- Census, green: 4/4 tests pass, 0 real runs, about 80 s locally.
- Census still guards #21831: with the `ag.status <> 'running'` line
removed from `changed-tasks.ts`, the first test fails with 534 settled
checks and 540 backstop checks, the numbers #21831 recorded.
- Reporter, red/green: the real-Vitest case (a file awaiting a promise
that never settles, with a 10-minute test timeout) is still running when
the 20-second test bound ends without the reporter; with the reporter it
exits 1 in about 8 s and prints the annotation naming the file and the
test.
- Reporter on a real server file: `PAPERCLIP_VITEST_FILE_STALL_MS=3000`
on `issue-recovery-recorded-decision-idle.test.ts` prints the annotation
naming the file and step and exits 1.
- `node --test scripts/ci/stalled-file-reporter.test.mjs`: 5/5 pass.

## Risks

- Test and CI-tooling change only. No product code changed.
- The census's agents no longer take on-demand wakes. The census
measures recovery queue checks, not dispatch, and still fails without
the #21831 fix.
- The reporter stops the whole CI run when one file stalls for 8
minutes. A legitimate single test or hook that runs more than 8 minutes
with no other progress in its file would now fail; none exists today
(the slowest whole file is about 5 minutes). The limit is configurable.
- Other fake-timer recovery files still have long test timeouts and
unbounded pass waits (for example `issue-recovery-activity-hints`, 30
minutes). The reporter now caps any of them at 8 minutes without
progress and names the file; bounding them is follow-up work.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-16 19:50:54 -06:00
f33c4cf6f7 feat(links): cache rich link status in the database and load it on idle (#21746)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task text contains links to pull requests and other work.
> - A link has useful status that should be visible when a task opens.
> - The current cache lives in the server process and the browser.
> - A new browser or server restart has no stored status to show.
> - This pull request stores link status in the database and reads it in
batches.
> - Idle page work updates stale status without a hover or a delay to
task data.

## Linked Issues or Issue Description

**Problem or motivation**

PR status is unknown in a new browser until the user points to each
link. The process cache cannot supply saved status after a server
restart. Related work: #20593 and #20654. A search found no other PR for
database link caching.

**Proposed solution**

Store status by company, viewer and canonical resource URL. Read saved
status in one indexed batch query. After document load and core request
settlement, use browser idle time to update missing or stale status. Use
the same query for hover and idle work.

**Alternatives considered**

Adding provider calls to task reads would delay task loading.
Company-wide reuse of personal GitHub results would cross viewer access
boundaries. A periodic server worker would create work when no page
needs the result.

**Roadmap alignment**

This extends the shipped rich-link feature. It does not duplicate a
roadmap project or add a new provider integration.

## What Changed

- Add an additive, repeat-safe `link_previews` migration. Store channel,
resource kind, free-form status, JSON metadata and freshness. The table
has no task relationship or GitHub status enum.
- Bound each company/viewer cache to 1,000 rows across concurrent server
instances. Remove rows expired for over 30 days during writes. Limit
provider lookups to 600 requests per minute per scope/process. Keep
cached batch reads available during backoff.
- Add a database store owned by the link service. Reuse canonical
results across tasks. Keep company and viewer isolation. Save hover and
idle results. Preserve last known status during an outage and cache
unavailable results.
- Add a board-only batch GET endpoint, including viewer-role access. It
reads at most 50 URLs with one indexed select and makes no provider
calls. Both endpoints use private, no-store HTTP responses.
- Add a page coordinator for mounted rich links. Reuse browser data,
share pending reads, and defer provider work until document load, core
request settlement and browser idle time. Limit automatic work to 200
unique links and three requests in flight. Remove queued work when links
leave the page. Keep encoded batch URLs below 6 KB.
- Keep exact navigation URLs, status styling, anchored cards and
keyboard access. Add a page-load Storybook case and browser regression.

The link service owns cache policy and provider calls. The browser
coordinator owns page timing. The shared contract adds only the batch
limit and response shape. The database adds storage because process and
browser caches cannot serve a new browser session. Plugin resolver
registration is outside this change; storage accepts future provider
channels and status values.

## Verification

- **All 93 affected tests pass:** 40 link service, real PostgreSQL and
HTTP cases, plus 53 OpenAPI cases. Two SQL regressions and one HTTP case
failed before the storage repair. Concurrent writers now retain at most
1,000 rows per company/viewer; cleanup removes long-expired rows and
keeps recent stale status. Request backoff preserves cached reads and
isolates companies/viewers. The earlier eight concurrent hydration
regressions also pass. Focused store/service/unit types pass. Commands:
`pnpm exec vitest run server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1` and
`pnpm exec vitest run server/src/__tests__/openapi-routes.test.ts
--maxWorkers=1`.
- **All 50 affected server tests passed before the later server-only
repairs** on the master-integrated source: 20 recovery/deadline cases,
one authenticated review-counts HTTP case, 10 link-service cases, 13
real PostgreSQL cache cases and six link-route cases. Command: `pnpm
exec vitest run server/src/__tests__/issue-recovery-deadlines.test.ts
server/src/__tests__/tool-connection-review-counts-http.test.ts
server/src/__tests__/link-previews.test.ts
server/src/__tests__/link-preview-store.integration.test.ts
server/src/__tests__/link-preview-routes.test.ts --maxWorkers=1`.
- Reproduced the CI watchdog assertion locally twice before the test
repair. Ordinary assignment recovery can win `wakeReason` while the
watchdog wake coalesces into the same successful run with its full
mandate. The test now verifies the persisted watchdog identity, watched
issue and stopped fingerprint, plus exactly one review run. All 21
recovery/HTTP cases passed before integrating master; all 50 affected
cases pass after integration. No runtime behavior was changed for this
repair.
- **Fresh static Storybook build and all 22 browser cases pass**, with
two workers and retries disabled. They cover states, anchored hover,
keyboard access, exact navigation, duplicate requests, reload
persistence, expiry and saved-status hydration followed by one idle
refresh. The first browser attempt could not launch because this session
lacked Chromium; installing the expected browser resolved setup. That
attempt had no behavioral verdict.
- The cache repair remains covered by four PostgreSQL red/green
regressions for expired memory and newer durable status during provider
failure/timeout, plus check-time precedence and failed-read recovery.
Earlier UI/database/focused server type and migration
safety/compatibility/replay checks remain unchanged-input evidence.
- Master `77ddf260bf9a3de3dcb8456a60f4f809d67401fd` is integrated
cleanly. Link migration 0489 and its snapshot chain are unchanged. SDK
source validation checks 545 inputs with no stale source identities or
changed runtime/lock pins. Whitespace validation passes.
- `pnpm verify --explain` returns exit 2 and requires broad CI. No full
local suite or deployed acceptance is claimed. Earlier local full-server
type checks exceeded memory; current-head CI supplies broad
type/build/test coverage.
- [Previous CI
43900336895](https://github.com/paperclipai/paperclip/actions/runs/43900336895)
passed full types/build but failed on embedded PostgreSQL startup
(`ECONNREFUSED`), the watchdog assertion above, and unrelated plan-task
browser cleanup (activity-log foreign key on company deletion). One
failed-jobs retry was requested. Its browser shard 7 now passes without
code changes; the exact deletion-race cause remains unproven. The
watchdog assertion is repaired in this revision. The previous failures
remain recorded, not counted as an overall pass.
- [Prior CI
43905482848](https://github.com/paperclipai/paperclip/actions/runs/43905482848)
passed full types, build and all eight browser shards. Several test jobs
were cancelled; the run has no overall pass. Its concurrent update
finding is repaired. [Next CI
43909197530](https://github.com/paperclipai/paperclip/actions/runs/43909197530)
also passed types, build and all browser shards, but several server jobs
were cancelled. Greptile and Paperclip Review gave that revision 5/5.
Its security scan found unbounded retained cache rows. The quota,
retention and request-limit repair above addresses that finding. All ten
review threads are resolved.
- [Current CI
43912947278](https://github.com/paperclipai/paperclip/actions/runs/43912947278),
fresh code reviews and a fresh security scan are pending on
`335d89ef1668dcdeb66bb2059512a7c2da29e273`. Auto-merge is disabled until
the current revision clears its gates.

Published head: `335d89ef1668dcdeb66bb2059512a7c2da29e273`. Local head:
`b37b87c704a2ba8a65233c689a6a743f4011ce14`. Both have tree
`643ebd19b9729b1e6a6bb695442269c3db32be3a`; GitHub tree creation
verified equality.

## Risks

- Each company/viewer retains at most 1,000 rows. Writes remove excess
rows and entries whose freshness expired over 30 days ago. Inactive
scopes remain bounded and are cleaned on the next write. Company
deletion removes all rows. No periodic worker runs. Scope writes use a
transaction lock; the existing deadline and concurrency budget cover
that work.
- Last known status can be stale until idle or hover work completes.
Provider failures keep that status through the negative-cache window.
- Automatic enrichment adds bounded provider traffic. It runs only for
mounted supported links, once per mount, with the existing timeout and
server concurrency limit.
- Cache storage is viewer-scoped because provider results can be
private. New viewers load their own results. This prevents reuse across
users with different repository access.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
2026-11-16 14:33:39 -06:00
github-actions[bot]andlockfile-bot b816c95f96 chore(lockfile): refresh pnpm lockfiles (#21936)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-16 20:15:43 +00:00
DottaandClaude Opus 5.5 171e70c11d refactor: drop span and run-log event catalogs from the observability docs (#21779)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - `doc/observability.md` and `doc/run-log-events.md` described every
OpenTelemetry span and every run-log event
> - `rules/data-paths.mdc` told authors to update
`doc/run-log-events.md` for each event, so both files only grew: 2,313
and 1,263 lines
> - Each span and event is already defined next to the code that writes
it, and its name is a literal string that a search finds in one step
> - This pull request removes the per-span and per-event catalogs and
keeps only what code does not say
> - The benefit is two short docs that do not drift from the code, and
no rule that makes them grow again

## Linked Issues or Issue Description

Refs #21670, #21700, and #21743 (the same cleanup for other docs).

**Issue type**

Outdated or confusing documentation.

**Where is the issue?**

`doc/observability.md`, `doc/run-log-events.md`, and
`rules/data-paths.mdc`

**What's wrong?**

The two docs repeated span names, span attributes, and event payloads
that the code defines. Operator setup started at line 138 of
`doc/observability.md`. The rule required a doc edit for every new
run-log event.

**Suggested fix**

Keep operator setup and the Sentry privacy contract. Point to the code
for span and event definitions. Remove the rule line that requires a doc
entry per event.

## What Changed

- `doc/observability.md` is now 619 lines. It keeps: how to enable
tracing, `service.version` resolution, scope, how to enable Sentry, DSN
delivery, environment attribution, privacy settings, server request
data, the core of "Run failure context", browser data, fail-open
behavior, the default capture set, and operator responsibilities. Kept
text is unchanged except the intro and two sentences noted below.
- Deleted from `doc/observability.md`: budget, Slack receipt, task
detail, and browser tracing notes; controller preparation phases; native
Runner spans; sandbox startup spans; sandbox duplex transport; shared
skill preparation; invocation attribution; and about 500 lines of
per-failure detail in "Run failure context".
- Replaced that per-failure detail with one paragraph. It names the
failures that `unexpected` mode keeps out of Sentry, and says the
remaining diagnostic contexts are defined in `server/src/sentry.ts`.
"Default capture set" now says the same.
- `doc/run-log-events.md` is now 23 lines. It says what the run log is,
where events are defined and appended, and keeps the payload storage
note.
- `rules/data-paths.mdc`: the run-log line now says to define an event
where it is written, not in a doc. The span-allowlist line now says the
allowlist is in the code that opens the span. The review levels and
globs do not change.
- Repointed links in two feature maps, two agent-access map files,
`tests/runner-e2e/SUITES.md`, `doc/runner-api-tools.md`, and the
Telemetry README. Updated the `honeycomb-tracing` skill and one code
comment in `packages/adapter-utils/src/http2-bridge-server.ts`.

## Verification

- A script built the new `doc/observability.md` from line ranges of the
old file and asserts the first line of each range.
- A link check on all changed Markdown files found 0 missing files and 0
missing anchors. The remaining inbound anchors (`#enabling-tracing`,
`#sentry-error-monitoring`) still exist.
- No test reads either doc.
- `node scripts/check-maps.mjs` passes.
- `node --test .github/scripts/tests/*.test.mjs`: 808 pass, 0 fail.
- `pnpm verify` was not run. This worktree has no installed
dependencies. The only code change is a comment.

## Risks

- The privacy contract no longer lists each run-failure diagnostic
context field. `server/src/sentry.ts` is now the list. A reviewer of a
Sentry change must read that file.
- No browsable list of span names or run-log event names remains. A
search for the quoted name finds the producer.
- `.greptile/files.json` still scopes `rules/data-paths.mdc` to both
docs. Both files still exist, so it does not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code desktop, with tool
use. Earlier analysis in this session used Claude Fable 5.2.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [ ] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 10:06:05 -06:00
DottaandClaude Opus 5.5 3706f520fd test(scaling): keep ramping through recovered runs, read spend from cost records, and wait for termination before deleting agents (#21776)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The scaling effort uses a real-agent ramp harness
(`scripts/scale-agents-ramp.ts`, #21695 and #21757) to load a hosted
test instance through the product API
> - Its first run with agents that really ran (10 agents, 105 tasks, 106
runs) stopped on one failed run, recorded $0 model spend, and left
cleanup errors
> - Read-only inspection of the instance showed three harness faults.
The failed run had a same-run bootstrap retry scheduled 33 s later, but
the harness stopped and cleaned up first. Run rows carry no cost, but
cost records do (63.7 cents for the run's agents). Terminated agents
leave the company agent list, so the harness treated them as done and
sent deletes that Paperclip refused with 409
> - These faults make the ramp stop too early, make its spend cap inert,
and make cleanup report the wrong cause
> - This pull request keeps the ramp going through runs that Paperclip
recovers by itself, reads spend from cost records, records board latency
per route, and makes cleanup wait for termination and name what it left
behind
> - The benefit is that the next ramp grades the product, not the
harness, and its spend cap works

## Linked Issues or Issue Description

Refs #21695
Refs #21757

**What happened?**
The ramp stopped at step 1 with `failed_runs_zero failed (observed 1)`.
The run failed with `runner_ingress_bind_conflict` before its provider
session opened. Its run events show `same-run resume persisted` with
`recoverable: true` and `nextAttemptAt` 33 s later. The harness stopped
within 9 s and paused the agents, so the retry was blocked
(`native_retry_admission_ineligible`). The result showed `Recorded model
spend: $0`. The harness read `costUsd` from run rows, which is always
null; `GET /api/companies/<id>/costs/by-agent` holds 63.71 cents over
104 cost events for the 10 agents. Cleanup reported two `DELETE
/api/agents/:id returned 409: Complete termination before deleting the
agent`. Those agents were still `lifecycleState: terminating`; the
harness waited on the company agent list, which hides terminated agents,
so it treated them as done. A later `--cleanup-only` changed nothing
because it also found agents only through that list. The board API p90
failed (765 ms), but the result had no per-route timings to say which
request was slow.

**Expected behavior**
A failed run whose task finishes by itself is counted, not a stop. A
failed run whose task does not finish within a grace stops the ramp. The
share of failed runs is graded at the step end. Spend comes from
Paperclip's cost records. Cleanup deletes an agent only after
termination completes, and names the agents and sandbox leases still
pending.

**Steps to reproduce**
1. Run `node --test scripts/scale-ramp/scale-ramp.test.ts` with the new
fake and tests and the `origin/master` `ramp.ts`, `cleanup.ts`,
`criteria.ts` and `report.ts`.
2. 16 tests fail, among them every new test (recovered run, unrecovered
run, spend from cost records, spend cap, board per route, termination
wait, still-terminating agents and `--cleanup-only`, leases released
during the wait).
3. With this change, all 60 pass.

**Paperclip version or commit**
Instance on `2026.1106.0+2564.git.3244780bd`; harness from #21757.

## What Changed

- `criteria.ts`: replace `failed_runs_zero` with
`failed_runs_unrecovered_zero` (immediate) and `failed_run_rate` (at the
step end). Step metrics add `runsEnded`, `runsFailedRecovered`,
`runsFailedUnrecovered`, `runsFailedPending` and `board.byRoute`.
- `ramp.ts`: a run counts as failed once it was seen `failed` or
`timed_out`. It is recovered when its task reaches `done`. It is
unrecovered when the task rests not done, needs Retry, or has no live
run for `--recovery-grace-minutes` after the failure. A task whose
interruption has an automatic retry scheduled (`state: retry_scheduled`,
`nextRetryAt` or `retryWait`) does not count as needing Retry. Spend
comes from `costs/by-agent` for the run's agents on each poll, and each
step reports the spend in its window. Board probes keep a distribution
per route.
- `config.ts`: add `--max-failed-run-percent` (default 2) and
`--recovery-grace-minutes` (default 15).
- `cleanup.ts`: read agents by id as well as from the list, so
terminated agents in the state file are found. Wait for `lifecycleState`
`terminated` or `rejected` by id before a delete. Report
`agentsStillTerminating`, `activeSandboxLeaseIds` and
`sandboxLeasesReleasedDuringWait`, and name each lease still active with
its run and acquire time.
- `report.ts`: the summary shows recovered, unrecovered and pending
failed runs, board latency per route, and each cleanup error.
- `fake-paperclip.ts`: run rows carry no cost; `costs/by-agent` returns
cents per agent. Optional failure of the n-th run, a scheduled recovery
run, a termination delay (terminated agents leave the list and a delete
answers 409 until termination completes), and leases that stay active
after their run.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: the runbook
describes the failed-run criteria, the spend source and the cleanup
behavior.

## Verification

- `node --test scripts/scale-ramp/scale-ramp.test.ts`: 60 of 60 pass.
- Red check: with the new fake and tests but the `origin/master`
`ramp.ts`, `cleanup.ts`, `criteria.ts` and `report.ts`, 16 tests fail,
including all 8 new ones.
- `tsc -p scripts/scale-ramp/tsconfig.json`: no errors.
- Evidence from read-only GETs on the test instance: run `f1892d43`
events (bind conflict, scheduled same-run retry, blocked after the
pause), run rows (`costUsd` null on all 106), `costs/by-agent` (63.71
cents for the 10 agents), agent rows by id (`terminating` while absent
from the list), lease rows.
- Not run: a new real ramp (needs the owner's go-ahead).

## Risks

- Low risk. The change touches only the scale-ramp harness, its fake,
its tests and one runbook section. No product code changes.
- A failed run no longer stops the ramp at once. An unrecovered one
stops it after the grace (15 min by default), so a broken step can run
up to that long. The step-end failed-run rate still fails a step with
more than 2% failed runs.
- Spend now needs cost read access for the board token. If
`costs/by-agent` cannot be read, the result reports spend as not
measured and the spend cap does not trigger.
- Cleanup can now take up to 5 more minutes while it waits for
termination by id.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, read-only HTTP GETs to the test instance).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:39:28 -06:00
DottaandPaperclip 5b750fdf25 feat(ui): add foldable sidebar subtasks with bounded refreshes (#21649)
## Thinking Path

> - Paperclip helps people manage AI agents for work.
> - The sidebar helps people return to recent tasks.
> - Recent parents need foldable child tasks.
> - Busy companies must not refresh every child group for unrelated
work.
> - Exact unread counts must preserve personal reads and task access.
> - This pull request adds child groups and bounds their refresh work.
> - People can scan subtasks without repeated background requests.

## Linked Issues or Issue Description

**Subsystem affected**

Task navigation in the standard and Classic sidebars.

**Problem or motivation**

Recent parent tasks did not show their subtasks. The first
implementation refreshed every previously viewed group. One unrelated
update caused 52 child-page requests and one summary request after
viewing 50 groups. Exact child counts also repeated privacy and unread
checks per child.

**Proposed solution**

Show direct subtasks with independent folding and ten-row pages. Refresh
affected groups and summary batches. Stop offscreen reads. Bound
child-request concurrency. Aggregate viewer facts once and preserve the
existing inbox unread rule.

**Roadmap alignment**

This extends existing task navigation. The existing feature PR is reused
for its performance repair. No new PR duplicates it.

## What Changed

- Add shared parent groups to both sidebar layouts. Preserve Classic row
actions and active-child visibility.
- Keep unread state personal to each task. Folding and paging do not
mark tasks read. Counts include readable children beyond ten.
- Target live refreshes with authorized current and previous parent IDs.
Reuse parent metadata from the narrow run-binding read. Resolve fallback
lookups once per delivery batch. Keep the previous summary fallback for
older servers.
- Limit child requests to three at a time. Cancel active and queued
reads when groups leave the viewport.
- Discard extra cached pages on Show less and collapse.
- Compute exact navigation counts in one SQL statement. Materialize
readable candidate IDs once. Use indexed participation sources and
conservative unread candidates before the shared inbox rule.
- Skip recursive privacy walks for standalone open tasks without a
project. Keep the recursive rule for all restricted or inherited shapes
and all delegated principals.
- Add request-count, cancellation, reparenting, read-receipt,
inbox-parity and privacy regressions. Include a disposable PostgreSQL
benchmark.
- Update product and API documentation. Retain the production Storybook
states.

## Verification

- Published head: `4148861437c1b6b635479ad92509dbc71155a232`. Its tree
matches local `04be9d66d481d9bcd1c4d6158acf063a1e65ac7e`:
`a23533e93d3d155918f0254d296a7fd2f0c7c12f`.
- Rebased on master `ca23330c69b1f2b25bdadfcbda68317db97deaef`. Remote
publication preserves prior feature commits.
- The request regression failed before the fix and passes after it. An
unrelated event now sends zero child requests. Current servers also
avoid the unrelated summary request. Old-server events keep one summary
fallback.
- A relevant update after Show less reads one page. Reparenting
refreshes two affected groups. Offscreen reads stay at zero and resume
once on return. Active and queued cancellation tests pass.
- The same synthetic authenticated HTTP workload with 50,000 read
children improved from 767 ms to 92.84 ms median. Five warm samples per
case. The all-unread case improved from 463.85 to 351.65 ms. Untouched
children improved from 208.01 to 131.09 ms. The old-receipt case
improved from 799.70 to 245.80 ms. These local measurements are not a
production capacity guarantee.
- Focused PostgreSQL routes, inbox-parity, privacy and WebSocket tests
passed: 38 tests.
- All 325 affected UI tests passed. All 38 focused PostgreSQL, privacy
and WebSocket tests passed. UI typecheck and low-memory server
declaration checks passed.
- All 41 Chromium Storybook checks passed. They cover 20 states in both
themes and keyboard, folding, paging and unread behavior. They use
production components with HTTP fixtures.
- Both new review findings failed as regressions before their fixes and
pass afterward. Explicit child reads include waiting external
conversations. Deleting a child publishes a separate parent-authorized
update without deleted-child content. Both review threads are resolved.
- `pnpm verify --explain` returned exit 2 because product documentation
requires broader CI. [The prior CI
run](https://github.com/paperclipai/paperclip/actions/runs/43711489306)
found type errors in the new test fixtures. The same errors failed
locally. The fixture fix passes `node
server/scripts/typecheck-layers.mjs --layer test-tree-0` and all 21
issue summary route tests. Production code is unchanged. That CI run
then found a live-delivery read regression. The existing test failed
locally with eight statements against its six-statement budget. A new
PostgreSQL regression also found three parent lookups for two same-batch
calls. Run delivery now reuses parent metadata from its narrow
run-binding query. Fallback lookups resolve to one promise. Both
regressions pass without increasing the budgets. All 74 targeted route,
WebSocket and privacy tests pass. The affected below, above and
test-tree-0 typecheck layers pass. [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43715541480/attempts/2)
passes on the published head. All 62 check runs are terminal and clean:
57 succeeded, two were neutral and three were skipped. Typecheck, build,
policy, all general and serialized server suites, E2E and the [final
verify
gate](https://github.com/paperclipai/paperclip/actions/runs/43715541480/job/132829783231)
pass. The separate security status passes. Attempt 1's server shard 7/12
reached its 20-minute job limit without a test verdict. One retry of
that job passed without a source change. [Greptile completed
successfully on this
head](https://github.com/paperclipai/paperclip/runs/132816163906) with a
[5/5
score](https://github.com/paperclipai/paperclip/pull/21649#issuecomment-6883916606).
All five review threads are resolved. The final CI gate is complete.
- A full local Storybook build exceeded available memory while server
typechecking ran. Browser verification uses the current-source
development server. Rebuilding stale Runner and SDK declarations
resolved the initial server check errors. The first browser attempt
lacked Chromium; all checks passed after installation.
- Reproduce the database measurements with `pnpm --filter
@paperclipai/server exec tsx scripts/benchmark-sidebar-subtasks.ts`.
- Extra local shard diagnosis ran 164 suites. Under this shell's `0077`
creation mask, 157 suites passed and three had fixture permission or
timing failures. Rerunning those three with `0022` resolved the
permission failures. The remaining ten cases passed with one worker: two
workspace cases and eight archive cases. The eight archive cases also
passed on the pre-performance-fix revision, whose archive source is
identical. The broad local shard is not claimed as one passing
execution. The same-head CI retry supplies complete final CI coverage.
- Nothing is merged or deployed.

## Risks

- Exact counts still grow with the readable hierarchy and comment
history. The change reduces repeated work; it does not cap or
approximate counts.
- A relevant summary batch can contain several recent parents. Older
servers retain a broader summary fallback until upgraded.
- Live delivery adds bounded, permission-checked parent lookups. Short
delivery-window reuse limits repeated reads.
- The authorization fast path applies only to open tasks with no privacy
parent and no project. Restricted cases retain the shared predicate.
- No migration is required.

## Model Used

OpenAI Codex, based on GPT-6. Used code execution, native GitHub tools,
PostgreSQL and Chromium verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either linked a public issue or described the issue in this
PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal issue
id
- [x] I have run focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open findings
- [x] I will address all review comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-15 08:51:04 -06:00
Nicky LeachandClaude Opus 5.5 a78be8b601 feat(native): send launch preparation and run digests to plugin task prepare (#21717)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On each warm turn Core sends the plugin a `prepare` operation with
the full launch requirements for the new run
> - The requirements' `requirementsDigest` covers the run identity, and
the run identity changes on every turn
> - So a plugin cannot tell that only the run moved, and it must redo
its whole preparation every warm turn
> - This pull request gives the plugin two digests: one that stays the
same while its preparation stays valid, and one that names the run
> - The benefit is that a plugin can transfer only the run identity when
nothing it reconciles changed

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The `prepare` operation that Core sends to an environment plugin for a
plugin Runner task, on cold starts and on warm attaches.

**Subsystem affected**
Native runtime, plugin Runner tasks, the Runner launch contract, and the
plugin SDK environment task contract.

**Current behavior**
A warm attach sends the full requirements. The only digest available,
`requirementsDigest`, includes `runId`, `turnId` and `itemId`, so it
changes on every turn even when the launch is otherwise the same.

**Proposed behavior**
Core computes two digests for every `prepare`:
- `preparationDigest`: the launch requirements an environment reconciles
when it prepares a Runner:
  - the runner, lease and session identity;
  - provider, model and settings;
  - the prompt and instruction text;
- the runtime context's instruction and skill bundles, working copy and
connection instructions.

It leaves out the `runId`, `turnId` and `itemId`, and the runtime
context's MCP assignment and binding (with `aggregateDigest`, which
hashes the assignment). MCP reaches the Runner through Core's tool
authority on every run attachment, never through files the environment
publishes. Core already continues a warm session across an MCP-only
change (`nativeRuntimeContextsMatch` with tool refresh). The binding is
also named per run (`native-mcp:<runId>`). So the digest stays the same
across warm turns of one Runner, lease and session, including when the
agent's MCP access changes.
- `runDigest`: `preparationDigest` plus everything it leaves out:
`runId`, `turnId`, `itemId`, and the MCP assignment and binding. It is
unique to one run's launch.

A plugin that advertises `launchDigestVersion: 1` in its task
description gets both fields on `prepare`.

**Reason and benefit**
A plugin can compare `preparationDigest` with its saved preparation and
transfer only the run identity when they match, instead of a full
preparation.

**Breaking changes**
None. Plugins without `launchDigestVersion` get no new fields.

## What Changed

- `packages/paperclip-runner/src/live/runner-provider-launch.ts`: add
`runnerLaunchPreparationDigest` and `runnerLaunchRunDigest`. The
preparation digest drops the MCP assignment and binding; the run digest
takes the full requirements and includes them. Both use the same
canonical JSON as `runnerLaunchRequirementsDigest`, with their own
schema tags so the three digests never collide. Export them from the
package and the server's vendored runner shim.
- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`preparationDigest` and `runDigest` to the `prepare` operation, and
optional `launchDigestVersion: 1` to the task description. The
capability gates both digests.
- `server/src/services/native-runtime/plugin-runner-task.ts` and
`native-session-executor.ts`: send both digests on `prepare` when the
description advertises `launchDigestVersion: 1`.
- `server/src/services/environment-task-runtime.ts`: recompute both
digests from the requirements, and refuse a `prepare` whose digests are
missing in part or do not match.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refresh the SDK
public source pin.
- Tests:
- the preparation digest is stable across runs and changes when the
runner, lease, session, instructions, model, approval policy or settings
change;
- with a runtime context, an MCP-only change (the run's binding, no
binding, a changed access set) keeps the preparation digest and changes
the run digest;
- a change to the instruction bundle, instruction entry, a skill bundle,
the skill set, the working copy or the connection instructions changes
the preparation digest;
  - the run digest changes per run;
  - Core sends the digests only when advertised;
- the runtime refuses forged or partial digests before it calls the
plugin.

## Verification

- `npx vitest run src/live/runner-launch-preparation.test.ts` in
`packages/paperclip-runner`: 26 passed.
- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server`: 160 passed.
- `npx vitest run source-sdk-public-inputs.test.mjs
source-sdk-evidence.test.ts source-sdk-public-type.test.ts` in
`tests/runner-e2e`: 59 passed.
- `pnpm typecheck` in `server` and in `packages/paperclip-runner`:
clean.
- Not run locally: the full suite. CI covers it.

## Risks

Low risk. The digests are new optional fields behind a plugin capability
version. Core computes them and checks them again before it calls the
plugin, so a plugin can rely on them. They prove only that the launch
requirements are unchanged. A plugin that skips work on a match must
still check its own state, such as files it published into the sandbox.
If the runtime context carries values that change per run, the
preparation digest changes too, so a plugin does its full preparation.
That is safe, but the plugin saves nothing. The MCP binding is the one
such value Core sets per run today, and MCP is excluded entirely. A
plugin that does publish MCP configuration into its sandbox must not
rely on this digest for it. Values outside the requirements are not
covered at all: the account identity and credentials, the GitHub broker,
project IDs, and the plugin's own runtime revision. A plugin must keep
checking those itself. A match lets a plugin skip publishing its launch
configuration, not computing the descriptor: `prepare` must still return
a launch whose `requirementsDigest` matches the current requirements,
which carry the current MCP assignment and binding.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:07:54 -08:00
Nicky LeachandClaude Opus 5.5 8cc26221af perf(native): accept a plugin's in-preparation project-auth check on warm attach (#21710)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Environment plugins can run a native Runner as a plugin task, and a
warm session keeps that Runner between turns
> - On a warm turn Core attaches the kept Runner to the new run: the
plugin prepares the new run, then Core checks managed project
authentication
> - A warm attach has no submit step, so Core sends a separate
`check_project_auth` operation after `prepare`
> - That separate operation is one more plugin dispatch and Launcher
round trip on every warm turn, about 0.12–0.18 s, and it is serial
> - This pull request lets the plugin do the check inside `prepare` and
say so, so Core does not send the second operation
> - The benefit is a faster warm turn with the same proof and the same
failure behavior

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The warm attach of a plugin Runner task (`plugin-runner-task.ts`
`prepareLaunch`). When `launched` is true, Core always calls
`checkProjectAuth` after `prepare`.

**Subsystem affected**
Native runtime, plugin Runner tasks, and the plugin SDK environment task
contract.

**Current behavior**
Cold starts fold the check into submit (`projectAuthCheckAtSubmit`).
Warm attaches send `prepare`, then a separate `check_project_auth`.
Measured on a plugin sandbox, the separate operation costs 0.12–0.18 s
on each warm turn.

**Proposed behavior**
The `prepared` result has a new optional flag, `projectAuthChecked`. A
plugin sets it only when it ran the managed project-authentication check
inside this preparation, for this launch's configuration, provider and
projects, and the check allowed them. Core then records the same lease
proof that an allowed `check_project_auth` records, and skips the
separate check.

**Reason and benefit**
One fewer serial round trip on every warm turn. The plugin already runs
the check for this task; this removes a dispatch, not a check.

**Breaking changes**
None. The flag is optional. A plugin that does not set it, or that saw a
refusal, gets the current behavior. Plugins detect the flag with the SDK
schema before they send it.

## What Changed

- `packages/plugins/sdk/src/environment-tasks.ts`: add optional
`projectAuthChecked: true` to the `prepared` result, with a doc comment
that defines it.
- `server/src/services/environment-task-runtime.ts`: when a `prepared`
result has `projectAuthChecked`, record
`runnerTaskProjectAuthValidation` for the launch's configuration,
provider and projects, without `atSubmit`. This is the proof that
`submit` and `read_file` require.
- `server/src/services/native-runtime/plugin-runner-task.ts`: do not
send `check_project_auth` when the prepared result has
`projectAuthChecked`.
- Tests: a warm attach with the flag makes only `prepare`; without it,
`prepare` then `check_project_auth`. The runtime test covers all four
combinations of `projectAuthCheckAtSubmit` and `projectAuthChecked`.

## Verification

- `npx vitest run src/services/native-runtime/plugin-runner-task.test.ts
src/__tests__/environment-task-runtime.test.ts` in `server/`: 157
passed.
- Not run locally: full typecheck and full suite. CI covers them.

## Risks

Low risk. Core trusts the flag the same way it trusts a plugin's
`check_project_auth` answer and `projectAuthCheckAtSubmit` today. A
refusal still goes through Core's own check, so the conflict error is
unchanged. The plugin side ships separately and sends the flag only when
its SDK accepts it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, tracing queries).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 03:13:48 +00:00
DottaandClaude Opus 5.5 328a954cc1 test(scaling): add a real-agent ramp harness for hosted instances (#21695)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The control plane must run 1,000 agents at once on one workspace
(`doc/plans/2026-11-13-thousand-concurrent-agents.md`)
> - Above about 50 agents, the evidence is a load model only. Nobody has
run 200 or 1,000 real agents
> - Phase 3 of the plan proves the finish line with real runs on hosted
instances, in a fixed order
> - This pull request adds the harness for those runs. It does not run
it against any real instance
> - The benefit is that each proof step is one command with a written
cost estimate, explicit pass/fail criteria, and a guaranteed cleanup

## Linked Issues or Issue Description

**What problem does this solve?**
The 1,000-agent plan needs real runs at 50, 200, 500 and 1,000 agents.
No tool exists that creates real agents and tasks on a hosted instance,
ramps concurrency, measures the control plane, and judges the plan's
finish line.

**Proposed solution**
A script under `scripts/` that ramps real agents through the product
API, stops at the first failed criterion, writes a machine-readable
result, and always cleans up what it created.

**Alternatives considered**
Extend the restart-chaos hosted mode. That mode grades release swaps at
up to 30 concurrent tasks. The ramp reuses its target guard and its
cleanup order, but it has a different workload, different measures and a
different stop rule.

## What Changed

- `scripts/scale-agents-ramp.ts` and `scripts/scale-ramp/`: the harness.
It reads the base URL from a flag or the environment and the credentials
from the environment only. It creates or reuses a company, then creates
agents, tasks, a project and an optional provider secret. It ramps
concurrency in steps (default 10, 50, 100, 200, 500, 1,000) and keeps
each agent on one small deterministic task at a time.
- It measures per step: tasks started, finished, failed and needing
Retry; failed runs; peak running agents; queue wait; time to first token
and first activity (from sampled run events); run duration; board GET
latency. An optional metrics endpoint supplies event-loop, statement and
pool figures. A missing source is reported as "not measured", never as
zero.
- It evaluates the criteria from the plan's finish line plus harness
criteria, writes them into `result.json` and `SUMMARY.md`, and stops at
the first failure.
- Cleanup runs in a `finally`: pause agents, cancel live runs, archive
tasks, terminate and delete agents, delete the stored secret, archive
the project and the created company, and wait for sandbox leases.
`--cleanup-only <run id>` finishes a cleanup after a crash. `--dry-run`
prints the plan and the estimate and makes no request.
- Target guard: `--target-kind` is required and must match the host.
Shared staging and production hosts need a second explicit flag. The
script asks for the host name before it starts unless `--yes` is given.
A run above the spend threshold needs `--confirm-spend <max USD>`.
- `scripts/scale-ramp/scale-ramp.test.ts`: 28 tests against an
in-process fake API with a virtual clock. They cover ramp logic, stop on
failure, criteria evaluation, cleanup after a mid-run crash and an
interrupt, `--cleanup-only` with and without a state file, dry run, the
production-host guard, the spend gate, and that credentials never appear
in output or files.
- `scripts/scale-ramp/tsconfig.json` is added to the test programs that
CI typechecks. The test file joins the `test:release-registry` node test
list, which CI runs.
- `doc/plans/2026-11-13-thousand-concurrent-agents.md`: a Phase 3
runbook with commands, cost, criteria, proof order, target prerequisites
and open risks.

## Verification

- `node scripts/ci-profile/node-test.mjs
scripts/scale-ramp/scale-ramp.test.ts
scripts/typecheck-test-programs.test.mjs`: 47 tests pass.
- `node node_modules/typescript/bin/tsc -p
scripts/scale-ramp/tsconfig.json`: no errors.
- `node cli/node_modules/tsx/dist/cli.mjs scripts/scale-agents-ramp.ts
--base-url https://scale1.test.paperclip.app --target-kind test-stack
--steps 50 --hold-minutes 30 --dry-run` prints the plan, the estimate
and the criteria, and makes no request.
- The harness was not run against any real instance. That needs the
owner's go-ahead.

## Risks

- Low risk for the product: the change adds a script, tests and docs
only. No server code changes.
- A real run creates real agents, tasks and sandboxes and spends money.
The guard, the typed confirmation, the spend gate, the task cap and the
recorded-spend stop limit this. The runbook lists the open risks.
- The cost defaults are assumptions until a first real run records
actual spend.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:10:51 -06:00
DottaandClaude Opus 5.5 9b581d689b fix(chat): report an outcome once and create no run when an earlier reply gave it (#21489)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In agent chat, a finished delegated task or a declined tool action
wakes the chat agent so it tells the person
> - #21401 stopped a second completion report when an earlier reply
already gave it
> - But the suppressed completion still created a zero-second heartbeat
run, and declined tool actions still got a second report
> - Run accounting must count only real turns, and the person must get
each outcome once
> - This pull request settles an already-reported outcome before it
becomes a run, and applies the same rule to declined tool actions
> - The benefit is one report per outcome, no phantom run, and no extra
model call

## Linked Issues or Issue Description

Refs #21401

**What happened?**
Chat eval cohort 13 (OpenCode,
`openrouter/deepseek/deepseek-v4.1-flash-1017`, master `df1b12ad8e`)
showed three problems after #21401.
1. `questions-and-approvals.duplicate-answer`: the completion wake was
suppressed, but it still created a 0-second run (`runtimeMode: legacy`,
no usage). The bounded-native-runs check rejected that run.
2. `skills.studio-test`: the CEO's single reply reported the Studio
result after it read the test task done (0 passed, 1 failed, links
COM-2). The completion was then suppressed, but the eval waited for a
separate completion run and timed out after 240 s.
3. `questions-and-approvals.decline-tool`: the person declined the tool
review one second after the turn for their message started. That turn
read the declined action and said so. The decline continuation, which
waited behind that turn, then posted the same declined outcome again.

**Expected behavior**
Each finished task or declined tool action is reported once in the
conversation. An outcome that an earlier reply already reported creates
no heartbeat run.

**Steps to reproduce**
1. In chat, delegate a short task, or start a governed tool read that
needs approval.
2. While it waits, send a message that wakes the chat agent.
3. Let the task finish, or decline the tool review, while that turn
runs. The turn reads the outcome and reports it.
4. See the completion wake become a 0-second run, or see a second
decline report from the decline continuation.

**Paperclip version or commit**
master `df1b12ad8e`

## What Changed

- `server/src/services/chat-completion-delivery.ts`:
`settleReportedChatCompletions` marks each undispatched completion that
an earlier reply reported as delivered, with that reply as its response.
It returns true only when nothing else is owed. The outbox `deliver`
also checks this before it requests a wake, so no wake and no run exist
for an already-reported completion.
- `server/src/modules/wake-queue`: a new transaction port,
`skipAlreadyReportedChatWake`, runs in the release drain before a
deferred wake is promoted. When the earlier reply already reported every
outcome of the wake, the wake ends `skipped` ("Already reported by an
earlier reply in this conversation") and no run is created. Wakes that
carry queued comments or a failed-chat retry are never skipped.
- `server/src/services/chat-interaction-outcome-report.ts` (new): the
same structural rule for declined tool actions. A conversation agent run
that lists its interactions records `chat_interaction.declined_read` for
each declined tool action. The decline continuation is settled when the
same agent replied in the conversation after the decline, from a run
with that receipt, posted after the read. The rule covers only declined
tool-action confirmations. Answers, plan reviews and accepted actions
keep their own turn.
- `server/src/routes/issues.ts`: `GET /issues/:id/interactions` calls
the recorder for agent runs.
- The run-start check from #21401 stays as a fallback for a completion
turn that was admitted before the earlier reply landed.
- `tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.ts`:
the Studio oracle (v5) accepts either the completion turn's reply or an
earlier CEO reply as the delivery. The earlier reply counts only from
saved state: no completion run exists, the replying run has a
`chat_handoff.completed_read` receipt for the test task, the reply was
posted after that read, it is the only such reply, and it links the
saved result. No wording check is added.
- `doc/run-log-events.md`: documents both read receipts and the skipped
wake outcome.

## Verification

- `cd server && npx vitest run
src/__tests__/chat-completion-delivery.test.ts
src/modules/wake-queue/adapters/chat-declined-outcome-reported.test.ts
src/modules/wake-queue/application/use-cases.test.ts`: all pass.
- Red: with the drain check and the outbox check disabled, 5 tests fail:
"starts no completion wake or run ...", "settles a deferred completion
wake without a run ... (true)", "settles the decline continuation with
no run ...", and both "settles an already-reported chat wake without a
run" cases.
- Negative controls stay green: a linking reply without a done read, a
decline the earlier turn never read, a non-tool-action card, and an
accepted card all still run their continuation.
- `npx vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/chat-catalog-domains/integrator-skill-studio.test.ts`:
34 pass. Red on the old oracle: "accepts the earlier reply the server
took as the completion report" fails. New negative cases reject no done
read, a read of another task, a reply before the read, a reply from
another run, two replies, an extra completion run, and a missing task
link.
- A narrow paid eval run of `duplicate-answer`, `studio-test` and
`decline-tool` on this branch follows in a comment.

## Risks

- A completion turn admitted before the earlier reply landed (a narrow
race) still ends as a short succeeded run with no model call, as in
#21401.
- If an earlier reply read the outcome but did not mention it, the
person does not get a second message. The read receipt, the
reply-after-read order and, for completions, the task link make this
unlikely.
- The decline rule covers only declined tool-action confirmations. Other
interaction outcomes are unchanged.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 14:24:16 -06:00
github-actions[bot]andlockfile-bot 0b2b7aa737 chore(lockfile): refresh pnpm lockfiles (#21604)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-14 09:49:06 +00:00
DottaandClaude Opus 5.5 f6c5986b61 feat(connections): machine sign-in (MCP OAuth client credentials) for organization connections (#21473)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use apps through connections to remote MCP servers
> - Today an organization connection must sign in as a person (browser
sign-in) or use a static key
> - Service accounts and CI-style integrations have no person who can
sign in
> - The MCP OAuth Client Credentials extension (2026-07-28) defines
machine-to-machine sign-in for this case
> - This pull request adds "Machine sign-in" for organization
connections, with a Paperclip-held signing key (recommended) or a client
secret
> - The benefit is that organizations can connect automation-owned MCP
servers safely, with credentials only in the vault and bound to one
sign-in service

## Linked Issues or Issue Description

**Subsystem affected**
Connections: remote MCP OAuth (server `tool-access/oauth`, tool gateway
token refresh) and the custom MCP server setup flow in the UI.

**Problem or motivation**
Organization-owned automation (service accounts, CI integrations) cannot
connect an MCP server that requires OAuth, because no person can
complete a browser sign-in. The MCP OAuth Client Credentials extension
(`io.modelcontextprotocol/oauth-client-credentials`) solves this with
`grant_type=client_credentials` and RFC 7523 signed client assertions.

**Proposed solution**
Add machine sign-in for organization connections only. Paperclip creates
a keypair per connection, keeps the private key in the company vault,
and shows the public key (PEM and JWK) for the admin to register. A
client secret is also supported. The sign-in service (issuer and token
endpoint) is pinned at setup. Tokens use the existing organization grant
refresh path, so caching, concurrent dedupe and 401 re-request reuse the
existing lease and single flight.

**Alternatives considered**
Reuse the deployment-wide `PAPERCLIP_TOOL_OAUTH_*` client env path that
already sent `client_credentials`. Rejected: it is not per connection,
it keeps secrets in env, and it has no issuer pin. Route machine sign-in
through `connectGalleryApp`. Rejected: its rollback deletes the draft on
a failed test, which would lose a key the admin already registered.

**Roadmap alignment**
Phase 1 of the enterprise MCP plan (no enterprise single sign-on
needed). OIDC login and enterprise-managed authorization (ID-JAG) are
later phases.

## What Changed

- `oauth/client-credentials.ts`: keypair generation (RS256, or ES256
when the service requires it), RFC 7638 key IDs, and RFC 7523 assertions
(`iss` = `sub` = client ID, `aud` = token endpoint, 60 s `exp`, unique
`jti`) with Node `crypto` only.
- `oauth/client.ts`: the token exchange accepts a signed assertion and
then sends only `client_assertion_type` and `client_assertion` (no
`client_id` or secret), as the extension specifies.
- `oauth/client-credentials-service.ts`: discovery of the sign-in
service (protected-resource metadata, then RFC 8414/OIDC metadata with
issuer check, or a token URL entered directly), issuer pinning,
vault-only key and secret handling, draft creation, test-and-activate,
and no-downtime rotation (test the new secret first; stage a new key and
switch only after the provider accepts it).
- `oauth/grant-refresh.ts` and `oauth/connection-refresh.ts`: machine
sign-in requests a new access token with the connection's own
credentials instead of needing a refresh token. It never falls back to
deployment-wide OAuth clients.
- `catalog/discovery.ts`: a rejected token on a machine sign-in
connection never re-runs discovery or rewrites the pinned endpoints.
- `mcp-http.ts` and the tool gateway: declare
`io.modelcontextprotocol/oauth-client-credentials` in client
capabilities only for connections that use it.
- Routes (with OpenAPI and activity logging): `POST
/companies/{companyId}/tools/machine-sign-in`, `GET|POST
/tool-connections/{id}/machine-sign-in[/test|/rotate|/rotate/complete]`.
Removal activity now records `machineSignIn: true`. The runner API
treats these routes as sensitive.
- Shared validators and types for machine sign-in.
- UI: "Machine sign-in" under Advanced authentication in the custom MCP
server flow (organization connections only), `MachineSignInSetup`
(choose Paperclip key or client secret, register the public key, test),
and Storybook stories for each setup state.
- Docs: `doc/connections/MCP-CLIENT-CREDENTIALS.md`, linked from
`doc/connections/README.md` and `GENERIC-REMOTE-MCP.md`.

## Verification

- `server/src/__tests__/tool-access-client-credentials.test.ts`: 16/16
pass. It uses a fake sign-in service that verifies real signatures
against registered keys, and a fake MCP server. It covers RS256 and
ES256 assertion claims and signatures, assertion-only client
authentication, the secret flow (`client_secret_basic`), discovery and
direct token URL, issuer pinning (refuses a different sign-in service
and sends it nothing), the pinned endpoint surviving a hostile
challenge, the token cache (0 token requests while live), concurrent
dedupe (5 callers on 2 service instances produce 1 token request), 401
re-request, key and secret rotation, a refused personal connection,
secret redaction in responses, rows and activity, and vault cleanup on
removal.
- Red/green: with the grant-refresh machine sign-in branch and the
discovery pin guard disabled, 5 runtime tests fail (pin, cache/dedupe,
401 re-request, both rotations). Restored: 16/16 pass. The other tests
cover new routes and services that do not exist on `master`.
- `ui/src/features/connections/MachineSignInSetup.test.tsx`: 3/3.
`ConnectionSetupFlow.architecture` and `.oauth` tests: 19/19.
- `openapi-routes.test.ts`: 51/51. `runner-api.test.ts`: 2153/2153.
`mcp-http.test.ts`: 9/9. Existing `tool-access-service.test.ts` refresh
and legacy client-credentials tests: 54/54 (filtered).
- Server package `tsc --noEmit`: clean. UI `tsc`: clean except the
pre-existing generated Storybook catalog imports.
- Not run locally (machine under heavy load): full suites, repo-wide
typecheck, browser check of the live app. CI covers breadth.

## Risks

- No migration. New config lives in `config.oauth` (`grantType:
"client_credentials"` plus `clientAuth`). Older env-configured
client-credentials connections have no `clientAuth` and keep their
original path.
- The capability declaration adds one extension key only for machine
sign-in connections. Other requests are unchanged.
- Rotation has no settings-page control yet; it is available through the
API. This is a follow-up.
- Low risk to agent boot and time to first token: a live token is read
exactly like a browser sign-in token.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 03:10:32 +00:00
DottaandClaude Opus 5.5 a726dce4f0 feat(mcp): show live progress for long connected-app tool calls in the task chat (#21449)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents call tools from connected apps through the Paperclip MCP
gateway
> - Some tool calls take a long time: exports, builds, large searches
> - The MCP progress utility lets an app report how far a call is, but
Paperclip ignored it
> - So the task chat showed only a spinner until the call finished, with
no sign of life
> - This pull request reads the app's progress and shows it on the
tool's row while the call runs
> - The benefit is that people can see that a long call is working, how
far it is, and how long it has run

## Linked Issues or Issue Description

**Subsystem affected**
Connected apps (MCP gateway) and the task chat.

**Problem or motivation**
A long tool call from a connected MCP app shows only a spinner in the
task chat. Paperclip does not send a `progressToken`, and it drops
`notifications/progress` from the app.

**Proposed solution**
Send a `progressToken` on each `tools/call` that an agent run makes.
Read the app's progress notifications from the call's response stream.
Show a progress bar (or an indeterminate sweep), the latest message, and
elapsed time on the running tool row. Keep the latest state so that a
reload mid-call shows it.

**Alternatives considered**
A new transport for progress. The existing company live-events socket
already carries per-run events with run read checks, so this change uses
it.

**Additional context**
Spec:
https://modelcontextprotocol.io/specification/2026-07-28/basic/utilities/progress

## What Changed

- `mcp-http.ts`: `mcpHttpRequestInit` can add `_meta.progressToken`.
`readMcpHttpResponse` gives each notification on the response stream to
an observer before the result. The observer cannot fail or delay the
call. New `mcpProgressNotificationOf` matches a notification to its
token.
- `tool-gateway.ts`: remote HTTP and local stdio `tools/call` from an
agent run send a per-call token and forward progress to a reporter.
Calls without a run send no token.
- New `server/src/services/tool-call-progress.ts`: checks each update
(finite, increasing, plain single-line text of at most 200 characters).
It publishes a `heartbeat.run.tool_progress` live event at most every
250 ms (latest wins) and saves the latest update on the invocation at
most every 1 s. Nothing is awaited by the call. A final `done` event
goes out when the call ends.
- DB: nullable `tool_invocations.progress` jsonb column (migration
`0479`, `ADD COLUMN` only). A save never writes onto a finished call.
- Live delivery: the new event follows its run like transcript output. A
client gets it only for runs it follows, after the run read check, in
the run's company only.
- New route `GET /api/issues/:issueId/tool-call-progress` (task read
check) returns the latest progress of the task's running calls, for a
reload mid-call. Documented in `docs/api/issues.md`.
- UI: `ToolCallProgressProvider` (one per chat) keeps the latest update
per call. A running tool row subscribes and shows a bar or sweep, the
message, the percent and elapsed time (design tokens, reduced motion
honored). Finished rows drop it and never fetch or re-render for
progress. `LiveUpdatesProvider` ignores the event for query
invalidation.
- `packages/kv-demo-mcp-server`: new `slow_with_progress` demo tool
(`steps`, `stepMs`, `withTotal`) in its own file.
- Storybook: `Tasks/Tool progress` stories (running with total, running
without total, done).
- Docs: new `doc/connections/MCP-PROGRESS.md`;
`MCP-PROTOCOL-VERSIONS.md` no longer lists progress as unsupported.

## Verification

- Red/green: on the original source, 5 of the new server tests fail
(token in `_meta`, SSE with interleaved progress and result,
notification matching, run-interest delivery, gateway end-to-end). With
this change they pass.
- `server/src/__tests__/mcp-http.test.ts` (13),
`tool-call-progress.test.ts` (9: throttling, latest wins, bounded text,
out-of-order drop, finish, failures never throw, company/run/tool
scoping), `tool-call-progress-store.test.ts` (2, embedded Postgres:
company and task scoping, no write onto a finished call),
`realtime/live-event-delivery.test.ts` (run interest),
`tool-gateway.test.ts` end-to-end against the real SDK kv demo server
over Streamable HTTP: 6 provider updates over about 1.2 s give 3 to 7
throttled events, all for the right company, run, task and tool, the
last one `done`; no event reaches another company; the invocation keeps
the progress; the result is unchanged.
- UI: `tool-call-progress.test.tsx` (10: name matching, row states,
plain-text rendering, drop on complete, restore after reload, no fetch
for finished rows). All 67 task-chat test files pass.
- `packages/kv-demo-mcp-server`: 13 tests, including progress over HTTP
with and without a total.
- Storybook: checked the three stories in a browser. The bar fills to
40%, the sweep animates, the elapsed time ticks, and the done row shows
no progress.
- `pnpm verify`: see the CI checks for the final revision.

## Risks

- Low. The migration adds one nullable column.
- An SSE response is now bounded per message, not per stream, so handled
notifications (such as progress) do not use up the result's size limit.
Each message is still bounded.
- Progress code runs only while a run's tool call is in flight. It adds
a few bytes to the request and throttled, fire-and-forget work. Agent
boot and time to first token do not change.
- A row finds its call by run and tool name. If one run calls the same
tool twice at the same time, both rows show the newer call's progress.
- Progress from a call that became an MCP Task is not shown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 20:14:22 -06:00
Devin FoleyandPaperclip 0a96f972e1 Connect Tailscale networks through device sign-in (#21526)
## Thinking Path

> - Paperclip manages AI agents and the environments where they run.
> - SSH environments can reach machines through a company Tailscale
connection.
> - The initial connector requires a managed OAuth app that Tailscale
limits to its own tailnet.
> - Customers need to connect their own tailnets without registering an
app or supplying a key.
> - This pull request uses Tailscale's normal device sign-in for new
connections.
> - Customers approve the Paperclip device and then select their
machines in SSH environments.

## Linked Issues or Issue Description

**Problem or motivation**

The current Tailscale enrollment path cannot serve independent customer
tailnets through the documented self-service OAuth app model. It also
requires Cloud broker configuration and pilot admission before a
customer can start.

**Proposed solution**

Start a private Tailscale device in the tenant and open its normal
sign-in link. Save its encrypted device state before reporting a
successful connection. Retain the existing SSH machine selection and
agent account behavior.

**Alternatives considered**

Auth-key entry would require customers to copy a credential. A public
OAuth integration still needs a confirmed registration path. Device
sign-in uses a documented Tailscale flow without either requirement. See
[device
authentication](https://tailscale.com/docs/features/tsnet#device-creation-and-authentication)
and [OAuth app
limitations](https://tailscale.com/docs/features/oauth-apps).

**Related PR search**

Searched GitHub PRs for `tailscale` and `device sign-in`, and issues for
`tailscale`. This follows the merged network connector in
[#20617](https://github.com/paperclipai/paperclip/pull/20617) and
preserves the SSH agent-account support in
[#21248](https://github.com/paperclipai/paperclip/pull/21248). The open
[#15648](https://github.com/paperclipai/paperclip/pull/15648) proposes
storing an OAuth client and checking its health; this change instead
enrolls a device through interactive sign-in so customers do not need an
OAuth app.

## What Changed

- New connections use device sign-in. Existing connected devices and
pending legacy OAuth callbacks remain supported.
- Pending sign-ins are bound to their company and initiating user. They
have a fixed ten-minute deadline. Device state uses server-owned vault
secrets with trusted version and digest checks.
- The active Tailscale CLI waits for consent and device approval.
Cancellation, expiry, secret deletion, and stale completion cannot
restore a cancelled connection.
- The connector stays open during sign-in. It provides a link when
popups are blocked and shows device approval separately. Progress checks
stop on completion, failure, expiry, or unmount.
- Shared response types, OpenAPI contracts, SDK source pins, and the
API/setup documentation describe the new flow. No database migration or
Cloud provider configuration is required.

## Verification

- 109 focused tests cover network processes, the real database and
encrypted vault, routes, UI, and legacy connection behavior.
- Server and UI typechecks pass.
- Follow-up `29cf296a597`: reproduced the missing OpenAPI route failure
locally. All 70 OpenAPI and Tailscale route tests and server typecheck
pass after synchronizing the API contract.
- Follow-up `79de05f38b9`: the transient-recovery regression failed
before the fix; all 20 connector UI tests and UI typecheck pass after
it. The query-error policy check reports no findings.
- The public SDK input registry check passes.
- [CI attempt
2](https://github.com/paperclipai/paperclip/actions/runs/43535594257/attempts/2)
passed on the final head, `29cf296a597e470d432b84767050bc539caa280e`: 57
successful checks, no failures or pending checks. The interrupted
canary, E2E shard, and serialized-suite jobs passed on rerun. Greptile
is 5/5 on this head with no unresolved review threads.
- `pnpm verify --explain` exits 2 because the documentation change
selects broad CI coverage. Focused local checks passed; broad CI passed
on `29cf296a597e470d432b84767050bc539caa280e`.
- Live local qualification on
`29cf296a597e470d432b84767050bc539caa280e`: native device sign-in
completed with interactive user approval. An isolated Ubuntu 24.04 ARM64
EC2 machine appeared in discovery and passed SSH pairing and workspace
probes. The assigned Claude Code agent used a Paperclip-managed
subscription, created `hello_world.txt` containing exactly `Hello,
world!` plus a newline, posted a completion comment, and marked the task
done. An independent read through the private connector verified the
bytes, remote user, and hostname. The remote machine had no stored
Claude login and no leftover temporary credential directories.
- Restarted the local Paperclip server and verified the same Tailscale
device identity, online peer discovery, successful SSH probe, and
unchanged file without new consent. Temporary public SSH ingress was
removed.
- Local qualification covers one user tailnet and the final Core
revision. Browser UI acceptance, two unrelated tailnets, a hosted
restart, and an agent task on the final staging revision are not yet
verified.

## Risks

- The enrollment method changes. It must pass the live qualification
above before broad rollout claims.
- Existing encrypted device state and the vault master key must survive
a restart or redeploy. Device expiry or removal still requires
reconnecting.
- The existing one-serving-process-per-instance constraint remains.
Concurrent replicas need durable network ownership.
- Pending setup uses bounded progress requests and an event-driven CLI
with a deadline. It adds no idle database scan.

## Model Used

OpenAI GPT-6 Astra (`gpt-6-astra`) through Codex in T3 Code, with xhigh
reasoning. Context window size was not exposed by the runtime. No
subagents were used.

## Checklist

- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] My branch name describes the change and contains no internal
ticket or instance details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 17:38:55 -08:00
Nicky LeachandClaude Opus 5.5 1140984b82 feat(plugins): tell an opted-in workspace provider the run's environment (#21512)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A plugin can be a project workspace provider and also an environment
driver for the agent runs that use those workspaces
> - Core asks the workspace provider to prepare the project workspace
before the run acquires its environment lease
> - The provider knows the run, agent, project, and execution workspace,
but not the environment that the run selected
> - A provider whose compute exists per run (for example, a sandbox that
it creates from the environment's driver config) cannot start or find
the run's compute for the workspace work
> - This pull request lets a provider opt in to receive the run's
selected environment, when that environment is one of the provider's own
drivers
> - The benefit is that such a provider can prepare the workspace on the
run's own compute, with no change to the order of operations in Core

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Plugin workspace preparation for runs whose environment is a driver of
the same plugin.

**Subsystem affected**
Plugin SDK workspace provider params, the plugin manifest workspace
provider declaration, and `callWorkspaceProvider` /
`preparePluginWorkspace` in the server.

**Current behavior**
`prepare` params carry `runId`, `agentId`, `projectId`, and
`executionWorkspaceId`. They carry `taskLease` only when Core acquired
the lease first. They never name the run's environment.

**Proposed behavior**
A provider that declares `workspaceProvider.workspaceEnvironmentVersion:
1` receives `environment: { id, driverKey, config }` when the run's
selected environment is a plugin environment of that same provider.
`config` is the environment's `driverConfig`, the value that the
driver's acquire receives.

## What Changed

- SDK: optional strict `environment` field on
`pluginWorkspaceParamsSchema`.
- Manifest: optional `workspaceProvider.workspaceEnvironmentVersion: 1`,
next to the sibling version fields. Core sends the new field only to
providers that declare it, because providers parse the params with the
strict schema.
- Server: `callWorkspaceProvider` adds `environment` only when the
provider opted in, the environment driver is `plugin`, its `pluginKey`
is this provider, and it names a `driverKey`. The field is not part of
the operation ID.
- Server: `preparePluginWorkspace` passes the run's environment to its
prepare call. This covers normal, verify-existing, fresh, and
replacement prepares.
- Calls that do not have the run's environment do not send it: finalize,
inspect, remove, reservation reconciliation, and the retry-candidate
verification in `fresh-workspace-continuation.ts`.
- Lease acquisition, the `sessionHandoffVersion` gate, and the order of
operations do not change.
- Docs: `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` describes the
capability and its exceptions.

## Verification

- `packages/plugins/sdk/tests/workspace-provider.test.ts`: 11 passed.
The schema accepts the field and rejects malformed values.
- `packages/shared/src/validators/plugin.test.ts`: 65 passed. The
manifest accepts only version 1.
- `server`: `plugin-workspaces.integration`,
`plugin-fresh-workspaces.integration`, `project-environments`, and
`heartbeat/workspace-preparation`: 226 passed, against embedded
Postgres. The new integration test checks the exact shape, an unchanged
operation ID, verification prepares, and omission for an undeclared
provider, another plugin's environment, non-plugin drivers, a missing
`driverKey`, and no environment. It fails when the server stops passing
the environment.
- `pnpm typecheck` passes in `packages/shared`, `packages/plugins/sdk`,
and `server`.
- The full suite was not run locally. CI runs it.

## Risks

- Low risk. Providers that do not declare the capability get the same
params as before.
- `config` is the stored `driverConfig`, in the same form that the
driver's acquire gets. Secret references in it reach the provider in
that same form.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, code search, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [ ] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 15:43:10 -08:00
github-actions[bot]andlockfile-bot 0749d74f8b chore(lockfile): refresh pnpm lockfiles (#21516)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-13 23:17:45 +00:00
Devin FoleyandPaperclip 21bf7ebcee feat(connectors): connect Tailscale networks to SSH environments (#20617)
## Thinking Path

> - Paperclip manages work for AI agents.
> - SSH environments run tasks on computers that users own.
> - A Paperclip server needs network access before it can reach a
private computer.
> - A reusable Tailscale connection belongs in Connectors.
> - Connected machines should become choices in the existing SSH
environment form.
> - This change connects a tailnet and resolves selected machines
through a private network client.
> - Agents retain their choice of harness and AI account.

## Linked Issues or Issue Description

**Subsystem affected**

Connectors, environment settings, SSH execution, company secret storage,
and runtime packaging.

**Problem or motivation**

Entering a Tailscale address in the SSH form does not connect the
Paperclip server to that tailnet. Requiring a separate device key for
every environment also duplicates network setup.

**Proposed solution**

Add a Tailscale Connector with **Connect Tailscale Network**. Use
brokered OAuth consent to enroll one private network client per
connection. Let SSH environments select a connected network and stable
machine identity. Preserve the enrolled node state through the company
vault so a restart does not reuse the one-time token.

**Alternatives considered**

A separate Tailscale environment type would duplicate SSH behavior and
mix network access with harness selection. This change uses the existing
`ssh` driver. Related #15648 adds OAuth client credentials and a health
check; this change uses user consent and supplies outbound task
execution. Related #11743 concerns HTTPS brokering.

**Roadmap alignment**

This extends the existing Connected Apps, environment, and Personal &
Shared AI Accounts capabilities. This PR is stacked on #21248, which
provides SSH account selection independently of Tailscale. The hosted
OAuth broker is a companion control-plane change.

## What Changed

- Preserve Cloud reauthentication in the current browser tab. Explain
canceled or failed consent in the connector and offer a retry of the
same authorized saved connection.
- Record Cloud enrollment starts initiated through the Tailscale
connector in the company activity log.
- Check install-preparation descendant liveness through the kernel
process table in the Runner support test. Keep the direct child’s strict
reaping assertion and reject any non-zombie descendant.
- Align the application image stage contract with the Tailscale client
stage after the upstream Runner-image refactor. Keep the exact stage
list and check every application stage, including `tailscale-client`,
for prohibited Runner compiler commands.
- Group Tailscale consent, discovery, daemon lifecycle, pairing, and
deletion under `server/src/services/connectors/tailscale/`, with their
service tests. Update existing route, runtime, environment, vault, and
company-deletion imports. Keep the existing contracts, authorization,
audit, and transaction owners.
- Handle early stdin closure in the Kubernetes real-shell test helper.
Stop the input stream on `EPIPE`, preserve nonzero child exits, and
reject a zero exit when input was lost. Add two real-process regression
cases. Production transfer code and confinement checks are unchanged.
- Read configured Tailscale executable paths when a client starts, after
supported `.env` loading. Keep explicit command overrides available.
- Discard pending pairing results after switching to Direct SSH or
closing the form. Retain the existing company, connection, machine, and
port checks.
- Store network credentials with an immutable server-owned purpose from
their first vault row. Exclude them from generic catalogs, bindings,
environment projection, and value reads. Use a dedicated audited server
consumer for pinned connector reads. Block unclassified legacy
serialized node state at the same value boundary. Keep SSH private-key
binding separate from connector credentials.
- Gate network queries and setup on verified local-implicit or
instance-admin access. Hide Tailscale catalog rows for other members,
show an admin requirement on direct setup navigation, and avoid device
discovery before management access is known.
- Keep failed-client cleanup in the network manager. A failed request
cannot disconnect a newer retry for the same identity; explicit
connector disconnect and enrollment retirement still stop their
networks.
- Check for readable Ed25519, ECDSA, or RSA host public keys before
pairing edits `authorized_keys`. Print a known-hosts entry for every
available key, or fail before changing authorized keys if none are
available.
- Return self-hosted Cloud enrollment to the Tailscale connector. Save
the unprefixed setup path and let the shared callback add the verified
company prefix. Keep external and unrelated destinations rejected.
- Hide pairing instructions after an SSH port change. Ignore late
key-selection results for the old port. Operators prepare again for
matching known-hosts output.
- Retire connector references before deleting a company or its vault
credential. Stop company daemons after the database commit. Keep failed
secret retirement retryable and prevent late node-state writes from
restoring deleted connections. Discard retired state without blocking
behind the vault deletion lock.
- Reload a mismatched credential before replacing a ready daemon.
Recheck the active entry and manager generation after the read. An older
request cannot stop a newer healthy socket or cross shutdown.
- Pin connector credentials to the exact vault version and SHA-256
digest recorded by trusted enrollment and node-state writes. Ignore
generic secret rotations. Advance both fields with a compare-and-set
guard. Reject unpinned or tampered credentials before network startup.
- Save an explicit Direct SSH selection and clear old Tailscale fields
after config merging. Preserve the operator's entered direct host.
- Route all Tailscale SSH runtime access through the connection service
after SSH-key loading. Pin access to the environment's saved connector
identity and recheck after device discovery. Remove standalone key-based
daemon startup and reject raw enrollment keys or credential-only network
configurations. Align the shared config type and SDK pins.
- Correct two E2E assumptions exposed during connector qualification.
Keep the two-window chat fixture stale until its own mutation. Validate
the original failed run and its permitted write-revocation timestamp in
the legacy recovery browser test.
- Add the Tailscale Connector page, catalog entry, OAuth return flow,
connected-network management, device discovery, and disconnect. Document
all six routes in OpenAPI and the API reference, including their
instance-admin requirement.
- Add company-scoped connection storage in `0480_lazy_lethal_legion`,
trusted credential version/digest fields in `0481_round_pet_avengers`,
and server-owned secret purpose/backfill in
`0482_aberrant_rumiko_fujikawa`. Generate all three migrations through
the maintained command and retain the five newest snapshots.
- Add machine pairing that generates a fresh SSH key per request, stores
its private half in the company vault, and uses the generated public
half in the install command. Do not trust editable saved metadata.
Preserve existing authorized keys, including a file without a final
newline. Keep SSH username, workspace, and trusted host keys explicit.
- Use the existing SSH driver. Resolve the machine's current address
from its stable device ID. Reject disconnected, removed, offline, and
wrong-company selections.
- Hide Network when Direct is the only option. Select a sole connected
tailnet automatically and hide its selector. Remove the requested setup
helper copy. Do not restrict the harness to Claude.
- Run a private userspace Tailscale daemon with a separate socket.
Persist its node state before execution. Restore it after restart, bound
startup, serialize reconnect cleanup, and close it during server
shutdown.
- Guard OAuth completion with a durable attempt marker. Disconnect and
newer authorization invalidate older callbacks. Close replaced daemons
and remove cancelled enrollment secrets without changing newer
connection state.
- Preserve SSH host verification and tailnet-specific session identity.
Honor the configured Tailscale executable path, including spaces and
OpenSSH percent escaping.
- Package checksum-pinned Tailscale clients and their license in the
standard runtime image. Exercise the client build stage in PR CI and
include it in image smoke checks.

## Verification

- Merge status: all CI and image checks pass, and current-head Apex is
5/5 with no unresolved comments. GitHub rejected a direct merge because
the active master ruleset requires code-owner approval for
`.github/workflows/pr-trusted.yml`. The [CODEOWNERS workflow
pattern](https://github.com/paperclipai/paperclip/blob/master/.github/CODEOWNERS#L3)
applies; eligible maintainers already have review requests. Auto-merge
is enabled with the squash method for the verified head. The PR remains
open pending that required approval; no branch policy was bypassed.
- Final revision: `fdb29b2cc366f4945b7f4c3e2b2d25070329a21b`, including
master `999709b0a6dd416af335b5debded13e1de7fcb7a`. [Final CI
verification
passed](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132213508320).
All required checks pass, all applicable test/build gates are green, and
the branch is mergeable against master. SSH prerequisite #21248 and
Cloud broker paperclipai/paperclip-cloud#1314 are already merged.
- [Apex reviewed this exact revision at
5/5](https://github.com/paperclipai/paperclip/pull/20617#issuecomment-6776629437).
There are no unresolved review threads. The PR is classified as Critical
impact.
- Final image qualification passes for the [application Runner
path](https://github.com/paperclipai/paperclip/actions/runs/43518584522/job/132206784267),
[amd64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296198),
and [arm64 Runner
image](https://github.com/paperclipai/paperclip/actions/runs/43518584544/job/132207296156).
Typechecks, Runner support, native/Rust tests, provider tests,
serialized-server suites, browser shards, and the remaining applicable
CI checks pass. This is source/image CI qualification, not a production
rollout.
- The last master conflict affected only Docker stage declarations and
their boundary test. Both the checksum-pinned Tailscale stage and
upstream `runner-lockfile` guard are retained, including the upstream
`deps` inheritance. The Tailscale client stage is byte-identical. All
180 cases across five Docker/image contract suites pass locally, along
with server and UI typechecks. The maintained 543-input SDK check has no
drift. Runtime and lock pins are unchanged. Secret/PII and whitespace
scans pass.
- Seven OAuth regression cases failed before the fixes. The current-tab
test uses the real handoff helper, checks that the session is stored
before reauthentication navigation, mounts the shared return page,
resumes consent, and verifies handoff cleanup. Callback tests check
trusted saved-connection context, fixed safe outcomes, failure audit
metadata, denied membership and expired-state rejection, JSON API
errors, and no provider text in browser returns. UI retries require a
connection in the current company's catalog. All 80 OAuth
route/service/UI/shared-handoff cases pass; server and UI source/test
typechecks pass.
- [Server shard 4/12 on the prior
refresh](https://github.com/paperclipai/paperclip/actions/runs/43510727031/job/132180301436)
passed 2,784 cases and skipped 21, then failed a reassignment fixture's
`TRUNCATE companies CASCADE` cleanup with PostgreSQL `40P01`. Its
automatic retry found the surviving `REA` prefix. The unchanged test
passes all three cases locally. A controlled reader/cleanup overlap
reproduces the same deadlock, but the exact competing CI query was not
logged. Each case now has a distinct company prefix and identifier; the
owned database is removed at suite end. The three test bodies and
assertions are byte-identical. Eight local reassignment/lock-release
cases pass, and the [final server shard 4/12
passes](https://github.com/paperclipai/paperclip/actions/runs/43518584814/job/132207159392).
Production reassignment logic, deadlines, and retries are unchanged.
- The initial master integration passed 330 cases across 17 suites.
Coverage includes connector state, device discovery, network lifecycle,
pairing, routes, SSH binding and exports, environment persistence, and
connector UI. All three unpublished migrations were regenerated after
master's `0479_short_darkstar`: `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa`. Each SQL
body is byte-identical to its earlier unpublished migration. Generated
snapshots preserve existing master tables and add only the intended
Tailscale table and secret-purpose column. Exactly five snapshots
remain, 0478–0482. Numbering, safety, compatibility, and snapshot-drift
checks pass. The later Docker refresh did not alter this migration
sequence.
- Earlier attempts remain part of the verification record. On `1c020a0`,
Rust cache restoration and server shard 5/12 were interrupted by
shutdown signals, Runner shard 2/3 lost communication with no
retrievable log, and an amd64 provider download failed with npm
`ECONNRESET`. Same-head retries recovered, without establishing the
cause of the communication loss. Later superseded runs also had
cancellations with incomplete logs. These interrupted attempts are not
counted as passes. The final revision has fresh successful CI.
- Local platform limits remain recorded. The Kubernetes suite on macOS
passed 191 cases, failed 12 Linux `/proc/self/fd` transfer cases, and
skipped four Linux-only cases. Unchanged source reproduced the same 12
failures. An initial run also lacked provider dependencies until the
maintained preparation command installed them. Those local attempts are
not green qualification. Final Linux provider CI passes. Local Docker
qualification was unavailable; final image checks passed in CI.
- Before PR separation, a live tailnet/SSH Claude Code task completed on
Ubuntu ARM64 without provider login for the remote user. The selected
Paperclip subscription ran it, wrote the expected file, posted
completion, and marked the task done. File contents and private runtime
cleanup were checked. This is prior live evidence; the final revision
was not live-retested. Other providers have automated coverage only.
Hosted registration for unrelated tailnets and production concurrency
remain qualification limits below.
- The Superagent security and supply-chain reports remain neutral and
inconclusive. Earlier attempts to restart the failed security scan
returned HTTP 404. Supply-chain intelligence lacked coverage for
extracted pinned dependencies and the shared workflow reference. These
reports do not establish successful security qualification. The required
CI and review gates pass; this external coverage gap remains disclosed.

## Risks

- The Superagent security and supply-chain reports are inconclusive. The
security scan re-run request returned HTTP 404, and the supply-chain
scanner lacks coverage for the extracted dependencies and shared
workflow reference. These neutral results are not successful security
qualification.
- The server-owned secret purpose is internal database metadata, not an
HTTP-writable field. The generic secret service owns enforcement so
direct API reads and runtime projection share the same restriction.
Previously copied node state remains valid outside Paperclip until the
enrolled device is revoked in Tailscale; a local disconnect cannot stop
an external copy.
- Tailscale network configurations now require a connected Connector and
selected machine. The unreleased key-only path is removed; any saved
configuration from that path needs Connector setup.
- The connector remains a pilot. The local test used an OAuth app
registered in the consenting user's tailnet. Qualify hosted app
registration for unrelated customer tailnets before general
availability.
- This implementation supports one serving process per instance.
Concurrent replicas need exclusive ownership of the persisted network
identity before production rollout. Preserve the database and company
vault master key across restarts.
- Each machine-pairing preparation creates a new vault secret. Unused
generated keys follow the existing secret deletion lifecycle. A saved
SSH key remains available as an explicit choice.
- Apply additive migrations `0480_lazy_lethal_legion`,
`0481_round_pet_avengers`, and `0482_aberrant_rumiko_fujikawa` before
application rollout. The new credential version and digest columns are
nullable. Existing unpinned connections require fresh OAuth consent; the
migration deliberately does not trust or backfill a generic secret
value. Regenerate or renumber migrations if master advances the sequence
before merge.
- The target needs an SSH server, its chosen harness, and allowed
tailnet access. A reachable network does not grant SSH or provider
authorization. Pairing reads standard `/etc/ssh` host-public-key files;
custom host-key locations require manual known-hosts configuration.
- Roll back the application revision to remove new entry points.
Existing connection secrets and registered Tailscale nodes retain their
separate lifecycle. Do not drop the table or delete node state as part
of an application rollback.
- Production qualification still needs denied-policy, revocation,
cancellation, and deployment-overlap checks. Local Docker and a
production deployment were not performed.

## Model Used

OpenAI Codex, gpt-6.1-sol, xhigh reasoning, with local code execution
and tool use.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used
- [x] I have checked ROADMAP.md and this extends existing connector and
environment features
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have described the issue in-PR using the feature request fields
- [x] I have not referenced internal issues, local URLs, or private
machine identifiers
- [x] My branch name describes the change
- [x] I have run focused tests locally and they pass
- [x] I have added tests for new behavior and failure paths
- [x] I have updated documentation
- [x] I have documented migration and runtime risks
- [x] All Paperclip CI gates are green
- [ ] Required code-owner approval received for the workflow change
- [x] Greptile Apex is 5/5 on the final head with no unresolved comments
- [x] I will address CI and review findings before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 15:15:43 -08:00
Nicky LeachandClaude Opus 5.5 5df1ada9c1 feat(plugins): send configured environment on plugin task resume (#21352)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #21335 lets a plugin task provider that restarts the Runner on
`resume` opt in (`resumeCredentialVersion: 1`) to receive the claimant
run's harness credential
> - A provider restarting the Runner also needs the agent's configured
environment, which Core sends on `resume` only during a session handoff
> - Without it, a Runner a provider moves to a new computer starts
without the agent's configured environment variables
> - This pull request widens that opt-in to both launch inputs, under a
name that says so, before any provider adopts the old name
> - The benefit is that a provider can restart a parked Runner on resume
with the same credential and environment a `submit` would carry

## Linked Issues or Issue Description

Refs #21289
Follows: #21335

**What existing behavior does this improve?**
The opt-in resume launch inputs added in #21335.

**Subsystem affected**
Plugin SDK environment task schema (`describe` capability), the plugin
Runner task owner's `resume`, and the plugin environment task docs.

**Current behavior**
With `resumeCredentialVersion: 1`, `resume` carries `credential`.
`configuredEnvironment` is sent on `resume` only for a session handoff.

**Proposed behavior**
The capability is renamed `resumeLaunchInputsVersion: 1`. With it,
`resume` carries `credential` as before and also `configuredEnvironment`
(same builder as `submit`) when the provider advertises
`configuredEnvironmentVersion: 1`. Session handoff is unchanged.

**Reason and benefit**
A provider that restarts the Runner on resume gets both launch inputs it
needs.

**Breaking changes**
The capability name changes from `resumeCredentialVersion` to
`resumeLaunchInputsVersion`. No provider advertises the old name yet (it
merged in #21335 and has no adopter), so nothing changes behavior.

## What Changed

- Rename `resumeCredentialVersion` → `resumeLaunchInputsVersion` in the
SDK `describe` result schema, executor plumbing, owner input, docs and
tests.
- With the capability, `resume` also adds `configuredEnvironment` when
the provider advertises `configuredEnvironmentVersion: 1`. A normal
resume does not mark the variables as delivered, since a live Runner
keeps the environment it started with.
- Docs: the "Release and resume" section describes both fields; the
"Configured environment" section points to it.
- The runner-e2e SDK public-input pin is updated.

## Verification

- SDK vitest `tests/environment-task-release-resume.test.ts` and
`tests/environment-tasks.test.ts`: 68 passed.
- Server vitest `plugin-runner-task.test.ts` and
`environment-task-runtime.test.ts`: 153 passed, including a normal
resume with the capability (credential sent, variables added only with
`configuredEnvironmentVersion: 1`, delivered flag unset) and redaction
of both fields from metadata writes and sanitized errors.
- `native-session-executor.test.ts -t "uses a task-only"`: 5 passed.
- runner-e2e `source-sdk-evidence.test.ts` and
`source-sdk-public-type.test.ts`: 46 passed.
- `pnpm typecheck` in `server/` (builds shared and plugin-sdk) passed.
- Not run locally: the full suite and `pnpm verify`; required CI covers
them.

## Risks

- A provider restarting the Runner gets the claimant run's current
variables, which can differ from the ones the parked Runner started
with. That is intended.
- Rollback: revert this pull request. No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs), implemented by a sub-agent and reviewed
before opening.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:46:37 -08:00
Devin FoleyandPaperclip 12aa08721c feat(ssh): use the agent AI account in new environments (#21248)
Use the agent-selected AI account for new SSH environments. Preserve
remote-machine authentication for existing environments and expose it as
an advanced option. Keep managed credentials private while preserving
supported conversation state and credential refreshes across turns.

Validated with focused route, adapter, lifecycle, and real-process tests;
final CI passed on d634944df1.
Greptile Apex reviewed that revision at 5/5 with no unresolved threads.
External neutral security reports remain documented in the pull request.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-13 13:20:13 -08:00
DottaandClaude Fable 5.2 0199c963e2 feat(apps): ship the redesigned sign-in step for a pasted MCP server (#21469)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Apps lets an operator connect a remote MCP server by pasting its
address ("Connect your own MCP server")
> - Step 2 of that flow put every sign-in choice in an "Advanced
authentication" box with wrapped pill buttons and a hint paragraph
> - The box hid the choice behind a vague label, and its shape did not
match the access line ("Connects for everyone… Change") right below it
> - The owner approved a redesign: a stated sign-in default with a
Change link, and five options with their fields directly below each
option
> - This pull request wires that redesign into the real screen, removes
the old step, and updates copy that named the removed box
> - The benefit is a screen that reads as two stated answers and one
action, with the same default path (paste the address, press Check link)

## Linked Issues or Issue Description

No public issue exists. Enhancement:

**What existing behavior does this improve?**
The sign-in section of the "Connect your own MCP server" step in Apps.

**Subsystem affected**
UI: Apps connection setup (`ui/src/features/connections`). Small copy
changes in the server OAuth client registration message and docs.

**Current behavior**
Step 2 shows an "Advanced authentication" collapsible with five wrapped
pill buttons and a hint paragraph below them. The fields for the chosen
mode appear further down. Guidance banners tell the operator to look
"under Advanced authentication". The page also shows the "Unverified
server" badge twice (header and body). The stepper reads "Pick app · Add
your key", even when the server needs no key.

**Proposed behavior**
Step 2 states the sign-in choice in one line ("Paperclip asks the server
how to sign in.") with a Change link. Change shows five radio cards: Let
Paperclip check, Key or token, Browser sign-in, Custom headers, No
sign-in needed. The fields for the selected option are directly below
it. Done collapses the list to a one-line summary (for example "Signs in
with 1 custom header."). The list stays open while the chosen option
still needs a value. Client ID and secret are behind "Use your own
client ID" unless they have values or the server asked for them. The
stepper reads "Pick app · Connect".

**Reason and benefit**
The choice is visible as a stated default in the same shape as the
access line. Each option keeps its fields next to it. No copy points at
a box that does not exist.

**Breaking changes**
None. The five auth modes and their request payloads do not change.

**Additional context**
Related open PRs change server behavior for the same flow. They are not
duplicates. Refs #15192, Refs #15450.

## What Changed

- `ui/src/features/connections/GenericMcpConnectStep.tsx`: new step
component (same props as the old `LinkConnectStep`). It uses stable
field ids `generic-mcp-key`, `generic-mcp-client-id`,
`generic-mcp-client-secret` and a `data-testid` on the sign-in
Change/Done link.
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: render
`GenericMcpConnectStep`; delete `LinkConnectStep`,
`GENERIC_AUTH_MODE_OPTIONS`, `SegmentedOption` and unused imports;
change the pasted-link stepper label from "Add your key" to "Connect"
(`STEP_LABELS` is used only by the pasted-link flow; curated apps keep
their own `["Add your key"]`).
- `ui/src/pages/apps/generic-mcp-connect.ts`: `oauth_challenge` and
`oauth_manual_client_required` guidance now say "below" instead of
"under Advanced authentication".
- `server/src/services/tool-access/oauth/client-registration.ts`: the
manual-client message no longer names the removed box.
- `doc/connections/GENERIC-REMOTE-MCP.md`: describe the sign-in line,
Change, and the five options.
- Comments in `packages/shared` and `server/src/services/tool-access.ts`
reworded. `tests/runner-e2e/source-sdk-public-inputs.json` regenerated
for the two shared files (blob hashes only).
- `ui/storybook/stories/connection-setup/byo-mcp-sign-in.stories.tsx`:
14 stories for the shipped component; annotation is now `current` /
`production`.
- Tests: `AppsConnect.test.tsx` generic MCP block uses the sign-in
toggle and radio options; new tests for the collapsed summary, the
open-while-empty-key rule, and a single "Unverified server" label. New
`GenericMcpConnectStep.test.ts` unit-tests `genericMcpAuthSummary`.

Before: "Advanced authentication" box with pill buttons and a hint;
fields below the pills; two "Unverified server" badges; stepper "Pick
app · Add your key".
After: one sign-in line with Change; five radio cards with fields under
the selected card; one "Unverified server" badge in the header; stepper
"Pick app · Connect". Storybook: `byo-mcp-sign-in--choosing`,
`byo-mcp-sign-in--server-asked-for-key`.

## Verification

- `cd ui && pnpm exec vitest run src/pages/apps/AppsConnect.test.tsx
src/pages/apps/generic-mcp-connect.test.ts
src/features/connections/GenericMcpConnectStep.test.ts
src/features/connections/ConnectionSetupFlow.architecture.test.ts
src/features/connections/ConnectionSetupFlow.oauth.test.ts`: 5 files,
300 tests pass.
- `cd ui && node ../scripts/run-tsc.mjs -b`: pass.
- `pnpm -s typecheck` in `packages/shared` and `server`: pass.
- `pnpm check:token-gates`: all gates clean.
- `node scripts/storybook-catalog.mjs`: pass (generated indexes not
committed).
- `pnpm verify`: exit 2 (broader verification required, because
`doc/connections/GENERIC-REMOTE-MCP.md` is a global CI input). Required
CI supplies that coverage. The focused files above were rerun on the
final revision after the rebase.
- Headless Playwright against the Storybook dev server: the Default,
Choosing and Server-asked-for-key stories render the new step, the new
guidance copy, and the "Pick app · Connect" stepper.

## Risks

- Low risk. UI change on one screen; request payloads
(`genericConnectPayload`) do not change.
- Tests that click the first collapsed "Change" button now skip the
sign-in toggle, because the access line also has a "Change" button.
- The server message for `oauth_manual_client_required` changes text. No
code matches on the text; the client uses the error code.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless Playwright).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-13 14:35:49 -06:00
DottaandClaude Opus 5.5 64ec273845 perf(ci): run the typecheck lane's checks in parallel and drop its duplicate server type check (#21404)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the trusted PR workflow, and the `Typecheck
+ Release Registry` job is one of its required checks
> - That job takes about 9.2 minutes on the 4 vCPU / 16 GB fleet runner
(run 43344818387), against a target of 7 minutes or less for every job
> - The job runs six steps one after another, and the first step
type-checks the whole server program a second time, at about 13.5 GiB,
which makes the runner swap
> - Typecheck results are the check that authors most want back quickly,
so this job is on the critical path of most pull requests
> - This pull request removes the duplicate server type check and runs
the independent checks in parallel through one small script, inside the
same memory budget
> - The benefit is about 39% less wall time for the job's checks on the
same machine, with peak memory down from about 14 GiB (plus swap) to
about 8.5 GiB

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The `Typecheck + Release Registry` job in
`.github/workflows/pr-trusted.yml`. It takes about 9.2 minutes. The
target for every job in the workflow is 7 minutes or less.

**Subsystem affected**

CI (`.github/workflows/pr-trusted.yml`) and the typecheck scripts under
`scripts/` and `server/scripts/`.

**Current behavior**

After about 75 s of setup, the job runs six steps in sequence on a 4
vCPU runner, about 470 s in all (run 43344818387):
`typecheck:build-gaps` 121 s, test programs 85 s, server layers 128 s,
test projects 38 s, CLI guidance test 5 s, release registry tests 95 s.
Most of these steps use one thread. The `typecheck:build-gaps` step
builds the server with a full type check of the whole server program.
The server layer step then checks the same files again, and the `Build`
job checks them a third time. That full program peaks at about 13.5 GiB,
so the job peaks at about 14 GiB of 16 GiB and uses swap.

**Proposed behavior**

The job runs `node scripts/ci-typecheck-lane.mjs`. The script builds the
server without type checks (`PAPERCLIP_TSC_EMIT_ONLY=1`, as the Docker
image does) and still checks the runtime assets. It runs the steps that
write shared files first, one at a time. Then it runs the remaining
checks in parallel, one per CPU, with the server declaration layers
first, because the test layers and test projects read them. The checks
that are not on that path run at a lower CPU priority.

**Reason and benefit**

On the same runner, the job's checks take 61% of the time and about 60%
of the memory (see Verification). The job should take about 6 minutes on
the fleet instead of 9.2.

**Breaking changes**

None. The same files are type-checked with the same options. Local
commands do not change. `pnpm run typecheck:build-gaps` still works as
before.

## What Changed

- `scripts/ci-typecheck-lane.mjs` (new): runs the job's checks. It
writes shared outputs first, in sequence: links, plugin SDK, server
build without type checks, the runtime asset check, the sandbox provider
installs, and the Runner typecheck (it rebuilds the eval kernel, which
other checks read). Then it runs these checks in parallel: the server
declaration layers, each server test layer, each test project, the test
programs, the other build-gap workspaces, the CLI guidance test and the
release registry tests. A passing check prints its output in a collapsed
log group. A failing check prints its full output between `=== FAILED`
and `=== end of` lines, with an error annotation. The script ends with a
table of each check's start, duration and result, and exits non-zero if
any check fails.
- `server/scripts/typecheck-layers.mjs`: new `--layer <name>` option
that checks one layer against the configs and declarations of an earlier
`--declarations-only` run, and an exported `listLayers`.
- `scripts/typecheck-test-programs.mjs`: new `--prepare-only` and
`--skip-prepare` options, so the installs happen before the parallel
checks start.
- `scripts/run-typecheck-build-gaps.mjs`: new `--list-packages` option,
so the script still finds the build-gap workspaces and still fails when
one has no `typecheck` script.
- `.github/workflows/pr-trusted.yml`: the `typecheck_release_registry`
job runs the script instead of the six steps. No other job changes.
- Tests: new `scripts/__tests__/ci-typecheck-lane.test.mjs` (added to
`test:release-registry`), new cases in
`server/scripts/typecheck-layers.test.mjs` and
`scripts/typecheck-test-programs.test.mjs`, and the workflow wiring
assertions in `.github/scripts/tests/ci-plan.test.mjs` and
`scripts/__tests__/runner-release-crate-cache.test.mjs`.
- `tests/runner-e2e/source-sdk-public-inputs.json`: repinned, because
`package.json` changed.
- `doc/DEVELOPING.md`: new "Typecheck lane in CI" section, and the new
options.

## Verification

This PR's own CI ran the old job definition, because PR runs use the
workflow from master. So I measured the commands on a stand-in machine:
a temporary workflow (`.github/workflows/typecheck-lane-bench-temp.yml`,
now deleted) on GitHub-hosted `ubuntu-latest` (4 vCPU Intel Xeon
6975P-C, 15.7 GiB, no secrets, `contents: read`). It ran in runs
[43444356017](https://github.com/paperclipai/paperclip/actions/runs/43444356017),
[43445393958](https://github.com/paperclipai/paperclip/actions/runs/43445393958),
[43446786105](https://github.com/paperclipai/paperclip/actions/runs/43446786105),
[43448707837](https://github.com/paperclipai/paperclip/actions/runs/43448707837)
and
[43450786493](https://github.com/paperclipai/paperclip/actions/runs/43450786493).
Each step ran under `/usr/bin/time -v`, and a sampler read
`/proc/meminfo` every 0.25 s.

**Old steps, one at a time (3 runs, run 43445393958):**

| Step | Wall s | CPU s | Peak RSS of one process | System memory peak |
| --- | --- | --- | --- | --- |
| `typecheck:build-gaps` | 128–155 | 376–400 | 13.2–13.6 GiB | 14.2–14.7
GiB |
| ...of which the full server `tsc` | 40–51 | 146–164 | 12.8–13.8 GiB |
13.7–14.7 GiB, swap up to 738 MiB |
| ...replaced by the emit-only server build | 6–10 | 20–22 | 2.9 GiB |
3.9 GiB |
| Test programs | 67–87 | 147–170 | 1.9 GiB | 2.9 GiB |
| Server layers | 102–127 | 150–172 | 4.0–4.2 GiB | 5.4–5.6 GiB |
| Test projects | 31–37 | 49–59 | 2.1–2.2 GiB | 3.1–3.2 GiB |
| CLI guidance test | 2–4 | 2–3 | 0.2 GiB | 1.6 GiB |
| Release registry tests | 82–95 | 159–183 | 1.7 GiB | 3.0 GiB |

CPU seconds are about twice the wall seconds for the `tsc` steps even on
one checker thread. So each step uses about two of the four vCPUs.

**Old sequence and new lane on the same runner (4 runs, run
43450786493).** Each job copied one installed checkout, then ran both,
in alternating order:

| Run | Old steps, wall s | New lane, wall s | Ratio | Old memory peak |
New memory peak |
| --- | --- | --- | --- | --- | --- |
| 1 | 503 | 314 | 0.62 | 13.98 GiB | 8.33 GiB |
| 2 | 647 | 388 | 0.60 | 14.04 GiB, swap 637 MiB | 8.13 GiB |
| 3 | 627 | 385 | 0.61 | 13.98 GiB | 8.63 GiB |
| 4 | 382 | 237 | 0.62 | 14.10 GiB, swap 1.1 GiB | 7.99 GiB |

Hosted runners differ in speed by up to 1.7 times, so only the
same-runner ratio is meaningful. Both arms include a cold compile of the
Runner release crate (45–80 s) that the fleet restores from cache. The
fleet ran the old steps in 470 s; at the measured ratio, the new lane
should take about 290 s, and the job about 6 minutes with setup. The new
lane peaks at about 55% of memory, with no swap.

Local checks (on a loaded Mac, for correctness only):

- `node --test scripts/__tests__/ci-typecheck-lane.test.mjs` — 8 pass.
- `node --test .github/scripts/tests/*.test.mjs` — 807 pass.
- `node --test scripts/__tests__/e2e-shard.test.mjs
scripts/__tests__/release-verify-workflow.test.mjs
scripts/__tests__/runner-release-crate-cache.test.mjs
scripts/__tests__/run-typecheck-build-gaps.test.mjs` — pass.
- `pnpm exec vitest run scripts/typecheck-layers.test.mjs` in `server/`
— the new cases pass.

## Risks

- This changes a required job that every PR depends on, and its own CI
cannot prove it. Please watch the first `Typecheck + Release Registry`
runs on master after merge: the wall time (expect about 6 minutes), the
summary table at the end of the step, and any `terminated by` message.
- A type error now shows in that check's `=== FAILED` block, not in a
step of its own. Passing checks are folded, so the failure is the only
expanded block.
- When the server declaration layers fail, the test layers and test
projects are reported as skipped. Before, the test layers still ran
after a source error. Fix the source errors first; the next run checks
the tests.
- The Honeycomb CI time board and any step-name queries see one step,
`Typecheck, CLI guidance safety and release registry tests`, instead of
six. The script prints the duration of each check.
- The release registry tests run beside the type checks at normal
priority. Their slowest test took 46–78 s here, against a 180 s timeout,
and 114 s on the slowest stand-in runner.
- Low risk for memory: the new lane peaked at 8.0–8.6 GiB of 15.7 GiB,
far below the old 14 GiB.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code, with tool
use (shell, GitHub CLI, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:06:01 -06:00
DottaandClaude Opus 5.5 0ece025749 perf(ci): cut repeated CPU work in the Runner support suite (#21390)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Every pull request runs the "Runner support suite" CI job (`pnpm
test:e2e:runner:unit`)
> - That job took 8.1–10.6 minutes, depending on the spot instance type,
and was one of the last PR jobs to finish
> - The target for every PR job is 7 minutes or less
> - Measurement showed the suite is CPU-bound: each of the 693 Vitest
files loads `@playwright/test`, the runner catalog and shared modules
again in its own worker, and a few modules did quadratic or repeated
work when they loaded
> - This pull request removes that repeated work and reuses compiled
code across workers
> - The benefit is a job 75–86 s (13–16%) shorter on each instance type,
with the same files, tests and assertions. It does not reach 7 minutes;
the floor is explained below

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The duration of the "Runner support suite" PR job.

**Subsystem affected**
CI, the Runner E2E support suite under `tests/runner-e2e/`, and one
shared module (`packages/shared/src/connection-routing.ts`).

**Current behavior**
In 26 recent green PR runs the job took a median of 526 s on
`m7i-flex.xlarge` (17 runs), 549 s on `m7i.xlarge` (3 runs) and 604 s on
`m6a.xlarge` (6 runs). Vitest ran 693 files on 3 workers. The CPU was
about 82% busy. 52% of the tracked time was module evaluation.

**Proposed behavior**
The job is shorter on every instance type. The same files and tests
pass, and no assertion changes.

**Reason and benefit**
The PR workflow finishes sooner, so every pull request gets its result
sooner.

**Breaking changes**
None.

## What Changed

Measurements below are from the job's own runs on this PR. The spot
fleet picks `m7i-flex`, `m7i`, `m6a` or `m5a` (all 4 vCPU); speed
differs by up to 2x between them, so each comparison uses the same
instance type. This suite has no Vitest retries, so a flaky test fails
the job.

### Baseline (recent green PR runs on master, before this PR)

| Run (job id) | Job | Test step | Vitest wall | Vitest split |
| --- | --- | --- | --- | --- |
| 131933126599 | 523 s | 474 s | 469.8 s | import 52%, tests 43%,
transform 5% |
| 131932775003 | 510 s | 461 s | 457.1 s | import 51%, tests 44%,
transform 5% |
| 131926764360 | 527 s | 479 s | 475.2 s | import 51%, tests 43%,
transform 5% |
| 131924082813 | 541 s | 496 s | – | – |
| 131912527065 | 517 s | 468 s | – | – |

- One Vitest invocation (693 files, 3 workers, the Vitest default of
CPUs − 1), then 386 `node:test` tests in about 3 s. About 45–50 s of the
job is checkout and dependency restore before the test step.
- Summed tracked time was about 1,297 s for 470 s of wall time:
effective concurrency 2.76 of 3 workers. Module evaluation was 674 s:
4,164 modules evaluated 77,699 times, because each file runs in a fresh
isolated worker.
- A CPU sampler (temporary, removed) showed 82% average CPU use with 3
workers and 95–99% with 4 workers. With 4 workers and no other change
the wall time did not improve (455 s). The suite was CPU-bound, not
idle.
- Top modules by summed evaluation time (diagnostic run):
`@playwright/test` 158 s over 390 files, `tests/runner-e2e/catalog.ts`
56 s over 141 files, `packages/shared/src/connection-routing.ts` 40 s
over 317 files, `acpx/dist/runtime.js` 39 s.
- Slowest files: `chat-rendered-links` 43 s, `results-root` 43 s,
`chat-feature-flow` 43 s, `report` 40 s, `manual-private-transfer` 26 s,
`private-framework-report` 24 s, `chat-catalog-fingerprint` 24 s,
`hire-delegate-flow` 23 s, `source-sdk-evidence` 19 s, `unit-launcher`
16 s. No single file is near the 7-minute target, so there is no
longest-file floor.
- Slowest tests: the 256 MiB archive-cap test 23 s, the reload-once
reply wait 20 s (plus two at 10 s), the documented pnpm validation 16 s,
the retained name-only identity replay 14 s, the selected-case
fingerprint replay 13 s.

### Changes, one at a time, with what each one bought

1. **Compile cache.** `run-unit.mjs` sets `NODE_COMPILE_CACHE` to
`<tmpdir>/paperclip-runner-e2e-compile-cache` unless it is already set.
Vitest workers and the Node processes that tests start reuse V8 compiled
code. Node validates each entry against the source. `@playwright/test`
evaluation went from 158 s to 107–124 s, `acpx` from 39 s to 26–27 s,
and `zod` from 17 s to 9–10 s. Vitest went from 478 s to 441 s on a
comparable run (this run also included change 2).
2. **Reuse import scans in the chat catalog fingerprint.**
`fingerprintChatDependencyClosure` ran one esbuild parse for each of
about 2,300 files on every call. The parse result is now cached by file
path and content SHA-256 for the process. The fingerprint inputs and
digest are unchanged. `chat-feature-flow` went from 43 s to 12–14 s,
`chat-catalog-fingerprint` from 24 s to 11–13 s, and
`chat-campaign-identity` from 13 s to 8 s.
3. **Hash each suite definition once.** `buildRunnerMatrix` computed
`suiteDefinitionHash(suite)` for every execution, thousands of times,
every time a test file loaded the catalog. It now computes it once per
suite. The values are the same. `catalog.ts` evaluation went from 51 s
to 9.6 s. Vitest went from 441 s to 389 s.
4. **Fake timers for the reload-once reply waits.** These three tests in
`chat-rendered-links.test.ts` waited out the real 10 s render floor
twice. They now advance fake time with `vi.runAllTimersAsync()`. The
assertions on results, reload counts and attempt phases are unchanged.
The file went from 43 s to 2.8 s. This freed a worker slot; on its own
it did not shorten the CPU-bound run.
5. **Derive each aggregator identity once.** `connection-routing.ts`
called `aggregatorAppIdentity` (two regex replaces) for every index
entry for each of the 133 Arcade apps while it loaded, in 317 test files
and in the server. The identity of each entry is now computed once. I
compared the resulting `AGGREGATOR_SUPPORT_INDEX` (1,579 entries) and
`searchAggregatorServices` results with master: they are identical. The
SDK public input pins are refreshed for this change.
6. **Not shipped: four workers.** With the CPU work above removed, four
workers cut Vitest from 379 s to 341 s (job 418 s). In the next run,
`chat-installed-lane.test.ts` failed: the descendant of a timed-out
preparation child was still visible to `process.kill(pid, 0)` right
after cleanup. This is the process-tree-under-load risk, so the suite
keeps the Vitest default of 3 workers.

Docs: `tests/runner-e2e/README.md` describes the compile cache, how to
turn it off, why the worker count stays at the default, and the cost of
work at module load.

### Result

| Run | Job | Test step | Vitest wall | Effective concurrency |
| --- | --- | --- | --- | --- |
| Baseline median, `m7i-flex` (17 runs) | 526 s | 465 s | 462 s | 2.76 |
| Baseline median, `m7i` (3 runs) | 549 s | 476 s | 473 s | – |
| Baseline median, `m6a` (6 runs) | 604 s | 535 s | 531 s | – |
| This PR, `m7i-flex` (job 132007703795) | 440 s | 383 s | 379 s | 2.79
|
| This PR, `m7i` (job 132027384596) | 474 s | 424 s | 420 s | 2.78 |
| This PR, `m6a` (job 132033745288) | 526 s | 471 s | 466 s | 2.79 |
| This PR, `m5a` (job 132040621237; dependency cache miss) | 880 s | 768
s | 761 s | – |

The last three rows are the final configuration. The `m7i-flex` row is
the same code before a README-only change. Module evaluation fell from
77,699 to 63,135 evaluations; part of that drop came from master changes
merged into this branch, so I attribute time only to the per-change
measurements above.

## Verification

- CI on this PR is the experiment. Every push ran the real job on the
real runner. The table above lists each run.
- Local correctness checks, one process at a time:
`chat-rendered-links.test.ts` (58 passed), the six chat fingerprint test
files (37 passed), `catalog.test.ts` (92 passed), `packages/shared`
`connection-routing.test.ts` and `connection-search.test.ts` (21
passed), and `node tests/runner-e2e/run-unit.mjs
connection-auth.test.ts` (launcher path, 386 node:test passed).
- Equivalence check for the connection routing change: the index and
search results are byte-identical to master.
- Flakiness: 10 runs of this branch with 3 workers (including the three
final runs on `m7i`, `m6a` and the 2x slower `m5a`) passed with 0 failed
tests and 0 timeouts. No port conflicts, OOM kills or `ps` timeouts
appear in the logs. The suite has no retries. The 4-worker setting
failed once in 3 runs, so it is not used.

## Risks

- Low risk. No test is skipped or removed. No assertion is weakened.
- The worker count is unchanged, so the load on process, port and
temp-directory tests is lower than before (less CPU work per file).
- The compile cache directory lives in the OS temp directory. Node
rejects a stale or partial entry and compiles again.
`NODE_DISABLE_COMPILE_CACHE=1` turns it off.
- The connection routing change runs in the server too. It produces the
same index, and it makes loading the module cheaper.

### What sets the remaining floor

The job is about 45–55 s of checkout and dependency restore plus the
Vitest run. The Vitest run is CPU-bound on 4 vCPU (2 physical cores). A
fourth worker gives about 40 s more but fails a process-tree cleanup
test under load (`chat-installed-lane.test.ts`, which checks the
descendant immediately after cleanup). The largest remaining cost is
per-file isolation: every file loads `@playwright/test` (about 0.3 s CPU
each, 390 files), `zod` schemas from `packages/shared`, and `acpx`
again. Vitest estimates that `isolate: false` would save about 140 s
now. I did not use it: 116 files use `vi.mock`, and module state shared
between files would make failures depend on which files share a worker.
If more headroom is needed, the next step is a split across two machines
(out of scope here because it changes `.github/`). Two duration-balanced
halves (`vitest --shard=1/2` and `--shard=2/2`, about 190–235 s of
Vitest each on `m7i-flex`/`m6a`) would put each job at about 4–5 minutes
on every instance type except `m5a`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, GitHub CLI) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable (one test moved to
fake timers; behavior changes are equivalence-checked)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:05:05 -06:00
github-actions[bot]andlockfile-bot edb6945ff3 chore(lockfile): refresh pnpm lockfiles (#21439)
## Thinking Path

> - Paperclip uses a dependency lock for reproducible builds.
> - Dependency changes on master need matching lock entries.
> - Paid eval reports also pin a reviewed offline correction.
> - Reports must install before paid model work starts.
> - This automated PR keeps the lock and report proof together.

## Linked Issues or Issue Description

**What happened?**

Dependency manifests on master changed after the last committed lock
refresh.

**Expected behavior**

Frozen installs must match those manifests. Eval reports must retain
matching offline lock proof.

**Steps to reproduce**

Run a frozen install before and after this lock refresh.

## What Changed

- Refresh the dependency lock from the manifests on master.
- Refresh the sandbox provider lockfiles from their manifests on master.
- Include the matching offline report correction and hashes.
- Resync the SDK public source proof's lock pins from the refreshed
lock.

## Verification

- Pinned pnpm resolution completed with lifecycle scripts disabled.
- The report proof refresh accepted the reviewed committed baseline.
- The workflow rejected changes outside the lock and report proof files.

## Risks

The lockfile bot merges this PR without CI or review when every added
package version is at least 3 days old on npm. Otherwise the PR waits
for a maintainer, and the daily refresh tries again. The bot merges only
the exact commit this workflow generated from manifests already reviewed
on master. Unknown report lock baselines remain blocked. This update
changes no permissions or paid enable gates.

## Model Used

None — generated by pnpm and the deterministic report-lock updater.

## Checklist

- [x] Thinking path and problem description included
- [x] Generated lock and matching report proof included
- [x] Release-age gate decides between bot merge and maintainer review

Co-authored-by: lockfile-bot <lockfile-bot@users.noreply.github.com>
2026-11-13 17:39:38 +00:00
DottaandClaude Opus 5.5 e7e28ba241 feat(queue): show tasks waiting for an agent's slot as waiting, with their place in line (#21405)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each agent runs at most `runtimeConfig.heartbeat.maxConcurrentRuns`
tasks at once. Other runs wait in the agent's queue with status `queued`
> - On hosted staging, one agent had all 20 slots full and 295 queued
runs. Some tasks waited 3 to 7 hours and never started
> - The UI showed these tasks as live or working: a pulsing "Live" badge
in the task header and in task rows, a spinning "Working … for 4 hours"
in the task chat, and "N live" counts that included every queued run
> - People could not tell that work was waiting, why it waited, or what
to change
> - This pull request makes a deep queue an honest, visible state:
"Waiting for Codie · 20 slots max · 41 of 295 in line · Configure"
> - The benefit is that operators see the real state and the one setting
that changes it

## Linked Issues or Issue Description

Refs #21391, #21349, #21242

**What happened?**

A queued run that never started looked the same as a running run. The
task header, the task rows in lists and the inbox, the task chat and the
sidebar counts all showed queued runs as live or working.

**Expected behavior**

A run that waits for one of its agent's concurrency slots shows that it
waits. It names the agent, the agent's limit and the run's place in
line, and it links to where the limit is set. The place in line is the
order in which the server starts the agent's queued runs.

**Why does it matter?**

With a deep queue, the UI said "working" for hours while nothing ran.
Operators could not see the cause or the fix.

## What Changed

- Server: a new module
`server/src/services/heartbeat/queued-run-line.ts` now holds the facts
that the queued-run dispatcher orders by (`queuedRunDispatchFacts`). The
dispatch path in `queue.ts` and the new "place in line" read use this
one builder, so the line the UI shows is the dispatch order. The logic
moved without change, including the recovery-share fix from #21391.
- Server: `executionProjectionsForRuns` adds `agentQueue` (agent, limit,
running count, position, queued count) and the label "Waiting for
<agent>" with cause `agent_at_capacity` to a queued, never-started run
when its agent has no free slot. It reads one batch of agents and
running counts. Then it builds one line per full agent (at most 8 agents
per read), using only the three context keys that ordering needs. No new
polling.
- Server: the place in line uses `queuedRunDispatchLine`, an O(n log n)
replay of the dispatcher. The plain replay rescans the queue after each
step, which is quadratic (measured: 3.6 ms at 300 runs, 76 ms at 3,000,
600 ms at 10,000, all on the event loop). Dispatch itself is unchanged:
it stops after the free slots. Each agent's line is memoized for 5 s and
shared by concurrent reads. In a preview worktree, the line applies the
same creation-time cutoff as dispatch.
- Server: the task list summary adds `runningDescendantCount`
(descendants whose run started) next to `liveDescendantCount`.
`liveDescendantCount` still decides whether a blocked parent is covered.
The "n live below" chip uses the new count.
- Shared: the new `ExecutionAgentQueue` type and the optional field
`ExecutionProjection.agentQueue`.
- Server: `/companies/:companyId/live-runs` lists running runs before
queued runs, so a deep queue cannot push running work out of the 50-row
limit.
- UI: a new `AgentQueueWait` component shows "Waiting for Codie · 20
slots max · 41 of 295 in line · Configure", with no spinner and no
elapsed timer. Configure opens the agent's Advanced view, where the run
policy (`maxConcurrentRuns`) is edited. The task chat (both renderers),
the run ledger and the Storybook story use it.
- UI: the task header and the task rows (task list, inbox, legacy views)
show a quiet "Waiting" instead of the pulsing "Live" when every live run
of the task has not started. The full sentence shows on hover. "n live
below" counts only started sub-tasks. A waiting sub-task still covers
its blocked parent.
- UI: the sidebar "N live" counts (Dashboard, agents) count only started
runs.
- UI: the task chat pill now names other server holds on a queued run
(for example "Waiting for maintenance") instead of "Working".
- Docs: DESIGN.md records the waiting-for-slot hold.
- Storybook: `runs-agent-queue-wait` stories (full line, task chat, task
row, one slot, mobile), with play checks.

## Verification

- Red/green server test:
`server/src/services/execution-projection-agent-queue.test.ts` (embedded
Postgres). Before this change, the projection has no `agentQueue`. Now
it checks the agent name, limit, running count, positions 1 to 4 in
dispatch order (a person's retry first, then priority, not creation
order), no wait while a slot is free, and isolation from other
companies' runs. It also checks that the positions equal the line that
the dispatch builder produces.
- `pnpm exec vitest run` on the new test, `queued-run-dispatch-order`,
`recovery-wake-herd`, `heartbeat-queued-run-claim-isolation` and
`agent-live-run-routes` in `server/`.
- Runtime cost (rule `runtime-dependency-boundaries`): the assumption is
that the line stays cheap for a deep queue. `queued-run-line.test.ts`
checks that `queuedRunDispatchLine` equals the dispatcher's
claim-every-run order over 400 random queues (ages around the aging
boundary, recovery and person runs, blocked runs, deferred share runs,
all share counters). It also orders 10,000 runs in under 250 ms (the
quadratic replay takes about 600 ms). The DB test projects a 300-run
queue end to end with correct positions 1 to 300. Remaining limit: the
line is up to 5 s stale.
- Red/green: `issue-list-assignee-filter-routes.test.ts` now asserts
`runningDescendantCount: 0` for a parent whose only live descendant is
queued and never started.
- UI tests: `liveIssueIds.test.ts` (waiting tasks, started runs keep
"Live", no "running below" count for waiting sub-tasks) and
`TaskChatLiveRunPill.test.tsx` (waiting line with no spinner and the
Configure link, other holds named). The suites for the run ledger,
sidebar, sidebar agents, task detail, task columns and task chat also
pass.
- The Storybook stories render in headless Chromium and their play
checks pass.
- After deploy: open a queued task on staging and confirm the waiting
state.

## Risks

- Cost on read: a projection read that includes a queued run of a full
agent now reads that agent's queued runs (only small columns) and the
dispatch facts. This is at most one line per full agent every 5 s, for
at most 8 agents per read. Reads with no queued, never-started runs do
no more work.
- The place in line is the server's prediction of dispatch order. A
claim that is refused at start time (a gate, a blocker) can let the next
run start first. The position updates on the next refresh.
- The process-local "declined aged claim" hint that the dispatcher uses
is not visible to other servers, so the projection does not use it. This
can change only the aged-oldest share step.
- Configure opens the agent's Advanced view (the YAML run policy). The
agent's Runtime tab hides the run-policy fields today.
- Low risk otherwise: no schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, headless browser) and subagents for the UI survey.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 10:24:20 -06:00
DottaandClaude Opus 5.5 b7774a01cf fix: name activity actors and explain stopped tasks in plain words (#21406)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People must never see internal jargon or raw ids in chat replies or
task comments
> - Chat eval cohort 12 found three sources: activity records show the
board user only as the raw id `local-board`, agents turn internal field
names such as `checkoutRunId` into words like "no checkout", and a
recovery notice says "the original assignee is not invokable"
> - The recovery notice appeared on 170 fixture tasks. It is not a loop:
each task got it once, because recovery blocked every in-progress task
of an agent that a person had paused
> - This pull request names activity actors, lists these internal terms
in the chat guidance, rewords the notice, and leaves in-progress work of
a manually paused agent alone
> - The benefit is plain replies and no false "blocked" tasks after a
person pauses an agent

## Linked Issues or Issue Description

**What happened?**
- `activity.audit-change`: the agent read `GET
/api/issues/{id}/activity` and wrote that the change was made by "the
board user identity (`local-board`)". Activity rows carry only
`actorType` and `actorId`.
- `activity.company-overview`: the agent wrote "no checkout" to explain
why no one is working on a task.
- `tasks.organize-open-work`: 170 of 501 fixture tasks got one system
comment each: "Paperclip cannot safely continue automatic recovery
because the original assignee is not invokable...". The owners were
paused by a person, and the stranded-work sweep blocked each in-progress
task. The existing rule already keeps `todo` work of a manually paused
agent, because blocking it fights the pause and strands the task after
resume.

**Expected behavior**
Replies name actors by name ("Board", a member's name, an agent's name)
and describe states in plain words. A person's pause does not block the
agent's in-progress work. A stopped-task notice says what happened and
what to do, in plain words.

**Steps to reproduce**
1. Change a task's priority as the local board user, then ask the chat
agent who changed it.
2. Pause an agent that owns in-progress tasks with no active run, and
let the stranded-work sweep run.

## What Changed

- `server/src/services/activity.ts`, `server/src/routes/activity.ts`,
`packages/shared/src/types/activity.ts`: company and task activity reads
add `actorName` (agent name, member name, "Board" for `local-board`,
"Paperclip" for the system). Raw ids stay in their existing fields.
- `server/src/services/agent-conversations.ts`: the existing "no raw IDs
or internal codes" rule now names checkout, native run and local-board
as internal terms, and tells the agent to name an actor by `actorName`.
- `server/src/services/recovery/service.ts`: the notice is now "<agent>
can't run right now, so Paperclip stopped this task instead of retrying
it. It is still assigned to them. To continue, resume or fix the agent,
or assign the task to someone else." The manual-pause skip now covers
`in_progress` as well as `todo` work with no run.

## Verification

- `cd server && npx vitest run
src/__tests__/heartbeat-paused-assignee-new-task.test.ts`: 4 pass. With
the old recovery service the two new tests fail (`expected 'blocked' to
be 'in_progress'`, and the old notice text).
- `cd server && npx vitest run src/__tests__/activity-service.test.ts -t
"names each activity actor"`: passes (the method did not exist before).
- `cd server && npx vitest run
src/__tests__/agent-conversations.test.ts`: passes.
- `src/__tests__/activity-routes.test.ts` has three tests that time out
locally on unchanged master code too; CI runs them.
- Paid eval run of `activity.audit-change` and
`budgets-costs.spend-breakdown` on a branch with this change.

## Risks

- The chat prompt grows by about 40 words. This is a small part of the
existing directive, so time to first token does not change in a
measurable way.
- A budget- or error-paused agent still gets the notice, now in plain
words. Only a manual pause keeps in-progress work as is; the sweep picks
it up after resume.

## Model Used

- Claude, claude-opus-5-5, with tool use and code execution in Claude
Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 09:53:25 -06:00