Commit Graph
157 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 98ff5e002a feat(cloud): read-only durable-data eligibility for online enrollment (#22051)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted Paperclip servers can move from a server with a persistent
volume to a volume-free server, which allows deploys without downtime
> - The online move is allowed only when no durable data still lives on
the local disk; Core checks this in `readEphemeralServerDataEligibility`
> - Cloud can only learn the result by calling the preparation endpoint,
and that call starts a real move when the server is eligible
> - So Cloud cannot safely survey many servers, or watch a background
migration make progress
> - This pull request adds a signed, read-only GET that runs the same
check and returns the blocker names
> - The benefit is that operators can see which data blocks each server,
and move only the servers that are ready

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud can see a server's durable-data eligibility only through `POST
/api/instance/ephemeral-enrollment/prepare`. A refused call returns the
blockers. An accepted call starts the online move.

**Subsystem affected**
Server: Cloud control endpoints and ephemeral enrollment
(`server/src/routes/ephemeral-enrollment.ts`,
`server/src/services/ephemeral-enrollment.ts`,
`server/src/middleware/cloud-control.ts`).

**Current behavior**
There is no way to read eligibility without the side effect of
preparation.

**Proposed behavior**
`GET /api/instance/ephemeral-enrollment/eligibility` accepts only a
Cloud control assertion for the new action
`ephemeral-enrollment:eligibility`. It returns `{ eligible, blockers,
ephemeralServer }` and writes nothing.

**Reason and benefit**
A rollout to many servers needs to know which servers are ready, and
which data owners block the others, before it starts any move.

**Breaking changes**
None. This adds one endpoint and one action. Existing actions are
unchanged.

## What Changed

- `routes/ephemeral-enrollment.ts`: new `GET
/instance/ephemeral-enrollment/eligibility`. It requires a verified
Cloud control claim for `ephemeral-enrollment:eligibility` and sets
`Cache-Control: no-store`.
- `services/ephemeral-enrollment.ts`: new
`readOnlineEphemeralEnrollmentEligibility`. It uses the same check as
preparation (`purpose: "startup"`) and returns only the eligibility
flag, the canonical blocker names, and whether this server is already
volume-free.
- `middleware/cloud-control.ts` and
`services/cloud-runtime-identity.ts`: register the endpoint and the
action. Each assertion stays bound to one action, so a prepare token
cannot read eligibility, and the reverse.
- `__tests__/openapi-routes.test.ts`: list the route with the other
Cloud-control endpoints that are not in the board OpenAPI document.

## Verification

- `npx vitest run src/routes/ephemeral-enrollment.test.ts
src/__tests__/openapi-routes.test.ts
src/services/ephemeral-enrollment.test.ts
src/services/cloud-runtime-identity.test.ts`: 83/83 pass.
- New route tests:
  - a signed read returns the result and does not call preparation;
  - an unsigned caller and a tenant instance admin get 403;
  - a prepare assertion on this path gets 401.
- The three new tests fail without the route (red) and pass with it
(green).
- `tsc --noEmit` reports no errors in the touched files. A full local
typecheck needs the Runner build, so CI covers it.

## Risks

- Low risk. The endpoint is read-only and reachable only with a Cloud
control assertion bound to this stack and this action.
- The check runs a few indexed reads and pages skill installations.
Cloud calls it on request, not on a timer.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 13:02:37 -06:00
DottaandPaperclip cd656330fe fix(tool-gateway): preserve safe remote MCP error messages (#22025)
Preserve bounded, redacted and screened upstream JSON-RPC error messages for remote MCP tools and context reads. Audit blocked instructions, retain received error semantics, and redact URL/session and decoded credential echoes.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-17 12:01:23 -06:00
DottaandClaude Opus 5.5 3659a3a947 fix(auth): stop holding a signed-in user's requests on an unchanged Cloud tenant sync (#21997)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Cloud instance authenticates each signed-in request from trusted
Cloud headers in `resolveCloudTenantActor`
> - Each such request, static files included, waits for one shared
per-user sync that writes the user, company and membership rows
> - The sync takes row locks even when nothing changes. When another
open transaction holds the same user's rows, the sync waits for the
lock. Every request of that user then waits for the same sync, with no
bound
> - On a hosted instance this hung all requests of one signed-in user
for about 6.5 minutes. Other users, background work and requests without
identity headers stayed fast
> - This pull request makes a request wait for the sync only when the
sync can change the actor's authority
> - The benefit is that a lock held elsewhere no longer takes a
signed-in user's pages down

## Linked Issues or Issue Description

No public issue exists.

**What happened?**
For one signed-in Cloud user, every request hung for about 6.5 minutes
and then all of them finished together. This included API routes, the
HTML page, fonts, plugin UI files and `/api/health`. Requests without
the user's identity headers (hashed `/assets/*.js`) took 6 to 18 ms in
the same minutes. Background workers kept logging normally. The process,
the event loop and the database stayed healthy.

**Expected behavior**
A user whose membership is already in place authenticates from the
membership read. A lock held by another transaction on the user's rows
does not hold the user's requests.

**Steps to reproduce**
1. Resolve a Cloud tenant actor once, so that the user and the
membership exist.
2. In another transaction, take `FOR KEY SHARE` on the user row and `FOR
UPDATE` on the membership row, and keep the transaction open.
3. Resolve the same actor again from a new client (no process-local
write debounce).
4. Before this change, the call does not return until step 2 commits.

**Relevant logs or output**
Request logs showed a p50 of about 10 s (the UI aborts at 10 s) and
tails up to 252 s. Every hung request carried the same Cloud user
header. All of them completed together when the lock cleared.

## What Changed

- `resolveCloudTenantActor` reads the user's memberships first. When the
stored primary membership is active and has the requested role, the
request does not wait for the tenant write sync. The sync finishes in
the background, a failure is logged, and idle and drain checks count it
until it settles.
- A new user, a role change, a status change or a failed membership read
still waits for the sync, as before.
- New PostgreSQL regression test: another transaction holds the user's
rows, and an already-synced user still authenticates in under 3 s with
the same companies and role. The existing grant-repair assertion now
waits for the background repair.

## Verification

- Red before the fix: `npx vitest run
src/middleware/cloud-tenant-sync-postgres.test.ts -t "already-synced
user"` timed out (`expected 'timed_out' not to be 'timed_out'`).
- Green after the fix:
`src/middleware/cloud-tenant-sync-postgres.test.ts` and
`src/middleware/cloud-tenant-actor.test.ts` pass (50 tests).
- Related suites pass: `cloud-tenant-actor-statements`,
`cloud-tenant-company-provisioning`, `auth-session-route`,
`company-cloud-floor` (25 tests).
- `tsc --noEmit -p server/tsconfig.json` passes.

## Risks

- Low. Authority still comes from the database read on each request. The
change only skips waiting for writes that cannot change that authority.
- A missing default grant for an unchanged membership is now repaired
just after the request instead of before it. Until the repair lands, the
gap fails closed (denied access), as it did before the first repair.
- A sync that stays stuck now holds idle work open in the background
instead of holding requests. The instance does not sleep until it
settles.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Claude Code agent with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 07:33:48 -06:00
DottaandClaude Opus 5.5 8f1d39da79 perf(server): cache the instance settings row per process (#21768)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - One server process runs every agent on one Node main thread, and
each database statement costs main-thread time
> - The `default` instance settings row is read on hot paths: live run
event delivery, run log redaction, the stranded-task recovery check,
runner tool calls, adapter routes and run preparation
> - A staging measurement with 7 live runs showed the full-row settings
read at 10.24 statements per second, and 3.5 to 5.4 per second when idle
> - The row changes only when an admin or a boot step writes it, so
almost every read returns the same value
> - This pull request adds a per-process read-through cache for that
row, with in-process invalidation on every write and a 2 second age
bound for changes made by another process
> - The benefit is at most one settings read per 2 seconds per server
process, whatever the load

## Linked Issues or Issue Description

**Is your feature request related to a problem?**
The full `instance_settings` row read (`select id, singleton_key,
default_environment_id, general, experimental ...`) runs at 10.24
statements per second on staging with 7 live runs, and 3.5 to 5.4 per
second at idle. It scales with run events and recovery checks, not with
settings changes.

**Describe the solution you would like**
Cache the row per process. Drop the cached row when a settings write
commits, in this process and, through the existing work-signal
notification, in the other process during a deploy overlap. Bound the
age of a cached row to 2 seconds, so any other change (a direct database
edit, a second process outside an overlap) takes effect within 2
seconds.

**Describe alternatives you have considered**
A database trigger with LISTEN/NOTIFY would also catch direct edits, but
it needs a migration, and work notifications cross processes only during
a deploy overlap today. The existing signal plus a 2 second bound keeps
the change small.

## What Changed

- New `server/src/services/instance-settings-cache.ts`: a read-through
cache keyed by the database client. Concurrent misses share one read. A
cached row is at most `INSTANCE_SETTINGS_CACHE_TTL_MS` (2000 ms) old,
measured from the start of the read. Every caller gets its own copy.
Reads through a transaction handle skip the cache.
- `instanceSettingsService` `get`, `getGeneral` and `getExperimental`
use the cache. Write paths (`update`, `updateGeneral`,
`updateExperimental`, and the feedback vote save) still read the row
fresh.
- Every settings write publishes an `instance-settings-changed` work
signal through the existing work-signal channel
(`@paperclipai/db/work-signals`). Inside a transaction it is the first
signal of the write, so its `pg_notify` payload goes out before the
other wakes of the same commit (for example `status-card-evaluation`).
This process drops the cached row when the write commits. During a
deploy overlap, when work notifications cross processes, the other
process drops its cached row on receipt, before a worker woken by the
same write reads the settings. A reconnect of the notification listener
also drops the row.
- `fillUnsetInstanceDefault` writes through `update({ db: tx })`, so its
signal is sent when its transaction commits.
- A read that starts before an invalidation never stores its result, so
an old value can not come back after a write.
- The test setup clears the cache before each test, because many tests
write the row directly.

Hot callers of the cached row:

- Live run event delivery: `live-run-event-redaction.ts`
(`censorUsernameInLogs`), once per delivery batch.
- Run output and run lifecycle redaction: `heartbeat.ts`
`getCurrentUserRedactionOptions`, used by `run-output.ts`,
`run-lifecycle.ts`, `run-completion.ts`.
- The stranded-task recovery check: `recovery/service.ts` reads
`getExperimental().enableAgentChat` once per checked conversation issue.
- Runner tool calls: `paperclip-runner-tool-authority.ts`
(`enableIsolatedWorkspaces`, `enableExternalObjects`).
- Adapter routes: `app.ts` (`enableNativeRunner`, `enableOpenAiDot`) per
request.
- Run preparation: `heartbeat.ts` (`instance_settings` preparation
step), `queued-run-line.ts`, `queued-run-dispatch-liveness.ts`,
`run-control.ts`.
- Issue and comment paths in `issues.ts` (`censorUsernameInLogs`,
`enableAgentChat`).

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/instance-settings-cache.test.ts` (embedded PostgreSQL):
- 150 hot reads (`getGeneral`, `getExperimental`, `get` from new service
objects) after one warm read: 0 statements. Without the cache the same
test measures 150 statements (red run).
  - 20 concurrent misses: 1 statement.
- `updateExperimental` and `updateGeneral` are visible at once to a new
read in the same process.
- `fillUnsetInstanceDefault` (transaction write of the default
environment) is visible at once after it returns.
- A change written by a second client (another process) becomes visible
within 2000 ms plus a 500 ms test margin.
- A caller that changes the returned object does not change the cached
row.
- A read through a transaction handle sees the transaction's uncommitted
write, two such reads issue 2 statements, and the cache does not keep
the write after rollback.
  - A read that started before an invalidation is not stored.
- A received `instance-settings-changed` notification from another
process drops the cached row (next read issues 1 statement).
- `updateExperimental({ enableStatusCards: true })` sends
`instance-settings-changed` as its first notification, before
`status-card-evaluation`; `updateGeneral` sends it alone.
- The server tests that use instance settings or write the row directly
(230 files): see the run notes in the first comment.

## Risks

- A direct database edit, or a change from a second process outside a
deploy overlap, takes up to 2 seconds to apply in this process. Before
this change it applied at the next read. This includes access flags such
as `enableOwnerInstanceAdmin` (read per cloud-tenant request) and the
default environment.
- A change made through the settings service applies at once in this
process, and during a deploy overlap at once in the other process.
- A new code path that writes the `default` row must call
`announceInstanceSettingsChange`, or its change applies within 2
seconds.
- No schema change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes (no
user-facing setting or command changed)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-15 09:50:19 -06:00
DottaandClaude Opus 5.5 8adf0a463a perf(db): build the agent request authentication statements once per client (#21707)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - One Node process controls every run of a workspace. At 1,000
concurrent agents, main-thread CPU for each database statement is the
scarce resource.
> - A CPU profile of the hosted server shows the cost is per statement:
building SQL text in Drizzle costs about as much as parsing the result
in postgres.js.
> - Every authenticated agent API request (about 0.3 per agent per
second, plus tool calls) runs three fixed statements in
`actorMiddleware`: the agent row, the run authentication projection, and
the responsible user's company memberships. The server built the SQL
text of all three again for every request.
> - A local micro-benchmark shows that building the text is 15 to 40 µs
of CPU per statement for these shapes, and that a reused prepared query
costs the same as a raw driver call. Drizzle's `tracing.ts` wrapper is a
no-op (0.1 µs).
> - This pull request prepares the three statements once per database
client with placeholders, and executes them with the request's values.
The driver receives the same SQL text and parameters as before.
> - The benefit is about 40% less main-thread CPU for each authenticated
agent request in a local benchmark.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The CPU cost of agent JWT and agent key authentication on every agent
API request.

**Subsystem affected**
Server request authentication (`server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts`).

**Current behavior**
For every agent request, `actorMiddleware` builds three Drizzle queries
from scratch: `agents` by id, the `heartbeat_runs` authentication
projection (with `jsonb_build_object` and two `run_identity_contexts`
subqueries), and the `company_memberships` read with an `exists` on
`auth_users`. Building and preparing them costs 14, 38 and 40 µs of CPU
(median of 7 passes of 5,000 builds).

**Proposed behavior**
Build each statement one time for each database client, with
`sql.placeholder` values. Execute the prepared query with the request's
values. The SQL text, the parameters, and the returned rows stay the
same.

**Reason and benefit**
At 1,000 agents, agent API traffic is several hundred authenticated
requests per second on one main thread. Measured CPU per authenticated
request (local, embedded PostgreSQL, `process.cpuUsage()`, 2,000
requests per pass):

| Run (interleaved) | Before: median (min–max) µs | After: median
(min–max) µs | Load average |
| --- | --- | --- | --- |
| 1 | 436 (363–599) | 200 (183–224) | 10.8 / 10.1 |
| 2 | 489 (430–535) | 312 (212–392) | 11.6 / 13.4 |
| 3 | 461 (442–562) | 281 (247–409) | 13.1 / 12.9 |
| separate | 383 (322–810), 301 (268–476) | 169 (156–439), 193 (174–215)
| 9.5 / 9.0 |

The machine is shared (18 cores), so the spread is wide. Every pair
shows the after run lower. The drop is larger than the build cost alone,
because the request also allocates less (fewer builder objects to
collect).

**Breaking changes**
None. The statements, their parameters and the result shapes do not
change.

**Additional context**
Per-statement cost attribution from the new
`packages/db/scripts/benchmark-statement-cpu.ts` (CPU µs per statement,
median of 5 passes of 2,000, two runs, concurrency 1, load average
20–26):

| Statement | Drizzle builder | Build only (`toSQL`) | Drizzle
`.prepare()` reused | `unsafe(text, params)` | `unsafe(..., {prepare:
true})` |
| --- | --- | --- | --- | --- | --- |
| `issues` parent by id | 105 / 122 | 17 / 18 | 59 / 75 | 57 / 63 | 30 /
33 |
| `issues` full row by id | 277 / 326 | 136 / 153 | 96 / 81 | 97 / 77 |
48 / 33 |
| `company_memberships` by principal | 128 / 103 | 35 / 33 | 71 / 66 |
80 / 63 | 37 / 32 |
| join with an in-list of 10 | 169 / 173 | 57 / 57 | 83 / 88 | 109 / 80
| 42 / 40 |
| update by id | 110 / 106 | 25 / 26 | 75 / 80 | 76 / 73 | 39 / 40 |
| insert `activity_log` | 116 / 125 | 34 / 35 | 65 / 57 | 64 / 55 | 41 /
30 |

Drizzle's no-op tracer (`startActiveSpan`) measured 0.1 µs per call.

## What Changed

- `server/src/services/prepared-statements.ts`: add
`preparedStatement(prepare)`, the single-statement form of the existing
`preparedStatements` helper (one prepared query for each `Db`, held in a
`WeakMap`).
- `server/src/middleware/agent-run-authentication.ts`: split the query
into `agentRunAuthenticationQuery(db, binding)`;
`loadAgentRunAuthentication` executes the query prepared once with
placeholders.
- `server/src/middleware/auth.ts`: add `agentAuthenticationQuery` and
`responsibleUserMembershipsQuery`; the JWT path, the agent key path and
`loadResponsibleUserMemberships` execute their prepared forms.
- `server/src/__tests__/agent-auth-prepared-statements.test.ts` (new):
proves that each prepared statement sends the same SQL text and
parameters and returns strictly equal rows, and that repeated agent
requests do not build the statements again.
- `server/src/__tests__/agent-auth-middleware.test.ts`: the test
double's select chain now supports `.prepare().execute()`.
- `server/src/__tests__/prepared-statements.test.ts`: covers
`preparedStatement`.
- `packages/db/scripts/benchmark-statement-cpu.ts` (new): CPU per
statement for the builder, build only, a reused prepared query, `unsafe`
with and without `prepare: true`, and the no-op tracer. Run with `cd
packages/db && ITERATIONS=2000 PASSES=5 node --import tsx
scripts/benchmark-statement-cpu.ts` (`CONCURRENCY`, `ONLY`, `VARIANTS`
are optional).
- `scripts/benchmark-agent-request-auth-cpu.ts` (new): CPU per
authenticated agent JWT request through `actorMiddleware`. Run with
`pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-agent-request-auth-cpu.ts`.

Both benchmarks start their own embedded PostgreSQL in a temporary
directory.

## Verification

- Red/green: the "builds the request authentication statements once per
database client" test failed on `origin/master` (`select` called 12
times in 3 requests) and passes with this change.
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-auth-prepared-statements.test.ts
src/__tests__/prepared-statements.test.ts
src/__tests__/agent-auth-middleware.test.ts
src/__tests__/agent-run-authentication.test.ts
src/__tests__/agent-jwt-request-authentication.test.ts
src/__tests__/auth-session-route.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/local-agent-process-boundary.test.ts
src/__tests__/liveness.test.ts src/__tests__/inbox-mine-routes.test.ts
src/__tests__/board-bootstrap-routes.test.ts
src/__tests__/agent-auth-identity-transfer.test.ts
src/__tests__/prepared-statements-driver.test.ts
src/services/fresh-continuation-fallback.test.ts`: all pass.
- CPU before and after: the tables above.

## Risks

- Low risk. The prepared query uses Drizzle's own `PgPreparedQuery`, the
same execute path that the builder uses. The test pins the SQL text, the
parameters and the rows against the builder.
- Drizzle's `.prepare(name)` on postgres.js does not send a statement
name to the database (see #21201). With `DATABASE_PREPARED_STATEMENTS`
unset, the driver sends the same unnamed statement as before, so
connection poolers see no change.
- A test double for `Db` must now support `.prepare().execute()` on
these selects. One test double changed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, local test and benchmark runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 20:43:39 -06:00
DottaandClaude Opus 5.5 cf2a4221e3 perf(server): skip the per-request stale instance-admin purge when no row exists (#21680)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Cloud-hosted instances authenticate each request through trusted
Cloud headers in the actor middleware
> - That path deletes stale `instance_admin` rows for the user on every
request, static files and live-events upgrades included
> - On an idle hosted instance this delete ran about 1.2 times a second,
which is a write on an instance-wide table for each request
> - Almost always there is no row to delete, so the statement is wasted
work and a database round trip
> - This pull request finds out if a stale row exists inside the
membership read that the request already does, and deletes only when a
row exists
> - The benefit is one read and no write for each unchanged
authenticated request, with the same purge on the very next request when
a stale row appears

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud trusted-header authentication in the actor middleware.

**Subsystem affected**
Server authentication (`server/src/middleware/auth.ts`).

**Current behavior**
`resolveCloudTenantActor` in `server/src/middleware/auth.ts` sends
`delete from instance_user_roles where user_id = $1 and role =
'instance_admin'` for every trusted-header request. The actor middleware
runs before the static UI, so asset requests run it too. Each
live-events WebSocket upgrade runs it again through `resolveCloudActor`.
The retry wrapper runs it again after a transient connection error.

**Proposed behavior**
An unchanged request sends no write. A stale `instance_admin` row is
still removed on the next request after it appears.

**Reason and benefit**
Idle hosted instances send about one delete for each request. This costs
a round trip and a write statement on the main database for no change.

**Breaking changes**
None. The resolved actor and the authorization decisions do not change.

## What Changed

- The Cloud trusted-header path reads the user's active memberships
together with an `exists` check for an `instance_admin` row, in one
statement.
- The purge runs only when that check finds a row, when the user has no
active membership row yet (first request), or when the read fails.
- The session path and the Cloud path share one membership filter
function, so both still resolve the same access set.
- New test `server/src/__tests__/cloud-tenant-actor-statements.test.ts`
meters the statements of one authenticated request on the embedded test
database.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-tenant-actor-statements.test.ts
src/middleware/cloud-tenant-actor.test.ts
src/__tests__/cloud-tenant-actor-round-trips.test.ts`: 41 passed.
- Red/green: with the `origin/master` version of `auth.ts`, the new test
fails. An unchanged member request sends 2 statements, and one of them
is the delete. With this change it sends 1 read and 0 writes.
- The test inserts a stale `instance_admin` row between two requests.
The next request deletes it, and the request after that sends no write
again.
- The test changes the stack role from member to admin and back. The
membership role and the experimental-settings access change on the very
next request each time.

## Risks

- Low risk. The actor that the middleware returns is the same.
Authorization decisions do not change.
- The purge now runs after the membership read, not before the tenant
sync. The sync does not touch `instance_user_roles`, so the order does
not matter. The request still waits for the purge before it continues.
- A race between the read and a new row insert existed before too
(delete, then insert). The next request removes such a row.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-14 18:34:30 -06:00
DottaandPaperclip 7c0c3c4f31 fix(mcp): preserve Tasks sleep authority and re-enable recovery (#21603)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected apps can retain unfinished provider work through MCP
Tasks.
> - Managed defaults can enable Tasks without a stored experimental
flag.
> - The sleep check used the stored flag and could miss an active
provider task.
> - Enabling Tasks also failed to notify an owner that had become idle.
> - This pull request uses the effective setting and wakes the existing
owner after the setting commits.
> - Pending work keeps its recovery path. Completed history can become
quiet.

## Linked Issues or Issue Description

Refs #21263, which added durable provider Tasks. The current roadmap
covers the governed MCP Tool Gateway. This change repairs two existing
paths.

**What happened?**

The gateway could enable MCP Tasks through managed defaults while the
idle check reported no pending work. After an operator disabled and
enabled Tasks, an existing task could stay idle until a separate task
write woke its owner.

**Expected behavior**

The gateway and sleep check must use the same effective setting.
Enabling Tasks must notify the existing owner. It must retain the task's
original deadline and backoff.

**Steps to reproduce**

1. Retain a task with `next_poll_at`. Use managed Tasks defaults without
a stored true flag. Read the normal sleep projection.
2. Disable Tasks, retain the same task, then enable Tasks through the
settings service without writing the task again.

## What Changed

- Read the effective Tasks flag inside the same read-only sleep
transaction. Bind that value to the existing pending-task query.
- Register the existing Tasks queue notification before the settings
transaction writes. The normal commit protocol owns the wake.
- Add two real-PostgreSQL regressions to the existing nine-case suite.
Extend eight mock transactions with the read projection they now need.
Keep their assertions unchanged.
- Publish the Tailscale enrollment fixture state by rename so its CLI
watcher cannot parse a truncated JSON file. Existing enrollment
assertions and time limits remain unchanged.

## Verification

The two regressions fail against the preceding current-master products:
the pending-work projection reports `none`, and re-enable does not rearm
the owner within the original two-second oracle. After the fix, all nine
PostgreSQL cases pass at the recorded source, including completed
history, disabled Tasks, credential debt and fail-closed query errors.

At current `72bc32f`, the normal DB build and migrations, all nine
PostgreSQL cases, 155 affected mock cases, and the complete normal
Server typecheck/planner pass. The mock run includes three selected
safety families and six complete files; it explicitly excludes 196
existing durable-PG cases. All original case deadlines and assertions
remain. The startup file's earlier 133-case pass is limited to its
unchanged mock contract; a new hot-restart implementation is not
relabeled as tested. Root independently verified all current outputs and
process closure. The MCP product and regression files remain
byte-identical at `321c4b2ae2f3`; its additional change is the Tailscale
test fixture repair below. Exact current-head CI passes: 55 successful
checks, four explicit skips and one neutral result; Snyk succeeds. Fresh
Greptile is 5/5 for this exact head. The complete latest edited summary
and all paginated threads were inspected, with no open findings.

An earlier additional whole-eight-file run reached its 180-second
aggregate watchdog while the existing mixed PostgreSQL suite was still
running. It produced no individual failing assertion. Its startup file
passed 133 cases. The original failure remains recorded. The follow-up
selects the three affected mock families and runs the other affected
mock files whole; it does not claim that the interrupted PostgreSQL file
passed. No test assertion or case deadline changed. Full required CI
remains a merge gate.

The first remote run at `72bc32f` passed 51 checks but failed one
Tailscale enrollment case and an interrupted browser shard; two
aggregate gates then failed. The browser log records a runner shutdown
signal; its underlying cause is unknown, and one specific-job recovery
was requested. The Tailscale case failed its status CLI on both original
CI attempts. A local control then held publication until the original
file watcher observed an empty state: the CLI threw `Unexpected end of
JSON input`, and the existing enrollment assertion failed. Publishing
the state atomically passed the same control. All nine original
enrollment cases now pass in 4.393 seconds with the fixture repair. The
recorded remote interleaving remains unknown; the original logs and the
initial local non-reproduction are retained. No retry count, assertion,
timeout or product behavior was weakened. The later exact-head run
passes all required checks. A subsequent server shard was interrupted by
a runner shutdown and later recovered at the same source; the
interruption and unknown underlying cause remain recorded.

Use Node24/pnpm9, one worker, an owned external PostgreSQL test database
and the existing fixture/admin URL options. The two new regressions are
in `server/src/__tests__/idle-sleep-diagnostics-postgres.test.ts`.

These checks do not establish deployed suspension or a live provider's
availability.

## Risks

- The read adds the normal effective-settings projection to the existing
bounded snapshot. Query errors must still block sleep.
- A settings update now notifies the existing task owner. Transaction
rollback must not publish that notification.
- This adds no schema, provider protocol, timer or replacement task
framework. Existing active task polling remains required until its work
settles.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI `gpt-6.1-sol` implemented the regressions and advised the fix.
GPT-6-based Codex also reviewed and integrated it. The host does not
expose every supervising model ID or context-window size. The agents
used source editing, Git, code execution and regression tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (the focused scope above)
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (the
existing pending-task contract remains accurate)
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-14 05:00:50 -06:00
Devin FoleyandPaperclip 089748fbf9 fix(recovery): settle uncertain runner API receipts instead of fencing them forever (#21522)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs change Paperclip through the runner `call_api`
tool. Each mutation reserves a receipt before dispatch, so a crash
cannot replay a mutation that may have committed.
> - When the response of a mutation is lost, the receipt records an
unknown outcome. `priorRecoveryApiReceipt` then fences every later
mutation with the same operation id on that task, in every later run.
> - Nothing can settle that fence. The agent gets "Reconcile the
existing operation", but it has no tool to do that. The task stays stuck
for agents.
> - This pull request records the server's own final status for each run
mutation, and settles an unknown receipt from that record. A later
change by a board user on the task also lifts the fence.
> - The benefit is that a lost response no longer blocks a task's
mutations forever, and the fence still holds when the outcome is really
unknown.

## Linked Issues or Issue Description

**What happened?**
A routine run sent `PATCH /api/issues/{id}` to hand the task to a board
user. The server committed the change, but the server was slow and the
response took longer than the 30 second client timeout. The receipt
recorded `api_transport_failure` with `outcome: "unknown"`. Hours later,
the board user commented and reassigned the task to the agent. The new
run did the work, but both of its hand-off PATCH calls returned
`api_outcome_unknown` within 2 seconds and never reached the route.
Every later run on that task would get the same result.

**Expected behavior**
When the server has a record of the earlier call's result, the uncertain
receipt settles from that record. When a person has changed the task
after the uncertain run, the agent can act on the current state.

**Steps to reproduce**
1. In a native run, send a `call_api` mutation that commits on the
server, but let the client time out before the response.
2. Start a later run in the same retry or continuation ancestry.
3. Send any mutation with the same operation id. It returns
`api_outcome_unknown` and is not dispatched.

**Relevant logs or output**
```
call_api PATCH /api/issues/{id} -> { ok: false, status: null, error: "api_outcome_unknown", outcome: "unknown",
  guidance: "Reconcile the existing operation before repeating it. ..." }
```

No related public PR exists. I searched open and closed PRs for this
fence.

## What Changed

- `runner-api-client.ts`: `executeRunnerApi` sends the receipt key in
`X-Paperclip-Runner-Call` with mutations. Reads do not send it.
- `paperclip-runner-tool-authority.ts`: `#callApi` passes the receipt
key of the reserved mutation to the client.
- `middleware/runner-call-outcome.ts` (new): for an agent JWT request
with a run id and a valid call key, it records the final HTTP status as
a `runner.api_completed` activity row of the run. It records when the
handler ends the response, so it also records after the client has gone
away. It records one time per request.
- `recovery-effect-receipts.ts`: `priorRecoveryApiReceipt` settles an
unknown receipt:
- A recorded 2xx is an applied effect. An exact retry from the ancestry
gets an applied result and is not dispatched again. Other calls
continue.
- A recorded definite 4xx rejection (not 408) no longer fences, the same
as a completed 4xx receipt today.
  - A recorded 5xx, or no record, keeps the fence.
- A board user's `issue.updated` or `issue.comment_added` on the task
after the uncertain run finished lifts the fence of that run. Agent,
system, read and inbox activity do not.
- The fenced result now tells the agent to read the current state and
not to repeat a change that is already there.

## Verification

- `pnpm --filter @paperclipai/server run typecheck` passes.
- `npx vitest run src/middleware/runner-call-outcome.test.ts
src/services/task-retry.test.ts
src/services/native-runtime/runner-api.test.ts
src/services/native-runtime/runner-api.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/repeated-tool-failure-guard.test.ts
src/__tests__/teams-catalog-service.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/skill-import-discovery.test.ts` (in `server/`): 2418
passed, 2 skipped.
- `npx vitest run src/__tests__/runner-api-recovery.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts
src/services/native-runtime/confirmation-comment-api.integration.test.ts
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/runner-api-log-redaction.test.ts
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/native-case-tools.integration.test.ts
src/__tests__/connection-management.integration.test.ts`: 404 passed, 1
failed. The failure is `runner-api-artifacts.integration.test.ts >
preserves exact HTTP bytes in the immutable asset and downloaded helper
input` ("Artifact installation is unavailable to this run or provider").
It fails the same way with the unchanged `master` files in my
environment, so it is not from this change. CI covers it.
- Red before the fix: with the `master` version of
`recovery-effect-receipts.ts`, the new settlement and supersession cases
in `task-retry.test.ts` fail.
- New tests:
- `task-retry.test.ts`: settlement by recorded status 200, 422, 500, and
a record of a different call only; supersession by a later user, agent
or system change, including a change before the run finished and a read
marker.
- `runner-call-outcome.test.ts`: records the final status one time;
ignores reads, a missing or malformed key, agent API keys, board actors
and non-UUID run ids.
- `runner-api.test.ts`: the call key header is sent with mutations only.

## Risks

- Each run mutation that carries a call key adds one activity row
(`runner.api_completed`, entity `heartbeat_run`). The row has no request
body and no response body. Only the status and the receipt key hash.
- The settlement trusts the server's own final status. Routes commit
before they respond, so a 2xx means the effect is applied. A route that
answers 2xx before it commits its effect is still treated as applied.
- A run can write its own outcome record with its own token. This can
only settle that run's own receipts, so it does not give an agent new
access.
- Receipts made before this change have no record. They keep the fence
until a board user changes the task.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-13 16:45:51 -08:00
Devin FoleyandClaude Opus 5.5 2756be2f32 fix(models): answer an unreachable OpenRouter catalog with 503 instead of a crash report (#21468)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent setup lists the models that an OpenRouter key can select. The
server gets this list from the public OpenRouter catalog.
> - The catalog request has its own 10 second deadline. When the
deadline expired, fetch rejected with a raw `TimeoutError` (a
`DOMException`).
> - The error handler answered 500 "Internal server error" and sent a
crash report with no first-party stack frames. The user did not see the
existing retry guidance.
> - A slow or unavailable third-party catalog is not a fault in the
Paperclip server. It must not create a crash report.
> - This pull request maps catalog outages to a 503 response with the
existing retry message and a stable code. Only those errors skip crash
reporting.
> - The benefit: users see "Retry or enter a model ID manually", and
Sentry does not get noise from a provider outage.

## Linked Issues or Issue Description

**What happened?**

A staging server answered `GET
/api/companies/:companyId/adapters/opencode_local/models` with `500 —
The operation was aborted due to timeout`. At the same time, other
outbound requests from the host were slow. Sentry recorded
`TimeoutError: The operation was aborted due to timeout`. The event had
only Node internal frames (`AbortSignal.timeout`).

**Expected behavior**

The route returns a clear "catalog unavailable" response. The UI shows
the retry guidance. A third-party outage does not create a crash report.

**Steps to reproduce**

1. Make `https://openrouter.ai/api/v1/models` slow (no answer for more
than 10 seconds), or make it unreachable.
2. Request the OpenCode model list with `provider=openrouter`.
3. See a 500 response and a Sentry `TimeoutError` event.

**Paperclip version or commit**

`e23549cd9c631e4595a3213e03d0c960360452c6`

I searched for duplicate PRs. #17083, #17961 and #10290 change the
OpenRouter catalog contents or model validation. They do not change how
the server handles a catalog timeout.

## What Changed

- `server/src/services/openrouter-models.ts`: The catalog now maps three
failure types to `HttpError(503)` with code
`openrouter_catalog_unavailable` and the existing retry message: its own
deadline, a classified transport failure (uses
`classifyRemoteConnectionError`), and a 408, 429 or 5xx status. A
module-local `WeakSet` records only the errors that it creates for these
outages. `isOpenRouterCatalogUnavailable()` reads that record.
- Other non-OK statuses become a 502 with code
`openrouter_catalog_error`. These are still reported. Malformed JSON and
an invalid catalog shape are also still reported.
- `server/src/middleware/error-handler.ts`: The handler does not send a
crash report for an error from `isOpenRouterCatalogUnavailable()`. It
still answers 503 with the message and code. The new check is next to
the existing MCP connection-failure exclusion.
- Tests: unit tests for the deadline, DNS, 429 and 5xx cases, and for
the failures that stay reportable. A route-level test sends a stalled
catalog request through the real error handler. It checks for a 503
response and no crash report. It also checks that a different 503 is
still reported.

## Verification

- `vitest run server/src/services/openrouter-models.test.ts`: 7 tests
pass. I ran this file alone because the full workspace install did not
fit in the agent sandbox disk.
- Red check: with the original `openrouter-models.ts`, 5 of the 7 tests
fail. The deadline test receives `DOMException TimeoutError`, which is
the same error as the production event.
- `server/src/__tests__/openrouter-catalog-error-handler.test.ts` needs
the full workspace dependencies. CI runs it.

## Risks

- Low risk. Successful responses do not change. The cache and in-flight
deduplication do not change.
- The failure status changes from 500 to 503 (or 502) for the three
routes that read this catalog: adapter models, fast-response models and
prompt-injection-prevention models. The UI already shows
`error.message`, so users now see the retry guidance in place of
"Internal server error".
- The crash-report exclusion is narrow. It applies only to error objects
that this module created after an actual outage. A copied message or
code does not match.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in the Claude Code
agent harness with tool use (shell, file edits, Sentry, Railway and
GitHub connectors).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 13:00:24 -08:00
DottaandClaude Opus 5.5 c6b497d831 perf(auth): authenticate a run's agent JWT in three statements (#21434)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Each running agent calls the Paperclip API with a signed run token
(an agent JWT)
> - The server authenticates each of these requests before the route
handler starts
> - This authentication sent 13 statements to the database for each
request, and none of them were cached
> - One server process starts to struggle at about 50 concurrent agents,
and the target is 1,000
> - This pull request cuts the work before the route handler to 3
statements, with no transaction and no row locks in the usual case
> - The benefit is less database work and less event-loop work for each
agent request, with the same security checks on each request

## Linked Issues or Issue Description

Refs #16577 (trimmed saved run JSON from agent authentication)
Refs #17931 (scoped run identity reads to the needs of each operation)

**What existing behavior does this improve?**
The authentication of an API request that an agent makes with its run
JWT. Before the route handler starts, `actorMiddleware` did 13 database
statements in sequence: a board key lookup, an agent key lookup, a
`select *` of the agent, the run read, a 7-statement locked identity
capture (BEGIN, run task read, task row lock, run row lock and wide row
read, pending steering read, context read, COMMIT), and two reads for
the responsible user.

**Subsystem affected**
Server: `server/src/middleware/auth.ts`,
`server/src/middleware/agent-run-authentication.ts` and
`server/src/services/run-identity.ts`.

**Current behavior**
Each agent request costs 13 statements. The identity capture is a
transaction that holds a pooled connection and takes `FOR NO KEY UPDATE`
locks on the task row and the run row. These are the same rows that
event appends and steering lock, so agent requests wait behind them.

**Proposed behavior**
Each agent request costs 3 statements and no transaction: the agent
status columns, one run statement, and one membership statement. A run
with a pending steering context still uses the locked capture, so
steering acceptance does not change.

**Reason and benefit**
Item 1c of the thousand-concurrent-agents plan. The load model at 50
agents (API requests only) shows the database statements for this path
go from 165.7/s to 30.1/s, transactions from 15.1/s to 0, and event-loop
utilization from 0.74 to 0.25.

**Breaking changes**
None. The actor shape that routes get does not change. No route handler
changes. No schema change.

## What Changed

- **Skip key lookups for a verified run JWT.** The middleware verifies
the JWT first (HMAC only, no database). Every stored board key and agent
key is `pcp_` followed by letters, `_` and hex (`createBoardApiToken`,
`createToken`). A verified JWT has three parts with dots. So its SHA-256
cannot equal a stored key hash, and both lookups always returned no row
for it. A token that does not verify still tries the board key, then the
agent key, in the same order as before.
- **Read only the agent columns that authentication checks.**
`companyId`, `status` and `lifecycleHolds` replace `select *`. The agent
configuration JSON stays in the database. The agent-key path uses the
same columns.
- **Read revocation and identity in one statement.**
`loadAgentRunAuthentication` now also returns the run task ids, a
`pendingIdentityContext` flag (`exists` on a pending context of this
run), and the active context `id` and `responsibleUserId` (only when
that context belongs to this run). The new `resolveRequestRunIdentity`
in `run-identity.ts` uses these columns:
- The run is not running, or it has no active context: it keeps the run
row values, as before.
- A pending context exists: it calls `captureRunIdentity(...,
"request_authentication")`, as before. This keeps the locked
accept-or-refuse step and the 409 for an unacknowledged steer.
- Otherwise: it checks the run task ids with the same rule as
`lockIdentityTask` (now `assertRunTaskIdentity`), and returns the
context from the statement.
- **Reuse the run row for the legacy responsible user.** The legacy
lookup had the same filter as the run read. The middleware now uses the
`responsibleUserId` from that read, with the same non-UUID guard.
- **One statement for memberships.** The user existence check is an
`exists` condition in the membership statement, in place of a second
parallel statement.
- No in-process memo and no time-based cache. Every request reads the
current run row.

### Why the request-authentication read does not need the locks or the
transaction

The capture locks are released at COMMIT, before the route handler
starts. So they do not protect anything that the route does. Inside
authentication, they have two effects:

1. **They protect the acceptance write when a pending context exists.**
This PR keeps this case on the locked capture.
2. **They make the read wait for a steering transaction that is in
progress.** In PostgreSQL READ COMMITTED, one statement reads one
snapshot. So the run status, the cancellation fields, the active
context, its responsible user and the pending flag are all from the same
committed state. Each interleaving with a steering writer gives a result
that a serial order also gives:

| Writer state when the statement runs | What the statement sees |
Result | Same as a serial order? |
| --- | --- | --- | --- |
| Reservation (`reserveSteeredIdentity`) not committed | Old active
context, no pending | Old identity, 200 | Yes: request before the
reservation. The old identity stays active until the steer is accepted.
|
| Reservation committed | Pending flag | Locked capture: 409, or accept
if acknowledged | Same code path as before |
| Acceptance in progress (pending committed) | Pending flag | Locked
capture waits for the acceptance, then returns the new identity | Same
code path as before |
| Acceptance committed | New active context, no pending | New identity |
Yes: request after the acceptance |
| Cancel or supersede committed | `cancelled`,
`executionCancellation.state = requested`, or
`freshContinuationSupersededAt` | 403 on a write | Yes, on the very next
request |

`activate()` writes `activeIdentityContextId` and `responsibleUserId` in
one update, and the statement joins the context by id and run id. So a
mix of an old context with a new user is not possible. The tests show
the first and third rows on real PostgreSQL.

### Notes for reviewers (moved from code comments)

- The two correlated subqueries in `requestRunIdentityFields` name
`"heartbeat_runs"` and an alias for `run_identity_contexts` in raw SQL.
Drizzle writes the columns of a one-table select without the table name,
so a `${heartbeatRuns.id}` inside the subquery would resolve to the
inner table. The first test run showed this (the identity came back
empty), and the statement-count test catches it.
- The `exists` condition in `loadResponsibleUserMemberships` keeps the
old rule: a membership counts only while its user exists.
- The legacy responsible-user value comes from the run read, because
that read has the same filter (`id`, `company_id`, `agent_id`) as the
removed legacy lookup.

## Verification

- New `server/src/__tests__/agent-jwt-request-authentication.test.ts`
(embedded PostgreSQL, real `actorMiddleware`,
`meterDatabaseStatements`):
- Statement count: 3 statements, 0 transactions, 3 waits for a running
agent with an active identity and a responsible user. Before: 13
statements. The test also checks that fewer than 8 KiB come back, with a
120 KB agent configuration.
- Revocation: a write that passes is refused on the very next request
after `status = cancelled`, after a Stop request
(`executionCancellation.state = requested`), and after a
fresh-continuation supersede (`agent_run_superseded`).
- Identity change between two requests: the next request uses the new
context and the new responsible user.
- Uncommitted steering reservation: the request does not wait, and keeps
the committed identity. After commit, the next request gets 409. After
the acknowledgement, the next request gets the new identity.
- Acceptance in progress: the request waits for it (locked capture),
then returns the new identity.
- Wrong claims: an agent token signed for another company gets 401. A
token that names another agent's run gets no identity from that run. An
active context id of another run is not used.
  - Pending steering without an acknowledgement still gets 409.
- Red/green: on `origin/master` code, the statement-count test failed
with 13 statements, and the uncommitted-reservation test failed because
the request was blocked on the row lock. Both pass with this change.
- `agent-auth-identity-transfer.test.ts`: the heartbeat_runs read count
is now exactly 1 (it was a lower bound of 3).
- These pass: `agent-auth-middleware.test.ts`,
`agent-auth-identity-transfer.test.ts`,
`agent-run-authentication.test.ts`, `run-identity.test.ts`.
- `pnpm verify`: it selected 2,639 of 3,459 Vitest files, because
`auth.ts` is imported by most route tests. That is close to the full
suite, so I stopped it at 30 minutes on this busy machine (exit 124, no
failures before the stop). CI supplies the broad coverage. The focused
checks above pass on the final revision (41 tests).
- `pnpm check:module-boundaries`: passed. Server `tsc --noEmit`: passed.
- Load model (`scripts/benchmark-agent-scaling-load.ts` from the plan
branch, not committed), `AGENTS=50 SECONDS=20`, one run at a time. The
machine had a load average of 25 to 33 from other work, so the timings
are noisy. The statement and transaction counts are not affected by the
noise.

| `SOURCES=api` | Before | After |
| --- | --- | --- |
| Statements/s (15 requests/s) | 165.7 | 30.1 |
| Statements per request (this model has no responsible user) | 11 | 2 |
| Transactions/s | 15.1 | 0 |
| Event-loop utilization | 0.741 | 0.252 |
| Event-loop delay p99 | 267.8 ms | 19.0 ms |
| API lateness p50 / p99 | 1104 / 5631 ms | 6.7 / 79 ms |

| All sources | Before | After |
| --- | --- | --- |
| API requests done | 300/300 | 300/300 |
| API lateness p50 / p99 | 291 / 1130 ms | 230 / 2681 ms |
| Delta appends done | 14977/20000 | 8260/20000 |

The all-sources run is limited by delta appends, which this PR does not
change. Its results move a lot between runs on this busy machine (the
delta count is lower after, with no change to that path), so it does not
show an effect of this PR in either direction.

## Risks

- The usual path no longer waits for a steering reservation that is in
progress. The request then uses the identity that is committed and
active. This is the identity that the run acts under until the steer is
accepted. The table above shows that each interleaving matches a serial
order.
- `loadAgentRunAuthentication` returns more columns (two small
correlated subqueries on `run_identity_contexts`, by primary key and by
the `(run_id, ...)` index). Other callers get extra fields and ignore
them.
- The JWT-first order depends on the stored key format. If a new key
format can contain three dot-separated base64url parts that verify as a
JWT, this order must be checked again.
- Low risk for routes: the actor object is the same, and no route
handler changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), by Anthropic, in Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-13 12:38:01 -06:00
DottaandClaude Opus 5.5 0904a65d82 fix(server): local board boundary identifies agent processes on Linux it cannot inspect (#21209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the operator's own local processes get
implicit board access without credentials
> - #21181 added a local board boundary. It refuses implicit board
access to requests that come from an agent run's process tree
> - On Linux CI hosts, the boundary sometimes did not decide a loopback
request. It logged a warning and kept the earlier permissive behavior
for that request
> - The cause was not IPv6. IPv4 and IPv6 loopback lookups worked. The
gap was a connection that belonged to a same-user process that the
server cannot inspect
> - This pull request makes the Linux boundary decide those connections
from the process ancestry that the server can read, and refuse them when
that ancestry is not clearly the operator's
> - The benefit is that the local board boundary now holds on Linux for
IPv4 and IPv6 loopback, and the operator's browser and CLI keep working

## Linked Issues or Issue Description

Refs #21181

**What happened?**
On Linux CI hosts, the local board boundary sometimes could not
attribute a loopback request to a process. It then logged that it could
not identify the local process and kept the earlier permissive behavior
for that request. In one chat eval cohort, 37 of 191 server logs
contained this warning. The warning is logged once per server process.

**Expected behavior**
The boundary decides every uncredentialed loopback request on Linux. A
request from an agent run gets no implicit board access. Requests from
the operator's browser and CLI keep implicit board access.

**Actual behavior**
Some requests on Linux were not decided by the boundary, and they got
implicit board access.

## What Changed

- The Linux inspector now records which same-user processes it cannot
inspect. Before, it only recorded that one existed.
- The Linux TCP table lookup now also returns the socket's owner UID. It
covers `/proc/net/tcp` and `/proc/net/tcp6`, including IPv4-mapped
addresses.
- If no inspectable process owns the connection, the server reads the
TCP tables again. If the connection is gone, has changed, or belongs to
a different user, the request is refused.
- If the connection is still open and same-user, the server checks every
process it cannot inspect. Implicit board access stays only if each one
has a readable parent chain that does not belong to an agent run.
Otherwise the request is refused.
- The "cannot identify" fallback now applies only when the platform has
no inspector, or when Linux cannot read the TCP tables. The server keeps
logging a warning in that case.
- `doc/DEPLOYMENT-MODES.md` describes the new Linux behavior.

**How the operator is recognized**
- Browser and CLI processes are normal same-user processes. The server
reads their open sockets and environment, so their requests are
classified as before.
- A browser helper process that the server cannot inspect is still the
operator when its readable parent chain leads to the browser and has no
agent-run markers.
- The server's own requests and requests from the server's ancestors
stay the operator, as before.
- Connections owned by other users were refused before, and they are
still refused.

## Verification

- `pnpm --dir server exec vitest run
src/__tests__/local-agent-process-boundary.test.ts` passes on macOS (15
passed, 10 Linux-only skipped).
- The same file ran on Linux (Docker, `node:24-bookworm`, non-root user,
IPv6 loopback on). Result: 25 passed.
- Red check on Linux with the previous boundary and the new tests: the
agent-run cases for processes the server cannot inspect failed for both
`127.0.0.1` and `::1`. The operator cases passed.
- New Linux-only tests run real processes over `127.0.0.1` and `::1`:
  - an operator process keeps board access
  - an agent-run process is refused
  - an operator process the server cannot inspect keeps board access
  - an agent-run process the server cannot inspect is refused
- New platform-independent tests cover the classification rules and the
`/proc/net/tcp6` fixture parsing.
- A paid chat-feature run on Linux CI, with temporary diagnostic logging
on a separate branch, checks the server logs for the warning and for
refused operator requests. The results will be in a PR comment.

## Risks

- A rare operator tool that the server cannot inspect, and whose parent
chain the server cannot read, now loses implicit board access. Such a
tool can use a board API key.
- The extra TCP table read happens only when a connection has no
inspectable owner. Normal requests do no extra work.
- Platforms without process inspection keep the previous behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, Docker on Linux) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 16:52:06 -06:00
DottaandClaude Opus 5.5 f95bbe081e fix(auth): agent shells can only reach the local API with their own run credentials (#21181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In `local_trusted` mode, the server gives board access to local
requests that carry no credentials, so the operator can use the app
without a login
> - Agent runs execute on the same machine, so their tools can also send
local requests
> - An agent run must act only with its own run credentials and
permissions
> - This pull request makes implicit board access belong only to the
operator's own local processes
> - The benefit is that agent shells and tools can only reach the API
with their own run credentials

## Linked Issues or Issue Description

**What happened?**
In `local_trusted` mode, a process inside an agent run could receive the
board identity for an API request that had no credentials. A permission
denial for the agent then did not stop the change.

**Expected behavior**
A request from inside an agent run uses the run's own credentials and
permissions. Implicit board access stays available to the operator's
browser and terminal tools.

**Deployment mode**
`local_trusted` (private, loopback). Authenticated deployments do not
use implicit board access and do not change.

**Additional context**
Found by the chat feature evals. The bundled board skill also described
implicit board access in words that an agent could follow.

## What Changed

- Add `server/src/middleware/local-agent-process-boundary.ts`. For a
request without credentials, it identifies the local process behind the
TCP connection (Linux: `/proc`; macOS: `netstat -anv` and `ps`). If that
process or one of its ancestors belongs to an agent run, the request
gets no implicit board access. An agent run is a descendant of
`paperclip-runnerd`, or a process that carries a run identity variable
(`PAPERCLIP_RUN_ID`, `PAPERCLIP_AGENT_ID`, `PAPERCLIP_AGENT_KEY_ID`,
`PAPERCLIP_RUNNER_INSTANCE_ID`) with a value that is different from the
server's own value.
- The walk stops at the server process and its ancestors. Requests from
the server process itself, and from an operator whose own development
server runs inside the same agent execution, keep board access.
- The result is cached for each connection. Only one process lookup runs
at a time. Connections that arrive during a lookup wait for the next
lookup, which handles all of them together (one `netstat` and two `ps`
calls on macOS, one `/proc` walk on Linux). All lookups are
asynchronous, so they do not block other work in the server.
- `actorMiddleware` and the live events WebSocket upgrade apply the
check in `local_trusted` mode. A refused request is unauthenticated
(`actor.type = "none"`), the same as an anonymous request in
authenticated mode.
- Platforms where the server cannot identify the process (for example
Windows) keep the previous behavior and log one warning.
- Rewrite the `paperclip-board` skill authentication section. It no
longer describes implicit board access to agents, and it says that a 401
or 403 inside an agent run is final.
- Document the behavior in `doc/DEPLOYMENT-MODES.md`.

## Verification

- `cd server && npx vitest run
src/__tests__/local-agent-process-boundary.test.ts`. The test starts a
real `local_trusted` server and real child processes:
- an operator process without credentials gets board access (green
before and after);
  - an in-process request gets board access;
- a process with a run identity gets no board access (red before this
change, green after);
- an agent process that starts a tool with an empty environment gets no
board access (red before, green after);
- an operator inside the same execution as its own server keeps board
access.
- A burst test with a slow inspector: 12 parallel connections and 2
connections that close early. It checks that only one lookup is active
at a time, that the burst needs at most two lookups, that the event loop
stays responsive, and that later requests still work.
- Parser tests for the Linux TCP table and the macOS `netstat` format,
and a test that the runner daemon itself is not treated as an agent
process.
- Ran `agent-auth-middleware`, `agent-identity`, `live-events-ws`,
`live-events-toast`, `live-events-run-read` and
`issue-create-deduplication-routes` tests locally. `npx tsc --noEmit -p
server/tsconfig.json` passes.
- Paid runner E2E evidence for the affected chat cases is linked in a
comment.

## Risks

- Medium. The check runs once for each new local connection without
credentials. On macOS it starts `netstat` and `ps` (about 30 to 80 ms).
On Linux it reads `/proc`. Bearer requests do not run it.
- If a process can not be identified on a supported platform, the
request gets no implicit board access. On Linux, when a same-user
process hides its sockets, the check keeps the previous behavior so that
the operator's browser continues to work.
- This is a boundary for the documented path, not a sandbox. Agent runs
use the operator's OS account. A process that intentionally leaves its
run's process tree, or reads the instance files directly, is outside
this check. The broader option is a product decision: bind implicit
board access to a capability that only the operator's browser and CLI
hold, or run agents under an OS sandbox that blocks the server port.
This pull request does not change `local_trusted` semantics for the
operator.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 15:05:15 -06:00
DottaandClaude Opus 5.5 4363a3115a fix(recovery): continue any unfinished run in a fresh run from saved work (#21135)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs are interrupted when a server restarts, a deploy swaps
servers, a controller lease is lost, or a sandbox Runner cannot be
reached again
> - Exact recovery (resume the same session, replace, retry) refuses in
many ways, and each refusal leaves the task blocked or idle for a person
> - On staging, 577 of 1,000 runs in 11 hours failed or were
interrupted, across 25 error codes. About 900 earlier PRs each fixed one
refusal code, and the rate did not converge
> - This pull request adds one universal fallback: when an unfinished
run cannot be resumed exactly, for any reason that is not the person's,
the task continues in a fresh run from its last saved workspace and its
conversation
> - The benefit is that a new failure code next week falls back without
code changes, and tasks stop waiting on a person for infrastructure
faults

## Linked Issues or Issue Description

**What happened?**
After a server restart or deploy swap, many runs end with codes such as
`native_continuation_requires_reconciliation`
(`runner_remote_process_identity_unavailable`), `controller_lease_lost`,
`process_lost`, `server_shutdown_interrupted` and
`native_session_interrupted` (including `runner_harness_state_mismatch:
resumed_not_suspended` / `resumed_unavailable`). The disposition worker
then settles the recovery hold as "blocked, no replay". The task stays
Blocked until a person presses Retry or reconciles the run.

**Expected behavior**
A task waits on a person only for reasons that are the person's: an
approval, a question, a credential to share, a budget stop, a provider
usage limit, or a Stop they asked for. For every other failure, the
system starts a new run for the same task from the last verified saved
workspace, without a person.

**Steps to reproduce**
1. Start a task on a sandbox agent.
2. Restart or swap the server while the run is in progress, so the new
server cannot re-reach the Runner.
3. See that the run fails with
`native_continuation_requires_reconciliation` and the task stays Blocked
with no successor run.

**Paperclip version or commit**
`master` at 9e910ff9ae.

## What Changed

- New module `server/src/services/fresh-continuation-fallback.ts`. One
entry point, `ensureFreshContinuation` / `continueFreshInTransaction`,
takes a run that ended without finishing and guarantees one of:
something already follows it (a newer run, a linked replacement, an
operator decision), exact recovery is still under way (checked again
later), the cause is the person's, or one fresh continuation is
recorded.
- Classification, not an allowlist: `PERSON_OWNED_FAILURE_CODES` plus
task-level waits (pending approval, pending question, paused or
unavailable agent, paused company, review in progress, a Stop that a
person asked for, a Blocked status that another decision owns). Every
other code, including unknown future codes, falls back.
- Idempotent per source run. The task row lock serializes callers on
every server; the recorded `freshContinuation.sourceRunId` makes later
calls a no-op.
- Isolation, never waiting on proof. A local process still running on
this host is settled by its own owner first. If the old local writer is
proven gone, the task workspace is kept. Otherwise the continuation gets
a new isolated workspace (`fresh_isolated_workspace`), so two runs never
write one directory. The decision records the stop proof honestly
(`providerStopProof: isolated`).
- Claim admission no longer waits on the cleanup of a predecessor that a
fresh continuation replaced (`heartbeat/queue.ts`, predecessor loop).
Without this, the successor stayed queued behind the unreachable lease,
which is the staging failure itself.
- Only reconciliation holds that this decision owns are settled.
Workspace restore holds that workspace recovery still owns, and resolved
history, are untouched.
- `continuation_user_authorization_missing` is the system's failure (a
Retry's authorization was not carried forward), not the person's.
- Safety: the old run keeps its fencing. When it still owns its remote
workspace epoch, the epoch is abandoned
(`abandonUnverifiedRemoteWorkspaceEpoch`), so the old writer can never
publish and the next run restores the last committed save into a new
root on a new allocation. The unverified files stay in the abandoned
root, unapplied. The native coordinator records `replacementDenied:
fresh_continuation`, so no exact replay also starts.
- The replaced run loses its write authority. The fallback records
`freshContinuationSupersededAt` on the old run. The agent-JWT middleware
(`loadAgentRunAuthentication` reads one more field in its existing row
read) and `assertAgentRunWriteAllowed` refuse every write of the old
token with `agent_run_superseded`. Before this change, a failed or
interrupted run's token could still write. So an agent that is still
alive in an unreachable sandbox cannot act on the task next to its
continuation.
- The opening context (`recoveryOutcomes` in the execution continuation)
says that the previous attempt was cut off and lists the actions with
unverified outcomes, so the agent checks before it repeats a side
effect. No copy asks a person to inspect or repair a sandbox, workspace
or run. The hold text written at run end no longer says "Inspect the
original failure and reconcile".
- Delivery reuses the existing path: the settled hold gets an
`executionReconciliation` decision and `continuationDelivery: pending`,
and `deliverReconciledExecutions` starts the run (fresh session,
`retryOfRunId` = source).
- Budget per task, not per restart: at most 6 fresh continuations per
task per rolling hour (the server-restart lane's cap). At the cap the
hold stays with the system and is checked again when the window has
room. The failure allowance is never charged.
- Call sites:
- `settleUnrecoverableExecutions` (execution-recovery-resolution.ts):
every hold whose exact recovery is ruled out now goes through the
fallback before it would block the task. Interrupted preparations keep
their own replay lane first.
- `continueHeldExecutions` (same worker): picks up holds saved by older
releases (settled-but-blocked no-replay holds, finished or exhausted
saved-work recoveries) for the task's latest unfinished run.
- Wake admission (heartbeat/queue.ts): an ordinary system continuation
wake (sweep, monitor, timer) that a hold would defer now calls the
fallback in the same transaction and becomes the fresh continuation
itself. This is the path the idle-task sweep uses. A person's Retry,
message or queued send keeps its exact admission; the disposition worker
settles the hold, and the saved input then continues.
- `executionControlSchedule.disposition` wakes the worker for held tasks
and for the next due recheck.

### Human gate versus system uncertainty

One documented rule (`isHumanGate` plus `personOwnedTaskWait`) decides
what the fallback never touches. The rule goes by cause, never by owner
type:
- Human gates: a pending approval, a pending question, a user-owned
hold, an owner-gone attestation, a recorded explicit continuation, an
operator reconciliation, a person's Stop.
- System uncertainty: a no-replay record ("outcome unknown after an
interruption"), a hold that is `board`-owned only by default, an
unverifiable controller process space or host.

For uncertainty the fallback starts a fresh run, never a replay. The
interrupted turn is not re-sent. No tool call or write is re-executed.
Governed actions go through their approval gate again. The earlier
`automaticRecovery` decision is kept on the hold as
`automaticRecoveryHistory`, beside the attributed
`freshContinuationSettlement` and an activity entry.

### Restart recovery refused for an unverifiable controller

The case is `controller_process_space_unverified`, its remote variant,
or `controller_host_unrecorded` / `controller_host_unprovable`. Its
lease has lapsed; a live, renewing lease is still waited for (server
overlap). After `NATIVE_NAMESPACE_UNVERIFIED_WAIT_MS` (2 minutes) the
run ends (failed, `native_restart_recovery_blocked`). That frees the
task's execution lock and the agent's slot, and a system-owned hold lets
the fallback continue the task. Recording the stops is optional. The "a
board operator must record both stops" text is replaced.

Board/operator attention sites in native restart recovery:
- `surfaceNativeControllerNamespaceAttention`: copy replaced; the run
now ends after the bounded wait.
- `requestNativeControllerStopConfirmation`
(native-controller-host-recovery.ts): copy says confirmation is
optional; the same bounded end applies.
- The blocked restart-recovery hold (`nextAction` "Inspect the preserved
checkpoint and reconcile…"): copy replaced; the fallback settles the
hold by cause.
- `owner-gone-attestation.ts` (generic resource records from older
releases): unchanged. It is not a run-recovery path, and it is listed as
a follow-up.

### Every wait and its bound

| Wait | Owner that rechecks | Worst-case total before the fresh
continuation |
| --- | --- | --- |
| `exact_recovery_pending` (execution settling, native finalizer,
controller lease, interrupted-preparation lane, saved-work recovery,
restore hold, retained-source fallback, live local process) |
Disposition worker, woken at each deadline; rechecks at 1, 2, 3 and 4
minutes, then every 4 | About 10 minutes from the first pending check.
Up to 15 while the owner shows progress (a new event, output, lease
renewal or coordinator update since the last check). |
| Saved-work (workspace) recovery that is `scheduled`,
`awaiting_evidence`, or `running` without progress | Held-run scan, now
selected and scheduled like the other holds | About 10 minutes; 15 at
most |
| Saved-work recovery that is `running` and advancing (revision,
attempt, state or hold update since the last check) | Held-run scan |
Its own 30-minute budget |
| Unverifiable lapsed controller in restart recovery | Restart recovery
(`retryAt`) | 2 minutes, then the run ends and the fallback continues it
within one disposition pass |
| Hourly cap (`backoff`) | Disposition worker, woken at the deadline |
Until the rolling hour frees a slot, at most 1 hour. This is the
deliberate restart-lane bound, not a stuck state. |
| Person-owned or dependency wait | The person's own action wakes the
task; rechecked hourly | Not bounded: it is the person's |

Restart-recovery classifications (`claimNativeRestartRecoveries`, which
`recoverNativeRunsAfterRestart` calls):

| Classification | Who rechecks, when | Bound |
| --- | --- | --- |
| Claims: `reattach_existing_runner`, `reattach_remote_runner`,
`resume_dead_runner`, `bootstrap_incomplete` | The claiming server
resumes exactly | Exact resume; no fallback |
| `already_finalized` | Nobody; it is terminal | None needed |
| `blocked` | The run is terminal and its hold goes to the fallback
(disposition worker) | Fallback bounds above |
| `awaiting_evidence` / `current_controller_preparing` | This process's
own executor | Ends with that executor |
| `awaiting_evidence` / `recovery_rows_busy` | Restart dispatch after 1
second | Self-resolving: the row lock clears |
| `awaiting_evidence` / a live lease (`controller_lease_held`,
`controller_lease_grace`, `controller_still_alive`) | Restart dispatch
at the lease-renew interval | Never ended while the old controller
renews its lease |
| `awaiting_evidence` / unverifiable lapsed controller (process space or
host) | Restart dispatch every 2 minutes | Ends 2 minutes after the
lease lapsed |
| `awaiting_evidence` / any other reason (startup cleanup, finalizer or
cleanup owner, recovery failure, routine busy, future reasons) | Restart
dispatch at the owner's `retryAt`, or within 2 minutes when it has none
| Ends 2 minutes after the lease lapsed, or after 10 minutes, whichever
comes first. It never ends while a lease is live. |

Ending a run sets it failed (`native_restart_recovery_blocked`),
releases the task lock and the agent's slot, and records a system-owned
hold. The fallback then continues the task once, isolated.

No wait needs longer. Saved-work recovery and retained-source checks
keep their own attempt budgets. After 10 to 15 minutes the fallback
continues anyway in a new root, without the unverified files, which stay
recorded and unapplied.

### When the fallback preempts saved-work recovery

The unverified files are not destroyed. They stay where recovery found
them: in the abandoned remote epoch's root, or in the old local
workspace that the task is no longer bound to. They are recorded and
never applied. Settling the hold records the decision on it, so the
retained-workspace worker stops scheduling it (its query requires no
`executionReconciliation` unless recovery already succeeded). If an
attempt already in flight finishes later, it cannot publish into the
fresh run's workspace. For a remote workspace, the epoch it targets is
abandoned and the writer is cleared. For a local one, the task now
points at a new workspace. Whenever a wait is preempted, the
continuation is isolated: a fenced remote epoch or a new local
workspace.

### What people see

The task's interruption presentation shows "working" while the primary
fresh-continuation record is pending and has not started (coalesced
holds never count, and they follow the primary's final state), and a
timed wait with no Retry while a check is scheduled. The Retry banner
appears only when the system has given up or the cause is the person's.
Holds settled together with the primary record are `coalesced`; they no
longer show "closed or reassigned". A pending continuation is never a
second run: if a person's Retry or saved input started a run after it
was recorded, delivery links to that run instead. A task that finished
before delivery gets no run.

### Readers of what the fallback writes

- Recheck deadline (`freshContinuationCheck`):
- `unresolvedExecutionDispositions` and
`heldExecutionsAwaitingFreshContinuation` read it to skip work that is
not due.
- `nextFreshContinuationCheckAt` and
`executionControlSchedule.disposition` read it to schedule the worker.
- Tests: the cap deadline is scheduled, and a cancelled task goes quiet.
- Isolation decision:
- The task's workspace binding is cleared. Every path that starts the
next run (system continuation, saved input, Retry, sweep wake) resolves
the workspace from that binding at run start (heartbeat.ts
`issueContext.executionWorkspaceId`).
- `freshContinuationIsolation` in the run context is read by
`readFreshWorkspaceReadiness` for the shared-directory case.
- `freshContinuationSupersededAt` on the old run is read by the auth
middleware, `assertAgentRunWriteAllowed` and the claim gate.

## Verification

- `pnpm vitest run src/services/fresh-continuation-fallback.test.ts`
(server): every test passes (it covers the cases below).
- The continuation starts (leaves the queue) while the unreachable old
lease is `active` or `pending_cleanup`. This was red before the
claim-admission fix.
- Isolation: a writer on another host gets a new isolated workspace; a
process alive on this host waits for its owner; a proven-gone writer
keeps the workspace.
  - Owned holds only: a restore hold and resolved history are untouched.
  - The recheck schedule goes quiet when the task is cancelled.
- `continuation_user_authorization_missing` continues under the task's
responsible user.
- One test per top-five staging code (native and legacy hold shapes):
the task gets exactly one fresh successor run from saved state.
- `runner_harness_state_mismatch` (`resumed_not_suspended`,
`resumed_unavailable`) falls back.
  - An unknown code falls back.
- Two concurrent callers plus the disposition worker create one
successor.
- Old-saved/new-resumed: a settled-but-blocked hold from the previous
release, with its coordinator in `awaiting_runner_reattach`, continues
on the first pass.
- An ordinary system continuation wake is admitted as the fresh
continuation.
- Fencing: after the continuation is recorded, the old run's token is
refused for writes (`agent_run_superseded`) and the continuation's token
is not. The old run's remote workspace epoch is abandoned: the writer is
cleared, the location and provider root are cleared, and the epoch
advances. So the next preparation restores the last save into a new root
(`prepareRemoteWorkspace`: `sameAllocation` is false when the location
is null, server/src/services/remote-workspaces.ts:437-470).
- The opening context names the cut-off run and an unverified tool
action.
- Person-owned causes (pending approval, credential not shared, budget
stop) are not continued.
  - The hourly cap backs off without blocking, then continues.
- Red/green: with the wiring reverted to `master`, 9 of these tests fail
(the task stays Blocked with no successor).
- Affected suites pass locally: `execution-recovery-dead-controller`,
`completed-turn-cleanup-hold-settle`, `execution-control-work`,
`heartbeat-process-recovery`, `issue-recovery-actions`,
`native-safe-replacement`, `native-failed-workspace-recovery`,
`task-retry`, `runner-lost-mid-turn-continuation`,
`workspace-recovery-fallback`, `explicit-native-continuation`,
wake-queue `postgres`, `agent-conversations`, `native-startup-retry`,
`governed-wait-runner-loss`, `native-preclaim-continuation`.
- `pnpm verify --explain` selects the full suite (the wake queue is
shared), so CI supplies full coverage.

Existing expectations changed because they asserted the "wait for a
person" behavior this PR replaces:
- `issue-monitor-scheduler.test.ts`: "explains held monitor dispatch
without replaying uncertain execution" is now "continues a held task in
a fresh run when its monitor is due". A due monitor on a task held by a
`process_lost` hold now starts one fresh run instead of deferring.
- `agent-run-authentication.test.ts`: the auth row also reads
`freshContinuationSupersededAt` (still under 512 bytes), and a
superseded run is refused.
- `assigned-work-invariant.test.ts` (#21139): "with the real wake queue,
a hold that defers the continuation leaves a violation…" is now "…a hold
reached by the sweep's wake becomes one pending fresh continuation". The
sweep's wake now yields one pending fresh continuation, and the next
sweep adds no attempt.
- `heartbeat-native-restart-suspension.test.ts` and
`native-runner-restart-recovery.integration.test.ts`: awaiting-evidence
dispositions now carry a `retryAt`, because every wait is rechecked. A
lapsed unverifiable controller no longer waits forever.
- `native-remote-retry-admission.test.ts`: the blocked-hold text is now
"This run stopped before it finished. Your saved work is kept." The old
text asked people to reconcile.

Holdless legacy runs: in the 1,000 staging runs, 0 tasks end on
`controller_lease_lost` or `process_lost` (every one has a later run).
12 end on `server_shutdown_interrupted` with no retry scheduled (attempt
0). On staging now, 9 of those 12 sit `in_progress` with no run and no
hold, 1 is done and 2 have a live run. Those 9 belong to the idle-task
sweep (#21139).

## Risks

- Speed: nothing is added to the normal path before `turn.start`. The
fallback runs only on the disposition worker after exact recovery is
ruled out, or at wake admission when a hold already defers the wake. The
auth middleware reads one more JSON field in the row read it already
does.

- Behavior shift: tasks that today stay Blocked for a person after an
infrastructure failure now continue automatically. A side effect with an
unknown outcome is not replayed; the new run is told about it and must
check first.
- After deploy, the disposition worker continues tasks already blocked
by these codes, at most 25 per pass and at most 6 per task per hour. No
migration. No manual Retry is needed.
- The old executor is not proven stopped before the successor starts.
Fencing is: its API writes are refused, its workspace epoch is
abandoned, the native `replacementDenied` marker is set, and the
successor has a separate run identity.

Known gaps (follow-ups):
- No quarantine attachment: unverified files stay unapplied in the
abandoned workspace root (recorded on the workspace in
`remoteWorkspaceAbandonedEpochs`). They are not read into a task
attachment, because the sandbox is usually unreachable in these
failures.
- Agent Chat conversations are excluded at wake admission. They keep
their owner-bound continuation.
- Under a `reuse_by_environment` lease policy the new allocation can be
on the same provider sandbox. It always uses a new root and a new epoch,
and the old run cannot publish or write.
- Tasks left `in_progress` with no run and no hold (for example after
`server_shutdown_interrupted` with no retry scheduled) belong to the
idle-task sweep. That sweep's wake reaches this fallback when a hold
exists.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-12 13:34:34 -06:00
DottaandPaperclip 12f70d0d4c Keep Slack form custody and receipt transfer durable (#19825)
## Thinking Path

> - Paperclip manages work for AI-agent companies.
> - Chat connections deliver provider messages and user answers.
> - Slack requires a response within its original three-second deadline.
> - Forms and their delivery receipts must survive retries and tenant
restarts.
> - This change uses the existing connector queue and durable form
owners.
> - Current authorization, exact receipt checks and normal ACK rules
stay in force.
> - The tenant can settle useful work while keeping the connection
enabled.

## Linked Issues or Issue Description

Refs #19019 and #19348. Companion Cloud PR:
paperclipai/paperclip-cloud#1250.

## What Changed

- Preserve form grants, sessions, encrypted evidence, terminal receipts
and company-authorized Retry.
- Check initial form enrollment before the provider send. Keep a typed
preparation wait only while an exact current enrollment obligation
exists.
- Refund the attempt only when the claimed attempt and credential lease
still match and provider invocation has not started. Possible or
accepted sends retain uncertain-delivery custody. An ordinary error with
the same text still uses the normal five-attempt limit.
- Notify the existing publication worker when normal enrollment commits.
Reconstruct pending work from durable state after a lost reply or owner
restart.
- Page settled history before proving idle safety. Keep unfinished,
malformed and foreign receipts visible beyond the first 1,000 rows.
- Keep Slack callback, answer, receipt and ACK rules in their provider
owners. Support body-bound Cloud idle capabilities when a complete
catalog exceeds the header budget.
- Add focused regression coverage, an enabled quiet/reactivation recipe
and explicit fixture port checks.
- Keep SDK evidence tests tied to a normally built reviewed reference,
with a separate strict refusal for actual unreviewed CI outputs.

## Verification

App head: `6284d6881b5dc38a9ee149bcbde35e69a67dd8ca`, including actual
master `dff8bed007e01c32f3826144b893ec80603d1335`. Companion Cloud head:
`f83b38187dc7e6b2e219fe865dc16d268cc66fdb`, including actual Cloud
master `b1dcc3421a91efffee631a61536cf48f357c22fc`. Cloud #1250 is merged
at `1d07d81f23d4627ab7b6dcf49176b4128a2f8f9b` after exact-head passing
checks and complete Greptile 5/5. Current exact-head CI typecheck,
build, tests and aggregate verification are green; the complete edited
Greptile review is 5/5 and all 13 review threads and nested comment
pages are resolved.

- Both latest enrollment findings were reproduced against the preceding
product. The pure readiness control returned no wait for the first
enrollment. The real-PostgreSQL control observed a provider call while
that enrollment was still staged.
- The repaired source passes three whole files: 21 readiness controls,
80 real-PostgreSQL milestone controls and 37 tenant-actor middleware
controls. All 138 pass with zero excluded or TODO cases. The SQL case
reconstructs the worker through seven waits, preserves the attempt
budget, recovers a controlled lost enrollment reply, and commits one
provider-port call and one exact grant. Replaying the selected attempt
does not duplicate them. Authorization, SDK and enrollment-response
ports are controlled in that SQL case.
- The same files cover ordinary five-attempt exhaustion, accepted and
possible sends, stale ownership, malformed bindings, attention states
and current actor restrictions. One synthetic mismatched-binding
parameter uses generation 2 against current generation 1; it is not
evidence for an older-generation control.
- The normal server-package typecheck passes at the measured source. It
runs the normal Runner TypeScript, protocol, SDK and declaration
prerequisites and all seven main App programs. The separate declaration
configuration is also recorded. A worker union-narrowing error was
found, fixed with an explicit callable/null branch, and verified by that
normal compiler. No cast or test relaxation was used.
- The normal merge of current master preserves all 78 owned paths and
15,783 unrelated master entries and modes. The three measured test files
and their changed feature paths are unchanged. New upstream DB code adds
only the normal nonunique concurrent native-session index and its
migration classification. Shared and adapter-utils changes alter
unrelated prompt or denial wording. The 138-case proof carries for those
unchanged behaviors; it is not reported as a rerun after the merge. The
normal Shared/SDK ensure, DB build, adapter-utils build and complete
server-package check also pass at the current merged head. Their fresh
inventory contains 8,143 outputs; canonical SQL and journal copies
match. This build did not apply the new index migration.
- Earlier real-PostgreSQL retained-history coverage reproduced the
missing proof with 1,005 settled rows. Its repair retains the existing
two-second/64-query inspection budget, current endpoint bound and exact
owner snapshot. The 46 source-bound controls cover unfinished and
invalid tails, exact cross-page action references, Retry, rollback and
lost receipt responses.
- Earlier enabled quiet/reactivation coverage produced ordinary useful
work, kept configuration enabled, accepted fresh signed work plus a
duplicate, and produced exactly one second useful reply. Its two
App-worker windows lasted at least 30 seconds with zero measured PG
writes, connection or outbound activity after settlement. That evidence
stays tied to its measured revision. A missing worker EXIT marker limits
that earlier proof; parent evidence does not replace it.
- Earlier three-second callback, terminal ACK, current-question race and
complete body-size controls remain source-bound. The
500-owner/1,000-binding body measured 313,875 bytes with a 618-byte JWT;
signature tampering failed closed.
- Final managed-provider acceptance now passes all five fresh Node
boots: A/A, A/AB, B/A rejection, B/AB recovery, B/B final. Two
independently configured, challenged, verified, linked and enrolled
Slack identities exercise ACK and retirement debt separately. Normal
provider factory/materialization delivers the configuration through a
controlled external stack-provider boundary. The registered opening
returns HTTP 200 in 21 ms within the original 3-second deadline. Two
answers persist, normal ACK is accepted, exact retirement and lost
acceptance/GC replies recover, and the original canonical snapshot
remains unchanged without duplicate provider sends or submissions.
- The separate complete committed-preflight case passes. A committed
`update_attempted` response is lost before SDK entry; normal
service-owner reconstruction after the original lease/deadline recovers
the same useful update and one answer without a second trigger. This
reconstructs service owners within one App process; the separate five
fresh Node boots cover process restart and managed signing-key rotation.
- Assertion-time PG socket reports balance 15 opens/15 closes across the
five boots and 5/5 in preflight, peak 2 and zero live sockets or
unexpected attempts. Worker terminal EXIT markers were missing and are
not replaced by parent EXIT markers. These are domain/recovery and
assertion-time connection proofs, not a new whole-worker terminal quiet
measurement. All known owners/groups closed; independent SQL audits were
empty; only successful fixture databases were removed. Earlier failed
fixture databases/logs remain retained. No product guard, deadline or
assertion was weakened. Exact runtime source and normal outputs are
unchanged by the three test files in the current follow-up, so these
source-bound proofs carry without repeating those journeys.
- The preceding App CI ran 11,312 support cases: 11,289 passed, three
SDK-evidence cases failed, and 20 external cases skipped. Normal CI
dependency setup resolved six stale Slack patch-hash references and
produced an unreviewed lock. The public SDK proof correctly refused that
lock; the tests incorrectly assumed the CI output was a reviewed
reference. The three-file test repair keeps the protected lock policy,
source/compiler/parser/Zod pins, scanner and redaction rules unchanged.
It builds an owned reviewed reference using the existing normal
Shared/SDK producers for the positive seal test, while a separate
negative requires actual unreviewed-build privacy refusal and no
retained SDK bytes. It does not relabel the CI build as certified or
admit a new lock hash.
- All four whole SDK support files pass locally: public type 30,
evidence 16, public-input synchronizer 8 and lock preparation 10,
totaling 64 with zero excluded/TODO cases in 28.73 seconds. The genuine
fresh reference build/positive seal and separate invalidated-reference
privacy refusal passed within their original 120-second bounds. Original
installed dependencies, SDK outputs/cache and runtime source stayed
unchanged; normal global setup produced only its own Runner JavaScript
outputs. PostgreSQL was denied throughout, all four worker markers were
observed, and the owned process group/private test root closed. The
local canonical-lock negative mutates only its own reference; fresh
remote CI also passes the actual resolved-CI-lock branch: all 16
evidence and 30 public-type tests pass, including the genuine
reviewed-reference positive and the strict actual-unreviewed-build
refusal. The upstream synchronizer file runs its current 10 cases in CI.
- The first complete CI attempt on this head also exposed two separate
30-second timeouts: one real reporter-artifact test and the
`agent-test-environment` module-loading hook. Full original logs remain
preserved. Their exact test, sanitizer and evidence source blobs are
unchanged from the preceding head; inspection found no deterministic
setup cause. After the original run finished, one retry of only the
failed jobs passed on the same synthetic merge
`168851e210671ebe97e7a778e8cf891807e80502`: Runner support reports
11,304 passes and 20 external skips; serialized shard 6 reports all 395
passes, including all 67 route bodies. No source, timeout or assertion
changed. This proves recovery; the original causes remain unexplained.
- The final pre-merge tree audit uses actual master
`17136a6c25ef6cb7c816a29140a4c48c2c75fb31` (refreshed after the earlier
`85dec046` audit). The expected normal squash preserves all 81 owned
paths and 15,816 unrelated master entries and modes. Only OpenAPI has
edits from both sides, and its complete content composition preserves
the new project route and token help alongside this PR's API entries.
The shared inbound selection changes preserve the same candidate set;
new concurrent indexes, Native/workspace behavior and shutdown changes
remain upstream behavior. The scoped form proof uses process agents and
does not qualify the new Native/sandbox branches. The latest
question-schema change still accepts the explicit version-1 payload
exercised here. The last master delta also changes managed AI/Native
credential refresh and writeback; those branches are outside the
measured process-agent form paths. Existing 8,143 local outputs remain
tied to their measured revision; no new whole-tree output equality or
deployed suspension is claimed.
- Reproduce the public path with
`doc/testing/slack-cloud-qualification.md`, the companion Cloud
checkout, Node 24/22 and fresh owned fixture databases. Aggregate local
`pnpm -r typecheck`, `pnpm test:run` and `pnpm build` were not rerun as
one command set. Current exact-head required CI and the complete
Greptile 5/5 review now pass; the full edited summaries and all
paginated threads were read.

Local worker quiet does not prove deployed host or Neon suspension.
Photon and Discord remain separate work; this PR completes the described
Slack slice.

## Risks

Use compatible App and Cloud readers before transferring custody.
Signatures prove authenticity; current authorization and exact ownership
still control execution. Preserve old targets, secrets and terminal
facts until their normal owners release them. Keep the managed seal
identity stable while custody remains. The required Slack adapter patch
remains pinned. Upstream's concurrent index uses the normal top-level
migration path.

## Model Used

OpenAI `gpt-6.1-sol` implemented and tested the changes. A GPT-6-based
Codex managing agent reviewed and integrated them. The host does not
expose the managing agent's exact model ID or context-window sizes. The
agents used code execution and test tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used, its version and tool capabilities
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public related issues or PRs
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch name describes the change and contains no internal
Paperclip ticket id
- [x] I have run the local tests listed above and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented risks
- [x] All Paperclip CI gates are green for the current head
- [x] Greptile is 5/5 with no open findings for the current head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-10 12:17:50 -06:00
DottaandClaude Opus 5.5 99a6daf5bc perf(auth): resolve a Cloud tenant actor in one database round trip (#20390)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Paperclip Cloud, each request resolves its board actor from
trusted Cloud headers in `actorMiddleware`
> - This runs for every request after the liveness probe, also for
static files such as `favicon.ico` and `index.html`
> - The resolution ran three database statements one after another: the
stale `instance_admin` purge, the membership read and the
owner-elevation read
> - On a loaded database, each statement waits. Three sequential waits
make every request slow, also a request that needs no data
> - This pull request keeps the purge and the tenant sync in order and
runs the two reads at the same time as them
> - The benefit is one database wait per request instead of three

## Linked Issues or Issue Description

**What existing behavior does this improve?**
Cloud tenant actor resolution in `server/src/middleware/auth.ts`
(`resolveCloudTenantActorOnce`).

**Subsystem affected**
Server authentication middleware (Cloud trusted-header path).

**Current behavior**
Measured on the hosted staging instance on 2026-11-09 from a signed-in
browser, 45 paired samples in 40 s:
- `/api/liveness` (mounted before actor resolution): client median 92
ms, p90 255 ms, server time ≤ 1 ms.
- `/favicon.ico` with a session (after actor resolution): client median
413 ms, p90 1,751 ms, max 4.6 s.
- `/api/cli-auth/me`: server median 346 ms, p90 1,528 ms.
The same Node process answers liveness quickly, so the event loop is not
blocked. The database is busy (6–8 of 8 vCPU for the whole day), and
trivial primary-key reads waited 100–280 ms each. Three sequential reads
explain the ~320 ms median gap between liveness and favicon.

**Proposed behavior**
The purge and the debounced tenant sync stay ordered. The membership
read and the owner-elevation read run at the same time as them. The sync
writes only the primary company's membership, and the membership read
excludes that company, so the result does not change.

**Reason and benefit**
A synced request costs one database wait instead of three. With 150 ms
of injected database latency, a synced resolution goes from 914 ms to
310 ms (3 runs each, same machine).

**Breaking changes**
None.

## What Changed

- `resolveCloudTenantActorOnce` starts the purge-and-sync, the
additional-membership read and `resolveOwnerInstanceAdmin` together and
awaits them with `Promise.all`. A membership read failure still degrades
to the primary company. The owner-elevation read still fails closed.
- New regression test
`server/src/__tests__/cloud-tenant-actor-round-trips.test.ts`. It
connects through a TCP proxy that delays every client write by 150 ms
and asserts that a synced resolution finishes in less than 4 × the
delay. Before the change it takes about 6 × (914 ms); after the change
about 2 × (310 ms).
- `auth-session-route.test.ts`: the fake database binds each
`returning()` to its own insert, and the test matches inserts by content
instead of position, because the instance-settings insert can now happen
in parallel with the tenant sync.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-tenant-actor-round-trips.test.ts` — red on
`origin/master` (914 ms, 913 ms, 916 ms), green with the change (310
ms).
- `cd server && npx vitest run
src/__tests__/cloud-tenant-company-provisioning.test.ts
src/__tests__/auth-session-route.test.ts src/__tests__/liveness.test.ts
src/__tests__/cloud-warm-standby.test.ts
src/middleware/cloud-tenant-actor.test.ts`
- `pnpm verify` (selected tests), and CI.
- After deploy: compare `/favicon.ico` and `/api/cli-auth/me` client
time with `/api/liveness` on staging. Expect the median gap to drop from
about 3 database waits to about 1.

## Risks

- Low risk. Authorization does not change: the same statements run and
the request waits for all of them before it continues.
- Each request now uses up to three pooled connections at the same time
instead of one. The staging pool limit is 500, so this is not a concern
there. Smaller pools reach their limit sooner under a burst.
- The diff re-indents the purge-and-sync block. The statements in it do
not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, browser, code execution). Extended reasoning on.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 20:29:37 -06:00
DottaandClaude Opus 5.5 7a93cc6215 fix(server): redact invitation tokens from agent run logs (#20360)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People join a company through an invitation link. The link carries a
bearer token in its URL path.
> - An agent can create an invitation in chat and give the person the
join link.
> - The server writes each agent run to a run log file (`.ndjson`). This
file is a plain diagnostic copy. Local agents run as the same OS user,
so they can read it.
> - #20343 removed invitation tokens from HTTP logs. The run log still
kept the full join link and the raw token.
> - This pull request removes invitation tokens from run-log chunks when
the server writes them.
> - The benefit is that a live join link does not stay in a file that
other local agents can read. The person still gets the working link in
the chat reply.

## Linked Issues or Issue Description

Refs #20343 (same token policy for HTTP logs).

**What happened?**

An agent created an invitation in chat and replied with a
`/invite/<token>` join link. The agent run log file under
`data/run-logs/` kept the full token. The token was in the agent message
and in the tool result.

**Expected behavior**

The run log keeps the route and drops the token:
`/invite/***REDACTED***`. A raw `pcp_invite_...` token also becomes
`***REDACTED***`. The chat reply that the person reads keeps the working
join link.

**Steps to reproduce**

1. Check out master.
2. Run `server/src/services/heartbeat/run-log.test.ts` from this PR.
3. The "run-log invitation token privacy" tests fail. The token stays in
both the legacy text chunk and the canonical native envelope.

**Paperclip version or commit**

Master at the merge base of this branch.

**Deployment mode**

Source server, local run logs (`local_file`). The same functions also
format object-store run logs.

## What Changed

- Add `redactInvitationTokens()` in
`server/src/middleware/invitation-token-redaction.ts`, next to the HTTP
log redactor from #20343. It removes the token from `/invite/<token>`
segments and from raw `pcp_invite_` tokens inside free text and JSON
text. The character classes stop at JSON, Markdown and URL delimiters,
so the record structure stays valid. `redact-sensitive.ts` stays
byte-identical, because the chat eval pins its exact bytes before it
accepts a real invitation link.
- Apply it in `compactRunLogChunk()` and
`redactCanonicalNativeRunLogChunk()`. These are the two functions that
format every agent run-log chunk before the store writes it. The run
excerpt columns use the same chunk, so they are also redacted.
- `canonicalNativeRunLogRedactionForms()` also yields the earlier,
unredacted projection. A run log written by an earlier release still
verifies after an upgrade.
- `restoreCompactedEvent()` accepts a truncated record that the earlier
compactor wrote with the token. Cold recovery of such an old log still
works.
- Add a test that proves the chat reply comes from database run events,
not from the run log. The reply keeps the full join link, and the
presentation path never reads the run log.

## Verification

- From `server`: `node ../node_modules/vitest/vitest.mjs run --config
vitest.config.ts src/services/heartbeat/run-log.test.ts
src/__tests__/invitation-token-redaction.test.ts
src/__tests__/redact-sensitive.test.ts
src/__tests__/heartbeat-run-log.test.ts
src/__tests__/logger-invitation-redaction.test.ts
src/services/native-runtime/native-run-log-watermark.test.ts
src/services/native-runtime/native-run-log-recovery.test.ts
src/__tests__/heartbeat-run-output.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts`
passes.
- Before the `run-log.ts` change, the two new redaction tests in
`run-log.test.ts` failed (red), then passed (green).
- `native-run-log-watermark.test.ts`: a legacy-compacted record with a
token in its retained head cold-recovers. It failed with
`native_log_source_coverage_unproven` before the restore fix.
- `agent-chat-response-recovery.test.ts` (embedded Postgres): a final
reply with a join link is saved as the exact comment with the token. No
run-log read happens.
- `tsc --noEmit -p server` shows no errors in the changed files.
- A live chat eval (`access.invite-teammate`) run with this change
confirms the run-log file has no token. The run is linked in a comment.

## Risks

- Low risk. Run-log readers (run transcript view, feedback export,
support inspection) now show `***REDACTED***` in place of invitation
tokens. This removes a live credential from a diagnostic copy. It does
not change the chat reply or the stored run events.
- A path segment after `/invite/` in free text is always redacted, even
if it is not a token. This loses only diagnostic detail.
- A token that a provider streams across two separate chunks is not
joined first. Each chunk is redacted on its own. A complete token in one
chunk, such as a final message or a tool result, is removed.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 17:52:19 -06:00
DottaandPaperclip a67b936885 fix(server): redact invitation bearer paths from HTTP logs (#20343)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People join a company through an invitation link. The link carries a
bearer token in its URL path.
> - The HTTP logger removes query secrets and URL user information. It
keeps the path.
> - So invitation bearer tokens reach the HTTP log in the request URL,
the log message, and the referrer header.
> - PR #17848 fixed this, but it did not merge. Since then, #13742
changed the same redactor file.
> - This pull request carries the #17848 commits onto current master.
The original author is kept on both commits.
> - The benefit is that invitation tokens do not stay in HTTP logs. The
chat eval for membership management can then accept a real invitation
safely.

## Linked Issues or Issue Description

Supersedes #17848 (same commits, rebased onto current master after
#13742).

**What happened?**

The HTTP logger writes invitation bearer tokens to logs. A synthetic
POST to `/api/invites/<token>/accept` puts the token in `req.url`, in
the log message, and in the `Referer`/`Referrer` header record.

**Expected behavior**

Logs keep the invitation route, the method, and the HTTP status. Logs do
not keep the bearer token.

**Steps to reproduce**

1. Check out master.
2. Run `server/src/__tests__/logger-invitation-redaction.test.ts` from
this PR.
3. The synthetic token appears in the request URL, the message, and the
referrer. 12 tests fail on master.

**Paperclip version or commit**

Master at the merge base of this branch.

**Deployment mode**

Source server. The tests use a disposable local HTTP listener. They use
no database and no provider.

## What Changed

- Remove the bearer segment from `/api/invites/<token>/...`,
`/invites/<token>/...`, and `/invite/<token>` URLs, including mounted
paths and absolute URLs.
- Apply the same URL policy to `referer` and `referrer` request headers.
- Keep record IDs on the non-token revoke route
`/invites/<inviteId>/revoke` and on company-scoped invite routes.
- Add regression tests through the real HTTP logger for 202, 400 and 500
responses, with both referrer header spellings.
- Original work by @cryppadotta in #17848. This PR only rebases it onto
master.

## Verification

- From `server`: `node ../node_modules/vitest/vitest.mjs run --config
vitest.config.ts src/__tests__/logger-invitation-redaction.test.ts
src/__tests__/redact-sensitive.test.ts
src/__tests__/http-log-policy.test.ts` — 46 passed.
- With the master `logger.ts` and `redact-sensitive.ts`, the same new
tests fail: 12 failed.
- `http-log-aborted-response`, `cloud-principal-log` and
`setup-token-session` tests pass locally. Some other HTTP-log suites
could not load locally because workspace package builds were missing in
this worktree. CI covers them.

## Risks

- Low risk. Referrer query and fragment values are now removed from
request logs, as they already are for request URLs. This removes some
diagnostic detail to protect secrets.
- A route that uses `/invites/<id>/...` for a non-bearer record ID,
other than `revoke`, now shows `[REDACTED]` in logs for that segment.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution, used to rebase and verify. The original change was authored
in #17848.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-09 13:45:00 -06:00
DottaandClaude Opus 5.5 ac0d0374f2 perf(issue-detail): measure task opens on real instances (#20276)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People open a task from their issues list to see what an agent is
doing and to reply
> - On a busy staging instance, the composer and the latest messages
take 1.1–2.0 s to appear after a click. For one task with a slow run
log, they took 13.9 s
> - Local fixtures do not reproduce this. They do not have real network
round trips, real server contention, or real object-store latency
> - This pull request adds the measurement we need: a browser harness
that times a task open on a real instance, and a Server-Timing entry on
every API response that separates server time from network time
> - It also records the staging baseline and the improvement plan
> - The benefit is that each later performance change can be measured on
real data with the same tool, before and after deploy

## Linked Issues or Issue Description

**Problem or motivation**
Opening a task from the issues list is slow on instances with real data.
The composer and the latest chat messages can take seconds to appear. We
could not measure this on staging in a repeatable way. We also could not
tell server time from network and queueing time for most API requests.

**Proposed solution**
Add a reusable browser harness for task opens, and add a `paperclip_api`
Server-Timing entry to every API response.

**Alternatives considered**
Local Playwright fixtures only. They do not reproduce real network round
trips, server contention or object-store latency, so they did not show
the delays that people see. Per-endpoint Server-Timing only. It does not
cover the many other requests that run during a task open.

**Roadmap alignment**
This is measurement for existing task-loading behaviour. It does not add
product features. The staging baseline and the measured causes are in
`doc/plans/2026-11-09-task-loading-performance.md`. Follow-up pull
requests change the loading behaviour itself, starting with #20277.

## What Changed

- Added `server/src/middleware/api-server-timing.ts`. Every `/api`
response now has a `paperclip_api;dur=<ms>` Server-Timing entry. The
value is the server time from request arrival until headers are written,
including admission waits. The entry is appended, so the existing
`paperclip_issue` phases stay first.
- Registered the middleware first in `server/src/app.ts`.
- Added `tests/perf/issue-detail/staging-click-harness.js`. It clicks a
task row and records, frame by frame, when the composer and the latest
message become visible, layout shifts, movement of the latest message,
and the requests that run before the page is ready. The summary labels
each sample instead of using the task reference. It replaces query
values and every path segment that contains a digit. A fetch that is
still running when the sample ends is listed as pending.
- Documented the harness in `tests/perf/issue-detail/README.md`.
- Added the dated plan
`doc/plans/2026-11-09-task-loading-performance.md` with the staging
baseline, the causes, and the prioritized slices.

## Verification

- `cd server && npx vitest run src/__tests__/api-server-timing.test.ts`.
3 tests pass. They cover the JSON responses, the time spent in earlier
middleware, preserving route phases, and explicit `writeHead` calls.
- `tests/perf/issue-detail/issue-detail.perf.spec.ts` reads the first
`dur=` value. That value is still `paperclip_issue`.
- The harness was used on staging to capture the baseline in the plan
document.

## Risks

- Low risk. The header has only a duration, with no request, user or
company data.
- The middleware wraps `res.writeHead` once for each request. This is
the same pattern that `on-headers` uses.
- Server-Timing is visible to same-origin scripts only. No
`Timing-Allow-Origin` header is added.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, browser automation, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 09:06:37 -06:00
DottaandClaude Opus 5.5 43fe87795a fix(authz): make board-only denials final and actionable for agents (#20151)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in Agent Chat, and the agents act
through the Paperclip API
> - Many API routes are board-only. The shared `assertBoard` check
returns 403 `"Board access required"` to an agent, with no other detail
> - The chat guidance also told the agent to "use permitted discovery to
identify the supported operator path" after such a denial
> - So the agent keeps searching the API, `call_api` and the shell for a
second route. On a slow model, this search uses the whole reply window
and the person gets no answer
> - Earlier single-route fixes (#19990, #20041, #20008) showed the
remedy: a denial that says it is final and names where the person acts.
Reply time fell from 10-24 minutes to about 2 minutes
> - This pull request applies that remedy once, in the shared board-only
check and in the chat guidance
> - The benefit is a prompt, correct handoff for every board-only action
that an agent cannot do

## Linked Issues or Issue Description

Refs #19990, #20008, #20041 (same remedy for single routes).

**What happened**
In about 30 Agent Chat evaluation cases on a slower model, the agent got
403 "Board access required" (or read `authorization.actor: "board"` in
the API catalog). Then it searched for another route until the attempt
deadline. Examples: archive a company, set up a managed agent profile,
add a company secret. In one recorded run, the agent got the 403 at
minute 1, then tried PATCH, environment variables, the CLI and UI paths
for more than 3 more minutes, and never replied.

**Expected behavior**
A board-only denial is final. The agent stops, tells the person that a
board user must do the action, links the page in the Paperclip web app
where they can do it, and ends its reply.

**Steps to reproduce**
1. Open Agent Chat with a CEO agent.
2. Ask it to archive the company (or to add a company secret).
3. The agent calls `POST /api/companies/{id}/archive` and gets `{
"error": "Board access required" }`.
4. The body says nothing about finality or where the person acts. The
agent searches for other routes and replies late or not at all.

## What Changed

- `server/src/routes/authz.ts`: `assertBoard` now throws
`boardAccessRequired(req)`. For an agent actor, the 403 body adds `code:
"board_access_required"` and a `remediation`. The remediation says the
action is board-only by design, the denial is final, and no other route
can do it. When the API path maps to a known board page, it links that
page with a `<prefix>` placeholder: company settings, company secrets,
budgets, members, tool gateways, connection settings (decision model,
prompt injection prevention), an approval, or the agent profile tab that
holds the control (API keys, permissions, budget, identity, harness
settings). Otherwise it says to link a page only when the route is
already known, and not to search for one. Board and anonymous actors get
the same body as before.
- `environments.ts`, `instance-settings.ts` and the issue force-release
route used their own copy of the check. They now call `assertBoard`, so
they return the same denial. Status and error text do not change.
- `server/src/middleware/error-handler.ts`: for this code, the handler
does not repeat `code` and `remediation` inside `details`. They are
already top-level, and the copy only made the agent's context longer.
- `AGENT_CHAT_DIRECTIVE`: the board-identity paragraph now says that a
board-only operation (catalog actor `board`, or a 403 "Board access
required") is final. The agent must stop searching, tell the person
which control in the web app to use, and link it when it knows the
route. The old instruction to "discover the operator path" is removed.
- Claude Managed Agent setup: the Paperclip Runner configuration doc and
the `POST /api/companies/{companyId}/managed-agent-profiles` catalog
entry now say that a board operator creates and qualifies the profile
(limited environment, retention acknowledgement) and that agents cannot.
Before, an agent asked for an "Anthropic managed agent" explored hiring
and adapter docs for its whole reply window.
- Tests: new `board-access-denial.test.ts` covers the agent body for
each mapped page, the agent profile tabs, the general fallback, the
unmapped fast-response route, the org-access guard, the unchanged
non-agent body, the guidance and the managed profile catalog entry. Two
exact-body tests now also expect the new code.

## Verification

- `cd server && npx vitest run src/__tests__/board-access-denial.test.ts
src/__tests__/costs-service.test.ts
src/__tests__/browser-use-route-scope.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/agent-cross-tenant-authz-routes.test.ts
src/__tests__/company-portability-routes.test.ts
src/__tests__/issue-comment-cancel-routes.test.ts
src/__tests__/error-handler.test.ts
src/__tests__/managed-agent-profile-routes-authz.test.ts
src/__tests__/companies-route-cross-company-authz.test.ts
src/__tests__/environment-routes.test.ts
src/__tests__/environment-instance-routes.test.ts`: all pass.
- Real chat journeys on the chat evaluation branch (runner-opencode,
`deepseek-v4.1-flash`). Before this change, each case below got no reply
before its deadline:
- `companies.archive-company`: passed. Reply 47 s after the request
(before: none in 270 s). The agent read the remediation and linked
`/COM/company/settings`.
- `companies.configure-decision-model`: passed. Reply 68 s after the
request (before: none in 389 s). It linked
`/COM/company/settings/connections`.
- `companies.prompt-injection-guard`: reply 85 s after the request
(before: none in 390 s), with the correct handoff and link. The case
still fails later, in the scripted settings step: the saved model is the
default, not the model the script typed. That is a separate defect.
- `navigation-preferences.sidebar-order`: reply 99 s after the request
(before: none in 150 s). The case still fails in its scripted sidebar
step (a locator was not found).
- `secrets.configure-company-secret`: reply 6.5 min after the request
(before: none in 7.5 min). The reply is correct, but a grader keyword
check reads "chat is not a safe place to send it" as a request for the
secret.
- `agents.managed-profile`: still no reply in its 150 s window. After
the catalog change, the agent found the board-only profile setup within
30 s, but then spent about 90 s deciding what "Anthropic managed agent"
means.

## Risks

- Low risk. The HTTP status and the `error` text stay the same for every
actor. Board users and anonymous callers get the same body as before.
- Agent actors now get two more top-level fields (`code`,
`remediation`). A client that compares the whole 403 body of an agent
call exactly will see a change. Two server tests did this and are
updated.
- The page map uses the API path. A wrong map entry gives a less useful
link but does not change authorization.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:28:52 -06:00
Dotta eb7c7b083a Recover Muse verification after contact, restart and access restoration (#19909)
## Thinking Path

> - Paperclip manages AI agents and their work under company access
rules.
> - Muse readiness uses a saved connection, receiver contact and an
authenticated challenge reply.
> - A failed challenge write could leave a saved contact with no later
recovery trigger.
> - A server restart or restored access must recover that contact under
current authority.
> - This pull request uses the existing delivery coordinator and saved
Muse state for recovery.
> - It keeps the normal receiver schedule, authorization checks and real
deadlines.
> - It lets eligible verification recover and lets idle verification
stop work.

## Linked Issues or Issue Description

**What happened?**

The first Muse receiver contact could persist while readiness challenge
creation failed. The contact then had no durable retry or startup
recovery path. Later membership, lifecycle or feature restoration could
also leave pending verification without a notice.

**Expected behavior**

Retry a temporary failure through the existing worker. Recover pending
contact after restart. Recheck the current company, agent, operator,
credential and binding generation before issuing a challenge. Stop
verification work while it is ineligible or complete. Do not revive
revoked credentials.

**Steps to reproduce**

1. Use the normal invitation and OAuth registration, PKCE and token
flow.
2. Send one receiver contact and let its normal persistence schedule
run.
3. Fail the first readiness challenge operation before it commits.
4. Observe whether the existing worker retries without another contact
or a manual Verify action.
5. Restart the App with a new database owner. Check that it creates one
usable challenge from the saved contact.

**Paperclip version or commit**

The causal baseline test used the unchanged production implementation at
`90148317b2a447794ff29a4acae5648ad11a9770` plus its normal fixture. The
accepted positive test used `9d383041028776ad8469ac4af3fe14ba6c369425`.
The initial published head was
`fe518c9b72504daf6374225987f24d45bc7bcf75`. The lease/auth correction
was locally qualified at `cfd515a207416097fe8b586277d70c4fd4a679fa`. The
current candidate is `6efe722e7eb1e22acd466125c7b4a627611d0273`; its
added Cloud actor fixture correction passed the full normal owning
program and all 35 tests. Its current normal build, lifecycle owner
controls, healthy-wait and restoration measurements are described below.
Earlier results below are bound to their stated revisions.

Related public changes: Refs #19819 for original-principal
authentication. Refs #19750 for the transaction-owned external feature
read. These are related foundations, not duplicate fixes. The current
Muse title and head-branch searches found no duplicate
contact-verification recovery PR. Related maintenance work is #17000 and
terminal assignment settlement is #17793.

## What Changed

- Add a typed, exact-binding verification notice through the existing
Muse delivery queue.
- Register the first-contact notice before its durable write in the same
transaction.
- Recover pending contact from saved state at startup. Retry temporary
failures through the existing scheduler.
- Recheck current feature, lifecycle, credential and access state after
commit. Park unavailable authority without recurring verification scans.
- Add narrowly scoped notices to the existing access, lifecycle,
configuration, company, budget and settings write owners.
- Save a healthy lifecycle verification wait in the existing operation
JSONB. Keep generic unresolved-work and error retries unchanged.
- Commit authenticated confirmation and its lifecycle recheck in one
transaction. Retain a raced recheck under the active lease, and recover
saved waits once at startup.
- Retain a committed lifecycle recheck while an expired attempt still
owns the row, so its late wait cannot park accepted confirmation.
Replacement claims keep their existing owner fence and fresh fact read.
- Keep the receiver gap, persistence and per-segment contact limits
unchanged.
- Add real App OAuth, failure, retry, restart, challenge and quiet
evidence. Add two normal-route restoration tests.
- Document recovery, reconnect requirements and the existing
connected-receiver sleep restriction.

## Verification

- Causal baseline: the real invitation and OAuth flow saved one contact.
The first challenge operation failed. The normal test failed because it
observed one verification attempt instead of a retry. No setup or
cleanup failure was counted as this regression.
- At `9d383041`: normal own Shared/SDK, DB with migration 0441 assets,
Runner TypeScript core and surfaces, module boundaries, and all seven
normal App programs passed. One new redundant role comparison first
failed typecheck; that diagnostic remains recorded and the corrected
full App gate passed.
- At `9d383041`: all 17 maintenance component tests passed with zero
skips. All original deadline, rollback, drain, terminal settlement and
startup controls remain.
- At `9d383041`: the one normal App case passed. Real OAuth authorized
one receiver signal. A retry occurred about 5.01 seconds after the first
failure. A new Db/App owner created one challenge, advanced revision 2
to 3, and accepted authenticated confirmation and mailbox ACK. Query
instrumentation had an active positive control. All measured query,
callback, session and verifier counters stayed equal across the
unchanged scheduled 15-second quiet window. Token revocation and
old-token/signal rejection passed.
- Known App, worker, HTTP and database owners closed without cleanup
errors. The tests used bounded local PostgreSQL and HTTP fixtures. They
did not call a live Muse model.
- Current integrated semantics: the full normal App driver completed all
programs. Only a new test callback had two undefined-value diagnostics.
An explicit defined-value guard corrected that test. The full 416-root
owning program then passed with empty diagnostics. A separate collector
incorrectly required an explicit child `noEmit` field; the normal
inherited setting was verified without repeating the compiler. The
original failures remain recorded.
- The first restoration run failed both cases. Permission parking
recorded 120 main-pool queries without more verifier calls. The
lifecycle counter recorded two denials without their exact scope. These
observations remain unexplained at the individual-query/call level.
Source separately shows a lifecycle readiness retry every two seconds
while the real external reply is absent.
- At `fe518c9b`: normal Shared/SDK and DB producers, module policy, all
seven full App programs, and all 33 pure diagnostic controls passed. The
lifecycle owner file passed all 19 cases, including both
confirmation/park orders, rollback, duplicate notices, startup recovery
and authority controls.
- At `fe518c9b`: the normal OAuth failure/retry/new-Db reconstruction
case passed. Its new healthy-wait window measured 15,002 ms with
unchanged total activity before genuine confirmation; the existing ready
quiet and revocation checks also passed.
- At `fe518c9b`: both normal restoration tests passed in Vitest.
Ordinary viewer-to-operator restoration and Pause/Resume recovered the
same binding, grant and single contact; each committed one challenge and
advanced revision 2 to 3. All six parked, restored and revoked/disabled
quiet windows measured at least 15 seconds with active positive read
controls and unchanged total queries, callbacks, sessions and verifier
calls. Both cleanups passed.
- The restoration wrapper still recorded failure: it required lifetime
lifecycle denial count zero, whereas the measured count was one before
the parked quiet window. The reviewed fixture explicitly permits a
current-authority denial from an earlier observation and requires no
committed challenge before restoration. Precise denial timing relative
to Pause was not serialized. The original wrapper failure is retained;
no rerun or retrospective explanation of the earlier failures is
claimed.
- Review exposed an expired-lease race. The added real-store baseline
test failed after the late defer restored `waitingForSignal`; no setup
or timeout was counted as the regression. On `cfd515a`, the whole
lifecycle owner file passed all 20 tests, including that case. The full
normal production declaration program (1067 roots), verification program
(662), and auth owning program (405) passed; all 1942 declarations
remained byte-identical.
- The original remote auth-session shard failed three tests because
their partial query adapters lacked the new binding query's
join/order/limit chain. A fixture-only correction preserved the real
notifier and existing authorization assertions. All 12 auth-session
tests passed locally.
- Complete remote inspection also found 33 failures in the 35-test Cloud
actor fixture with the same missing `innerJoin` method and downstream
incomplete-auth assertions. Its original full log and exact synthetic
Source comparison are retained. At `6efe722e`, the three-line fixture
adapter repair passed the complete normal 662-root test program and the
whole 35-test file with zero failures or skips. All existing lock,
authorization, settings, debounce and unfinished-auth assertions remain.
- The earlier healthy/restoration quiet measurements remain bound to
`fe518c9b`. Those steady 15-second windows did not exercise a
still-owned attempt passing its 120-second lease. The targeted
expired-owner regression qualifies that new boundary; it is not a fresh
measurement of the earlier windows.
- The initial `fe518c9b` remote gates failed the auth fixtures and
reported a Greptile 4/5 P1. The aggregate failure reported the
underlying shard. Those actual causes were fixed and qualified above.
- Current `6efe722e` remote gates completed with **57 successful checks
and three explicit skips**. The full latest edited Greptile summary is
**5/5**, explicitly reviews this exact head, and has no new finding. The
original P1 thread is resolved. All general, inline and nested review
pages were read.
- One current-head CI attempt timed out after 30 seconds in the Runner
evidence/privacy file, with 6,864 passes and one failure. That file and
its transitive inputs are byte-identical to the CI base. A separate
local diagnostic preserved every assertion and the original timeout, ran
the full normal Runner test project and the entire 41-case file, and
passed all 41 with zero skips. Its private bounded timing records
measured the target body at 74 ms. The diagnostic was then removed,
restoring the exact original test. This is local recovery evidence; the
original CI stalled await was not recorded and its cause remains
**unproven**.
- One retry of only the failed jobs on the unchanged head passed the
Runner support suite and dependent aggregate verification. The complete
second attempt and original failure logs are preserved. No test,
timeout, privacy guard or product code was weakened. This result does
not turn the original timeout into an explained failure or a diagnosed
flake.
- Full recursive typecheck, full `test:run` and a full release build
were not repeated locally. The normal full App programs, focused
producers and tests above passed, and the required current-head remote
gates passed. Final review is against the exact unchanged head; local
quiet remains separate from deployed suspension.

Reviewer commands use the normal project entry points: `node
server/scripts/typecheck-layers.mjs`, the whole
`server/src/services/muse-maintenance.test.ts` file, and both normal App
PostgreSQL files. PostgreSQL fixtures require explicit owned
fixture/admin URLs and normal migrations before imports. Run the files
serially with one worker. The private controller retains exact Source,
output, worker and owner-closure receipts.

## Risks

- Missing or early producer notices could lose a restoration trigger.
Notices stay in the transaction that commits the write. The worker
checks current authority after commit.
- A stale notice must not grant access. Binding generation, current
membership, company, agent and credential checks remain authoritative.
Revoked credentials require reconnecting.
- The change adds no table, migration, alternate task queue or periodic
verification scan. It adds two optional fields to existing lifecycle
operation JSONB. Generic pending/error deadlines, setup holds and manual
repair remain obligations.
- Confirmation can race with an active verification attempt. The park
compare-and-set must retain the durable recheck. Startup recovery uses a
fixed cutoff and keyset pages; it must not become another recurring
scan. Wrong company, stale version, pause and revoked credentials must
stay fenced.
- The connected receiver still has its documented tenant-sleep
restriction. Worker quiet is not proof of deployed suspension or
external wake.
- The normal challenge reply is sent by an authenticated protocol
fixture. Live Muse background replies, cost/usage, full Muse lifecycle
and full hosted Dot behavior remain unqualified. Partial Dot profile ACK
and unsigned-device cutover stay withheld.
- Restoration coverage for the additional company, budget, configuration
and feature producers remains a separate qualification obligation. This
pull request does not mark the whole hosted area complete.

## Model Used

OpenAI `gpt-6.1-sol` authored the implementation and tests with local
code and tool use. A GPT-6-based Codex coordinator reviewed the changes
and owns integration. Its exact underlying model ID is not exposed.
Context window sizes are unavailable. No context size is inferred.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-07 16:41:15 -06:00
Dotta 3112e4c955 Verify original Dot OAuth authority for signed Cloud requests (#19819)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Dot uses a user OAuth grant to act in one company.
> - Cloud must use that original grant when it resolves a selected
user's identity.
> - A Cloud signature alone must not create user or board authority.
> - The tenant's OAuth owner must check the token and its current
restrictions.
> - This pull request adds a signed private HTTP boundary for that
check.
> - The result contains only the checked identity fields. It grants no
task or board access.

## Linked Issues or Issue Description

**Subsystem affected**

Server external-agent identity, public MCP OAuth, Cloud control
verification, and HTTP logging.

**Problem or motivation**

Cloud needs a private way to resolve the original Dot token for the
selected user, client, company and resource. The existing generic Cloud
control route creates an operator actor. That route is not the right
owner for this identity check.

**Proposed solution**

Add `POST /mcp/runner/oauth/cloud-principal`. Require the existing Cloud
signature and replay verifier. Bind the whole request to a new, closed
`mcp-dot:principal` action with a maximum 60-second lifetime.
Authenticate the original token once through the normal Dot OAuth owner.
Check current namespace authority before and after that call.

**Alternatives considered**

A Cloud-created grant or a board actor would change the authority owner.
Neither is used. The private route reuses the existing OAuth and Cloud
identity owners.

**Roadmap alignment**

This is a narrow boundary for the existing MCP and Cloud areas in
`ROADMAP.md`. It adds no work queue, public hosted profile, or provider
enablement.

Related public work: Refs #14846, #19427, #19750 and #19776. Searches
for the exact private path and action found no duplicate PR. The
matching private Cloud caller has merged. The current App head passed
its required checks and review.

## What Changed

- Add an original-token principal reader. Keep the original client,
selected user, company, resource, purpose, scope, membership and feature
restrictions.
- Add a 16 KiB raw JSON private route. Reject compressed, invalid UTF-8,
malformed and unknown-field bodies. Bind an ordered whole-body digest to
the signature and existing nonce fence.
- Compare the current stack, claim, instance, origin and connector
identity before and after OAuth. An absent owner or namespace fails
closed.
- Mount the route in normal `createApp` after hostname, runtime-claim
and standby checks, before session/actor work. Do not add it to generic
board-control routing.
- Keep private request and response logs content-free, including raw
buffers, headers, query data and reflected errors.
- Add controlled HTTP/privacy tests and a real normal-App
OAuth/Cloud-signer integration case. Document the machine-only contract
and correct the existing `assistant-connections/client-use` access
entry. No new capability ID or agent API entry is added. No schema,
lockfile, workflow, Runner, Muse or useful-task fixture changes belong
to this PR.

## Verification

- Current `70c03593`: normal module-boundary policy and all seven App
compiler programs passed. The public module factory has an explicit
portable Express return contract. Whole HTTP/log controls passed again:
30/30. The existing real paired OAuth/Cloud-signer/private HTTP case
then passed again on a fresh PostgreSQL fixture: 1/1, zero skipped. All
known child processes and five database connections closed; the final
independent SQL audit was empty.
- The original remote policy failure rejected an App import of a
module-internal path; dependent E2E/verify gates stopped on that
failure. The fix delegates through the public module API. A first local
declaration run reported TS2883 for its inferred return type; the
explicit public type fixed it. Original logs remain preserved. No
checker, compiler option, test body or authority check was weakened.

- The normal full App graph passed on the integrated source. The later
fixture-only repairs each passed the complete affected owning
program:405 entries, including 403 source roots. The remaining programs
and own dependency outputs carried only where their source was exact.
- Whole private HTTP/log files passed:30 tests, zero failures or skips.
The HTTP controls use a controlled OAuth owner with real signatures. The
log controls use the actual HTTP logger.
- The paired Cloud source passed its normal strict build and whole 26
signer/client/namespace and credential-error controls. These are
controlled caller tests, not provider tests.
- One normal-App paired case passed with a dedicated PostgreSQL
database. It used real DCR 201, PKCE, consent, token exchange and a
persisted unpaired grant; actual signed metadata publication and a
committed Cloud receipt; the real Cloud signer/client and actual App
HTTP; exactly-once authentication; refresh; current namespace
restrictions; and revocation. The Cloud registry/provider were the
existing controlled in-memory/Fake fixtures. Recorded inbound Hosts
matched, with zero metadata overflow. No grant, readiness or mirror ACK
was fabricated.
- The original principal helper's 21 controlled tests and two real-OAuth
tests passed before the HTTP slice. Those three files remained unchanged
through integration. They are separate source-scoped evidence, not an
extra combined run.
- Preserved failures: a helper import and test typing error were
corrected before the passed checks. An earlier suite failure did not
serialize its hook error and remains unexplained. Later runs captured a
missing normal managed-config initialization and HTTP 404 from the
loopback mapper. The fixture now calls the normal initialization owners
and uses the existing Host-preserving canonical HTTP adapter. All
original authority assertions and deadlines remain.
- Reproduce the local files with `pnpm --filter @paperclipai/server exec
vitest run src/modules/external-agents/cloud-principal-http.test.ts
src/modules/external-agents/cloud-principal-log.test.ts --maxWorkers=1
--no-file-parallelism`. The real integration file is
`server/src/__tests__/cloud-principal-normal-app.postgres.test.ts`; it
requires its explicit dedicated test DB and an own compiled matching
Cloud checkout.
- The normal `node scripts/check-maps.mjs` check passed for 35 areas and
192 capability IDs. This validates map structure, not business
acceptance. The later edits are documentation/map only; all executed
business source and own outputs from the paired case remained exact.
- Current-head remote CI passes: 59 successful checks and three
intentional skips. Greptile reviewed this exact head at 5/5 with no
findings or review threads. The prior Ponytail review was Ship 5/5 at
the earlier head; no current-head Ponytail turn is present, which is
nonblocking under the authorized review policy. The three
full-repository commands were not repeated locally; the focused normal
App graph and affected tests above passed, and required remote CI
supplies the broad gates. No deployed, provider-capacity, cold-wake or
full hosted lifecycle result is claimed.

## Risks

- This is a security boundary. Signature, current namespace and original
OAuth restrictions must continue to fail closed. The response gives
identity only; callers still need normal operation authority.
- The request carries the original token. The reserved route uses a
closed log projection on both accepted and refused requests. Privacy
regressions remain a material risk.
- Both repositories must agree on the action and digest contract. Older
code rejects the unsupported action; no fallback creates authority.
- This is a prerequisite private boundary. The remaining hosted
discovery, wake, capability, operation, event, retry/reactivation and
revocation lifecycle work, plus Muse, remains open. No profile or
capability is enabled here.
- No migration is needed. Rollback removes the private boundary and
paired caller; normal OAuth and Personal MCP remain with their existing
owners.
- The engineering documentation describes this private machine boundary
and its access restriction. No operator-facing feature is enabled.
Documentation for full hosted enablement remains with that later work.

## Model Used

OpenAI `gpt-6.1-sol` (Sol6.1), through Codex, produced the source and
tests with tool use and local compiler/test execution. A GPT-6-based
Codex coordinator reviewed the work. Its exact model ID is unavailable.
Context window sizes are unavailable.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-07 08:02:42 -06:00
DottaandPaperclip 6be71028ad Prepare online ephemeral enrollment through signed Cloud control (#19660)
## Thinking Path

> - Paperclip manages agent work through a control-plane web server.
> - A replacement web server needs shared durable data and the original
signing identity.
> - Existing enrollment exports that identity only after a global drain
and offline migration.
> - Online enrollment needs a separate preparation operation while agent
work continues.
> - This change adds a purpose-bound Cloud control operation with
existing data and key owners.
> - The result is an audited preparation receipt, with no permission to
stop agents or detach storage.

## Linked Issues or Issue Description

Refs #19038. Refs #19024.

**What are you trying to improve?**
Allow Cloud to prepare an already migrated tenant for volume removal
without the offline migrator or a global-idle gate.

**What problem does this solve?**
Cloud must preserve the original decision key before starting a
replacement with static ephemeral mode. A tenant session must never gain
key-export authority.

**What would the improvement look like?**
A fresh exact-action Cloud assertion obtains a private, deployment-bound
preparation receipt. Unfinished durable-data migration and invalid
replacement configuration block export.

## What Changed

- Add the signed `ephemeral-enrollment:prepare` control action and a
strict UUID-only POST route.
- Check existing durable-data eligibility and ephemeral configuration
before reading the original key. Never create a missing key.
- Audit only public preparation metadata. Keep HTTP diagnostics
secret-sensitive and the response non-cacheable.
- Bind the receipt to the attempt, stack, build commit, Railway
deployment, time, static mode and key digest.
- Always return `enrollmentReady: false`. Cloud still needs to arm a
replacement, verify old writers have stopped, repeat eligibility and
qualify native handoff before detaching the retained volume.

## Verification

- PASS: 80 focused tests across the preparation owner, signed route,
existing Cloud control, HTTP privacy and OpenAPI route coverage.
- PASS: 2 real PostgreSQL cases. Concurrent copies of one signed request
export once; the durable receipt still rejects reuse after clearing the
process fence. Normal run admission before skill pins is allowed;
missing durable skill installation remains blocked.
- Genuine prepatch RED: the same two PostgreSQL cases exported twice and
rejected ordinary admission. The fixed owner was restored automatically
after the comparison.
- PASS: server TypeScript check with workspace dependencies built.
- Fixed the first CI route-coverage failure by declaring this private
control endpoint in the existing explicit OpenAPI exclusion registry.
- Initial local HTTP execution lacked sandbox socket permission; the
reviewed run passed. Initial PostgreSQL fixtures enabled durable files
too late and reused an issue prefix; those fixture defects were
corrected before the red/green comparison.
- All required CI gates pass on
5346721847. Fresh Greptile review is 5/5
and all review threads are resolved. No deployment, volume detachment,
admission change or agent stop is performed by this PR.

## Risks

- The private response contains the original signing key. Only a fresh
stack-bound assertion for this exact action can obtain it. The Cloud
consumer must pass it directly to its existing vault owner and must not
expose it through admin responses or registry metadata.
- Eligibility is an observation. It cannot replace static writer
restrictions or qualified ownership handoff. This PR deliberately
supplies no detach or readiness authorization.
- Cloud transition integration is still required. This PR alone does not
enroll an instance.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, code execution and browser
tools. The host does not expose the exact serving model ID or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-06 18:00:28 -06:00
e482f0f632 fix(server): shed only retryable object-journal reads under load (#19176)
Client polling can exhaust the shared object journal queue and reject log persistence. Bound ordinary object-log and trace reads, keep persistence and internal verification queued, and return retryable 503 responses when reads are busy. Leave feedback exports unsent until every log page is available.

Validation: 72 latest-master affected tests and 10 workflow tests pass locally, with the workflow fixture passing strict TypeScript checks. The preceding broader run passed 280 server tests with two opt-in S3 skips. All applicable checks on d0a937768d are green (56 success, three skipped); exact-head Apex review is 5/5 with no unresolved findings. Background persistence still has no fixed queue bound during a prolonged storage outage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-06 13:47:25 -08:00
Devin FoleyandClaude Opus 5.5 2c7ee84221 fix(server): stop reporting work cancelled by a disconnected client (#19037)
A route that aborts its storage read when the response closes rejects with
AbortError after the client has gone. The error handler reported that
cancellation as a server crash. Skip the error sinks for an AbortError when
the response was closed before it finished; report it otherwise.

Fixes PAPERCLIP-BACKEND-5A

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-06 00:39:27 -08:00
StefanandArthur Dent 131c0c4609 fix(server): redact the credentialValues envelope in HTTP logs (#13742)
> **Attribution:** this pull request was written and opened by the
Paperclip agent **Arthur Dent** (Platform & Reliability Engineer,
Tharsia) from an automated run under the host GitHub credential. GitHub
records the author as `stefanriegel`; Stefan Riegel did not write it. No
other edit was made to this description.

## Thinking Path

> - Paperclip is the open source app that people use to manage AI agents
for work.
> - The server writes request details to the user journal. A middleware
module names the fields that are secret.
> - The connect endpoint accepts a `credentialValues` object. The object
maps a provider config path to a secret string.
> - One provider config path is `credentials.authorization`. The caller
chooses the key names. Therefore the key is one flat string, and it is
not a nested path.
> - The redaction path list did not cover the `credentialValues`
envelope. The recursive redactor compares keys by exact match, so
`credentials` and `authorization` in the key set did not match the flat
key `credentials.authorization`.
> - A failed connect request therefore wrote a GitHub token to the
journal in cleartext.
> - This pull request redacts the `credentialValues` envelope in both
layers.
> - The benefit is that no provider secret reaches the journal, for any
key name that a caller chooses.

## Linked Issues or Issue Description

No public issue exists. The report follows the bug report template.

**What happened?**

A `POST /api/companies/{companyId}/tools/apps/connect` request returned
HTTP 502. The server logs request bodies for non-2xx responses. The
journal line contained a GitHub fine-grained personal access token in
cleartext. The token was 93 characters long and started with
`github_pat_`. The line contained no redaction marker.

The logged body had this shape:

```
"reqBody":{"galleryKey":"github","connectionMethodKey":"mcp-key","name":"GitHub",
  "credentialValues":{"credentials.authorization":"<TOKEN>"}, ... }
```

**Expected behavior**

The journal line shows a redaction marker in place of the token. No
cleartext credential reaches the journal on any response status.

**Steps to reproduce**

1. Run a Paperclip server on version 2026.916.0.
2. Send a `POST` request to
`/api/companies/{companyId}/tools/apps/connect`.
3. Put a secret under `credentialValues` with the key
`credentials.authorization`.
4. Make the request fail, so that the server returns a non-2xx status.
5. Read the journal line for the request.

Result: the secret appears in cleartext. The two redaction layers both
miss it.

**Paperclip version or commit**

2026.916.0. Reproduced against `master` at
`c65fc9e3c81c41aafe421aa90a00514b84343285`.
The two middleware files are unchanged at `master`
`0f5fafe16b069b8f0c7f17eb9080d010aa6cd513`, so the defect is present at
the current head of `master` as well.

**Deployment mode**

Self-hosted.

**Installation method**

npm package.

**Access context**

Authenticated user, company-scoped route.

**Relevant logs or output**

Token value removed. The journal line has the shape above.

## What Changed

- Add `reqBody.credentialValues` to `HTTP_LOG_REDACT_PATHS` in
`server/src/middleware/http-log-redaction.ts`. This redacts the whole
envelope at the pino serialization boundary.
- Add `credentialvalues` to `SENSITIVE_KEYS` in
`server/src/middleware/redact-sensitive.ts`. This redacts the envelope
by name in the recursive pass.
- Add a test that sends a `credentialValues` envelope through a real
pino logger with the production path list. The test asserts that the
secret is absent.
- Add a test for the flat-dotted-key shape that leaked, and for the
nested shape as a regression.
- Add a test that the envelope is redacted below the top level.

The key-set change is the durable half. The path list only covers body
shapes that someone has already seen leak. The recursive pass matches
the envelope by name for any shape.

## Verification

Run these commands:

```
npx vitest run server/src/__tests__/http-log-redaction.test.ts server/src/__tests__/redact-sensitive.test.ts
```

Against the patched source, both files pass: **72 tests passed, 0
failed**. That is 54 tests in `http-log-redaction.test.ts` and 18 tests
in `redact-sensitive.test.ts`.

The same two files were then run against the **unpatched** source, which
is the parent commit of this branch. **Four tests fail there.** Those
four are exactly the tests this pull request adds:

- `http-log-redaction.test.ts` — "defines the HTTP auth and cookie
header paths that must be redacted". The path list does not contain
`reqBody.credentialValues`.
- `http-log-redaction.test.ts` — "redacts a credentialValues envelope at
the serialization boundary". This is the leak: a real pino logger writes
the token in cleartext.
- `redact-sensitive.test.ts` — "redacts the connector credentialValues
envelope on an app-connect body".
- `redact-sensitive.test.ts` — "redacts a credentialValues envelope
nested below the top level".

The four failures show that each half of the change is load-bearing, and
that both layers failed, not one. The control run also confirms that the
68 tests which pass on the unpatched source still pass on the patched
source, so the change regresses nothing that already held.

The tests assert that the secret string is absent, and they do not pin a
redaction marker. This is deliberate: the two layers write different
markers. `fast-redact` writes `[Redacted]`. `redactSensitive` writes
`[REDACTED]`. An assertion that requires `[REDACTED]` alone fails on a
correct path-list-only fix.

## Risks

Low risk. The change adds two redaction paths. It removes no existing
path.

One behavioral shift is intentional. The recursive pass now replaces the
whole `credentialValues` subtree with one marker, instead of replacing
individual leaves. Any non-secret sibling field placed inside
`credentialValues` is also redacted. This is correct, because the
envelope holds only provider secrets by definition.

One adjacent pull request, `#10784`, adds header paths to the same array
in `http-log-redaction.ts`. The two changes touch different regions of
the array. The second one to merge needs a trivial rebase at most.

## Model Used

- Provider and model: Anthropic Claude, via the Paperclip `claude_local`
agent adapter
- Capability details: tool use and code execution in a sandboxed
workspace
- The patch was authored and verified by automated agent runs

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [ ] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

Left unchecked on purpose: the CI and Greptile items cannot be true
before the pull request exists.

"I have run tests locally and they pass" stays unchecked as a
whole-workspace statement. The two test files named under Verification
were run locally, against both the patched and the unpatched source, and
the numbers above are from those runs. The full workspace suite has not
run.

## Related pull requests

Refs `#10784`, `#6373`, `#8841`.

Co-authored-by: Arthur Dent <arthur.dent@paperclip.local>
2026-11-05 14:39:15 -06:00
Devin FoleyandClaude Opus 5.5 c56aa689ac fix(server): say where secret-sensitive requests fail in Sentry (#19040)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The server sends unexpected 5xx errors to Sentry through the shared
Express error handler
> - Some routes carry credentials: chat setup, provider webhooks, MCP,
company observability settings, Muse, avatars, and similar routes
> - On these routes the handler replaces the error with a generic
`Secret-sensitive request failed` error, because the original name and
message can contain secrets
> - That replacement also removes every useful fact. The Sentry event
does not show the route, the status, the error type, or the throw site,
so a maintainer cannot find the cause
> - This pull request adds a small fixed-shape record to that generic
error. The record says where the failure came from, and it never copies
request data or error text
> - The benefit is that the next event on these routes points to a route
and a code location, while the privacy rule stays the same

## Linked Issues or Issue Description

**What happened?**
A hosted staging server reported one event with the message
`Secret-sensitive request failed`. The event had only the stack of the
sanitizer itself. It had no route, no status, no error type, and no
frame from the original error. The trace had no spans and no logs. A
maintainer cannot find the root cause from this event.

**Expected behavior**
The event keeps the generic message, so no secret leaves the process.
The event also tells the maintainer which secret-sensitive route family
failed, the HTTP method and status, the error type, a safe error code
when one exists, the matched route definition, and the original throw
site.

**Steps to reproduce**
1. Make a route handler throw on a secret-sensitive path, for example
`PUT /api/companies/:companyId/observability`.
2. Enable the backend Sentry DSN.
3. Look at the captured event. It shows only `Secret-sensitive request
failed` and the sanitizer stack.

**Paperclip version or commit**
`master` at e107a75755.

I searched open and closed pull requests for `Secret-sensitive`. I did
not find a pull request that changes these diagnostics.

## What Changed

- `server/src/middleware/http-log-policy.ts`: add
`secretSensitiveRouteFamily()`. It returns a fixed label for the policy
branch that matched, or `null`. `isSecretSensitiveHttpRequest()` now
calls it. The matching rules do not change; the combined `/mcp/runner`
and avatar pattern is split into two branches with the same rules.
- `server/src/middleware/secret-sensitive-request-error.ts` (new):
`SecretSensitiveRequestError`. It keeps the generic message, `name:
"Error"`, its own stack, and default grouping. It adds frozen
diagnostics:
  - `routeFamily`: from the closed list above.
  - `method`: one of the standard HTTP methods, or `other`.
  - `status`: the 5xx `HttpError` status, or 500.
- `errorKind`: `http_error`, `type_error`, `range_error`,
`reference_error`, `syntax_error`, `error`, or `non_error`.
- `errnoCode`: only a key of `os.constants.errno`. `databaseErrorCode`:
only a five-character SQLSTATE. Both are read from own data properties
of the error and at most three causes. Getters are not called.
- `routePattern`: the matched Express route definition
(`req.route.path`), only when it is a string in a narrow character set.
A RegExp route becomes `null`.
- `originFrames`: up to eight complete V8 frames from the original
stack. The extractor first removes the exact `name: message` header that
V8 writes. It rebuilds the header like `Error.prototype.toString` from
data properties only and requires `stack` to start with it, so a
multiline message can never pass as a frame. If the header cannot be
identified, no frame is kept. Each remaining line must match a strict
frame pattern: a code-shaped function name and a location that may
contain spaces, Windows drive or UNC paths, or a percent-encoded
`file:///` URL.
- `server/src/middleware/error-handler.ts`: the sanitizer now returns
this error. The HTTP response, the request log context, and Telemetry do
not change.
- `server/src/sentry.ts`: `captureException` sends these diagnostics as
`secret_route_family`, `http_method`, `http_status`, `error_kind`,
optional `errno_code` and `db_error_code` tags, and a
`secret_sensitive_request` context. Run tags for agent requests stay.
The fingerprint stays `{{ default }}`. The module header documents the
new event fields next to the other Sentry privacy notes.
- `server/src/__tests__/secret-sensitive-request-error.test.ts` (new):
unit tests for each bounded field, the header-first frame extraction
(multiline-message canary, unidentifiable header, paths with spaces and
Windows paths), the route-family policy, and an end-to-end test through
a real Express app, the error handler, and the Sentry gate.

## Verification

- `cd server && pnpm vitest run
src/__tests__/secret-sensitive-request-error.test.ts
src/__tests__/http-log-policy.test.ts src/__tests__/sentry.test.ts
src/__tests__/error-handler.test.ts
src/__tests__/agent-commentary.integration.test.ts
src/__tests__/agent-profile-avatar.test.ts`: 6 files, 108 tests passed,
5 skipped.
- Red before green: with the old `sentry.ts` and the new test, the
end-to-end test fails because the event has no diagnostics tags. With
this change, it passes. The three frame-extraction tests added for
review feedback fail against the previous extractor and pass now.
- The end-to-end test puts a canary credential in the query string, the
JSON body, the error name, the error message, a cause message, and
`HttpError.details`. It checks that no Sentry argument and no reported
stack contains the canary. It also checks that the HTTP response stays
`{ "error": "Internal server error" }` with the original status.
- The existing tests in `error-handler.test.ts` for chat setup and
observability credentials still pass without change. They check that the
reported error, the error context, and Telemetry carry no canary.
- Server typecheck with `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 node
scripts/typecheck-layers.mjs`: no error in server files. My sandbox had
only the server dependency subset installed, so the layers also reported
missing-module errors in `cli/`, `ui/`, and `packages/`. These errors
are not related to this change. CI runs the full check.
- Note for local runs: an empty `ANTHROPIC_BASE_URL` makes
`sentry.test.ts` fail at import on `master` too. Unset it before you run
the suite.

## Risks

- Low risk. Sentry stays opt-in, and the change affects only events from
secret-sensitive routes.
- Privacy: the new values are closed labels, bounded numbers,
source-code route definitions, Node errno names, SQLSTATE codes, and V8
frames that match a strict pattern after the message header is removed.
A frame contains a function name and a file location. In theory, code
can give a function a name from data. The frame pattern accepts only
identifier-like names of at most 120 characters, and the event never
includes the error name or message.
- If code changes an error's `message` or `name` after its stack was
formatted, the header no longer matches and the event has no origin
frames. This fails closed.
- `req.route` holds the last route that the router dispatched. If a
later middleware fails after a route calls `next()`, `routePattern` can
name that earlier route. The route family is still correct because it
comes from the URL policy.
- Grouping does not change. The reported error keeps the same type,
message, and sanitizer stack, so existing Sentry issues keep collecting
events.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), used
through Claude Code with tool use (shell, file edits, test runs).
Extended reasoning was not explicitly enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-05 11:14:00 -08:00
Devin FoleyandClaude Opus 5.5 3c7af35103 fix(errors): return tool gateway rejections with their own status (#19158)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents call connected tools through the tool gateway. Some tool
calls need a human review before they run.
> - A user approves a tool review from a task card. The approve route
calls the gateway.
> - The gateway can reject the approval with `ToolGatewayHttpError(409,
"approved_tool_target_changed")`. This is an expected rejection: the
tool or connection changed after the review.
> - `ToolGatewayHttpError` does not extend `HttpError`. The shared error
handler only keeps the status of an `HttpError`. Thus the route answers
500 "Internal server error" and sends the rejection to Sentry as a
crash.
> - The user does not see the reason. The user clicks Approve again, and
each click makes one more crash report. One user made 69 reports on
staging this way.
> - This pull request makes the shared error handler keep the status and
reason code of client (4xx) gateway errors.
> - The benefit is a clear 409 with a reason for the user, and no false
crash reports.

## Linked Issues or Issue Description

Refs #12775 (related community PR. It changes `ToolGatewayHttpError` to
extend `HttpError`, which affects every `instanceof HttpError` check.
This PR is smaller and keeps gateway server failures on the crash path.)

**What happened?**
A user approves a tool review on a task card. The tool definition or
connection changed after the review. The API answers 500 "Internal
server error". Sentry records the expected rejection "Tool definition or
connection changed; request a new review" as a server crash. Each retry
records one more event.

**Expected behavior**
The API answers 409 with `code: "approved_tool_target_changed"` and the
message "Tool definition or connection changed; request a new review".
Sentry does not record a crash.

**Steps to reproduce**
1. Make a connected tool that needs approval, and start a tool call so
that a review card appears.
2. Change the tool connection (for example, its config) before you
approve.
3. Click Approve on the task card (`POST
/api/issues/:id/interactions/:interactionId/accept`).
4. See 500 "Internal server error" and a new Sentry event from
`approveActionRequest`.

**Paperclip version or commit**
`master` at `482ec0aea9ab6f2684baaa1487f2c7a2eedf009e`.

## What Changed

- `server/src/middleware/error-handler.ts`: the handler converts a
`ToolGatewayHttpError` with a 4xx status to an `HttpError`. The new
error keeps the status, the message, the gateway details, and the reason
code as `code`. The original stack stays.
- Gateway errors with a 5xx status do not change. They still go on the
crash path, answer 500 "Internal server error", and go to Sentry. Thus
provider text in those messages stays hidden.
- `server/src/__tests__/error-handler-tool-gateway.test.ts`: new tests
for the 409 mapping, for the unchanged 5xx path, and for an Express
route that lets the gateway error escape.

## Verification

- `cd server && npx vitest run
src/__tests__/error-handler-tool-gateway.test.ts
src/__tests__/error-handler.test.ts` passes: 21 tests.
- Without the fix, the new tests fail with `expected 500 to be 409`.
With the fix, they pass.
- `src/__tests__/issue-thread-interaction-routes.test.ts` and
`src/__tests__/json-parse-error-handler.test.ts` also pass: 175 tests in
4 suites with the two error handler suites.
- I checked the runtime import graph of `services/tool-gateway.ts` (685
modules). It does not import `middleware/error-handler.ts`, so the new
import makes no cycle.
- I did not run the full server typecheck locally, because it ran out of
memory in an 8 GB sandbox. CI must do this check.

## Risks

- Low risk. Only gateway errors with a 4xx status change. Before, they
became 500. Now they keep their own status.
- Some routes outside the gateway router can now return gateway 4xx
details in the `details` field. The gateway routes already return the
same details through `sendGatewayError`.
- `middleware/error-handler.ts` now imports `services/tool-gateway.ts`.
Unit tests that import the error handler load more modules. The app
already loads this module.
- This PR does not fix the cause of some of these rejections. A routine
OAuth token refresh changes values in the signed approval snapshot
(token secret versions, the order of grant secret refs, and token expiry
times). This can make a pending review fail. That fix changes
`services/tool-gateway.ts` and comes in a separate PR.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used as a coding agent
with tool use (shell, file edits, test runs, and GitHub API).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
documentation change is necessary)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-04 18:14:52 -08:00
DottaandPaperclip 5877d751c0 feat(server): bind idle credential eligibility to incoming wake proof (#18639)
Reviewed and verified at d1e3afa4af.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-03 14:30:07 -06:00
DottaandPaperclip b8f8b904fa feat(routines): preserve automation ancestry through webhooks (#18772)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Routines let external services start agent tasks through a normal
webhook.
> - A repair routine must recognize failures from work that it created.
> - The receiver added in #18714 put Sentry payload parsing and repair
policy in core routine intake.
> - This pull request replaces that receiver with verified creation
provenance on normal routine calls.
> - Sentry can filter the resulting tags before it sends a normal
routine webhook.
> - The benefit is reusable provenance and loop prevention without a
provider-specific route.

## Linked Issues or Issue Description

**Problem or motivation**

A Sentry repair alert can start another repair when its own work fails.
The dedicated receiver in #18714 solved this through a Sentry-specific
payload contract. Ordinary routine webhooks also need a trusted way to
retain the automation that created their tasks.

**Proposed solution**

Use the existing public `/fire` route and its current authentication. An
optional source run credential proves a live Paperclip caller. Store the
caller separately from webhook JSON. Follow recorded task creation links
and export bounded ancestry tags on run-failure events. Let Sentry alert
filters exclude repair-origin work.

**Alternatives considered**

Keep the dedicated Sentry receiver, accept sender-supplied origin IDs,
or filter only the immediate routine. The first couples core intake to
one provider. The second accepts unverified claims. The third misses
child tasks, standalone follow-ups, and calls through another routine.

**Roadmap alignment**

This improves the shipped Scheduled Routines and Activity log
capabilities. It does not add a new scheduler or incident system.
Related work: #18714 is the implementation replaced here; #9727 adds
source-task navigation and UI attribution; #10206 proposes exception-led
routine dispatch. This PR uses existing task creation records and does
not implement either of those broader proposals.

## What Changed

- Remove the Sentry-only route, parser, authentication exception, and
payload-specific tests and documentation from #18714.
- Add optional `X-Paperclip-Source-Run-Token` support to normal routine
webhooks. Validate the standard run JWT, company, instance, agent, live
run, and cancellation state. Authenticated routine API calls also retain
their validated caller.
- Add nullable `routine_runs.created_by_run_id`. Keep the reference
after source-run deletion so missing history becomes unknown. Separate
callers during coalescing, including legacy open tasks.
- Follow company-scoped routine receipts, creator runs, task parents,
and legacy creation activity. Mark missing, foreign, cyclic, private, or
excessive ancestry for safe alert filtering.
- Add event-local `automation_origin_status`, `automation_routine_ids`,
and `task_shareable` tags through the existing Sentry SDK hook. Cap
ancestry at 32 reads and share a 750 ms deadline across metadata and
ancestry lookup. Keep the SDK one-second wait and eight-lookup
concurrency bound. Redact the source credential from HTTP logs and
Sentry request headers.
- Preserve unexpected database failures as server errors without
rejection bookkeeping in an aborted transaction. Document 409, 415, and
422 responses for the normal webhook.
- Keep `routineScopedRunId` as generic opt-in
`X-Paperclip-Idempotency-Scope: routine` support. Default trigger
receipts and timestamped HMAC replay protection retain their current
scope.
- Add [the Sentry repair
tutorial](https://github.com/paperclipai/paperclip/blob/codex/routine-creation-provenance/docs/guides/board-operator/sentry-run-failure-repairs.md),
API documentation, OpenAPI headers, and capability-map updates.

## Verification

At commit `0f4a0051528edd1367a921a76a8050c3fd087872`:

- `pnpm -r typecheck` and `pnpm build` passed for the full workspace.
Migration numbering and safety checks passed.
- The focused service, route, Sentry, redaction, and OpenAPI suites
passed 228 tests in five files. The database Sentry resolver and OpenAPI
invocation passed 33 tests in two files. These are separate invocations
with some overlap.
- The full routine HTTP suite passed all 15 tests. The authenticated API
case failed before the caller-wiring fix and passed after it. The
fixture loads the real route-owned services. No test deadline was
raised.
- The real `@sentry/node@10.71.0` suite passed all 12 tests. It uses an
in-memory transport and sends no events to Sentry. It covers terminal
failures, saved-message interruptions, event isolation, and source-token
redaction.
- Regression tests also cover forged, expired, foreign, and cancelled
caller credentials; payload spoofing; inherited routine ancestry; legacy
coalescing; deleted history; private work; cycles; query and time
limits; database error causes; and routine-wide receipts. The new error
and budget cases failed before their fixes.
- `node scripts/check-maps.mjs` and `git diff --check` passed. The map
check covers 35 areas and 192 capabilities.
- The full local Vitest command was stopped for a required master merge
after four chat shards passed 1,278 tests. No full local pass is
claimed. An earlier combined HTTP run hit an existing malformed-ID
timeout whose original cause is not established; the final full HTTP
suite passes. An extra local run of the unchanged heartbeat scheduling
file passed 12 of 13 tests and failed a readiness-status assertion. Its
logs also reported a missing fixture user membership. The same 13-test
file now passes in current-head CI.
- All GitHub Actions checks pass on this commit. Three jobs initially
lost their self-hosted runners. One server shard initially failed while
deleting a company in an unchanged heartbeat test because of a
`company_skills` foreign key. One retry passed all affected jobs without
code changes. That server shard passed all 499 tests, and all eight
browser shards passed.
- Fresh Greptile review is 5/5 on this commit. All six review threads
are resolved. The separate Paperclip Review check remains queued. Its
bot reports that work is paused. A maintainer must resume the assigned
review work to complete that check. No full green check state is claimed
while this external gate is pending.
- Live Sentry configuration and deployment are outside this PR's
acceptance evidence. The tutorial describes setup, a controlled failure,
and a repair-origin exclusion check for operators to run after
deployment.

## Risks

- Normal app webhooks retain the sender's full JSON in company-readable
routine history and the execution task. Alert filters and task
visibility are not content scrubbers. Operators must scrub sensitive
event data in Sentry before forwarding it and keep connection tests in
setup mode.
- Existing users of `/sentry-run-failures` must move to `/fire` and add
the documented Sentry filters before enabling repair intake. The old
route is removed. Stored tasks and receipts remain.
- The additive migration uses `ADD COLUMN IF NOT EXISTS`. It does not
rewrite historical tasks or infer missing provenance.
- Unknown or unshareable ancestry is excluded by the tutorial's alert
rule. Missing history, an enrichment timeout, or more than five routine
IDs can therefore suppress a repair webhook while retaining the Sentry
error.
- An external webhook without a verified source credential starts a new
receiving-routine origin. Its JSON cannot establish upstream Paperclip
ancestry. Sentry must not store a Paperclip run credential.
- Routine-wide receipt reuse needs a stable sender key. A static Sentry
custom header is not a per-event key. The tutorial keeps delivery
deduplication separate from loop prevention.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

OpenAI Codex, GPT-6 model family, with reasoning, repository access,
code editing, and test execution. The session does not expose the exact
model variant or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 09:33:17 -06:00
Dotta 5dee4993f1 fix(server): distinguish aborted HTTP responses in access logs (#18781)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use HTTP logs to investigate task updates and database
contention.
> - Node starts each response with status 200 before it sends any
headers.
> - The custom HTTP formatter printed that status when the client
disconnected before a response.
> - This made an aborted request look like a successful task update.
> - This pull request labels incomplete responses as aborted and
preserves error severity.
> - Operators can distinguish HTTP delivery from database commit
evidence.

## Linked Issues or Issue Description

- Refs: #18753.
- Related: #14369 removes writes from legacy comment attribution reads.
This change addresses a separate logging defect.

The missing-write incident in #18753 remains unconfirmed. We reproduced
a misleading `PATCH /issues/:id 200` log when the client received no
HTTP response. We did not reproduce an HTTP 200 delivered before the
mutation committed.

## What Changed

- Label premature connection closure as `request aborted before
response` or `request aborted during response` in both success and error
log paths.
- Log aborts as warnings, including on routes whose successful polling
logs are normally silent. Keep explicit errors and 5xx responses at
error severity.
- Keep private webhook paths and query values out of aborted-request
logs.
- Add real HTTP disconnect coverage and real PostgreSQL task-update
tests with synthetic data.
- Document that HTTP completion logs do not establish database commit or
rollback.

## Verification

- Baseline: the new HTTP disconnect test failed because the log said
`PATCH /api/issues/synthetic-task 200` at info level, while the client
received no response.
- `node node_modules/vitest/vitest.mjs run
server/src/__tests__/http-log-aborted-response.test.ts
server/src/__tests__/http-log-redaction.test.ts
server/src/__tests__/issue-update-lock-contention-postgres.test.ts`
passed: 81 tests in 3 files.
- `pnpm -r typecheck` and `pnpm build` passed. The build used a second
clean clone at the exact PR commit.
- GitHub CI completed at `e3e96d836119df5fa1c7f22615484265de5aa700`: 52
checks passed and two Storybook checks were skipped. Greptile returned
5/5 with no findings.
- Local `pnpm test:run` passed all six chat shards (1,834 tests). Its
general server stage then reported four failures. We stopped the run
after preserving these results and comparing the failing cases. The
later workspace and serialized stages were not completed locally. The
full local suite is not green.
- The four observed failures were the two OAuth concurrency cases in
`tool-access-service.test.ts`, the bounded workspace conflict case in
`workspace-runtime.test.ts`, and the `enableNativeRunner`-disabled
stop-control case in `muse-control.test.ts`.
- Comparisons used the same sanitized environment on unchanged master
`2b14ec03a48f17606db53fd7fc0edc53bc930409` and this PR. Both OAuth cases
pass in isolation on both revisions. The full tool-access file passes
all 498 tests on the PR; master has a different 15-second timeout, with
497 tests passing. The workspace case passes on the PR but fails
port-reservation setup on master. The Muse case passes on both. These
results do not explain or clear the original broad-suite OAuth and Muse
failures.
- Tested on macOS arm64, Node.js 26.4.0, pinned pnpm 9.15.4, and
disposable embedded PostgreSQL. The API fixture uses a synthetic
process-adapter agent. No model provider runs and no production database
or real credentials participate.
- The PostgreSQL fixture verifies an actual lock waiter with
`pg_blocking_pids`. A blocked issue update does not return 200 before
release. After release, status, monitor policy, inline comment, and
audit receipt are persisted before the completed response.
- Holding the company row with `FOR NO KEY UPDATE` does not block these
fixture writes. They return 200 with persisted data while the compatible
parent-row lock remains held.
- An injected database trigger failure returns 500 and leaves the issue,
comment, and mutation audit unchanged. Two influence-attempt receipts
remain under the existing documented cap policy.
- The combined lock/disconnect/failure test verifies no delivered
response, an aborted log, and rollback after an injected database
failure. The injected failure is not evidence of the reporter's
production root cause.
- Source inspection at release `v2026.1023.0`
(`1b4c803816e82b0265c8422e1a804327a7ed8c54`) finds the same
unconditional success formatter and awaited issue-update transaction.
That release was not runtime-tested.

## Risks

- Log consumers that parse the literal method/path/status message must
handle the aborted-request message.
- Client disconnects during streaming or SSE responses also use the
aborted warning. Expected disconnects may therefore produce more
warnings.
- A disconnected request may still commit after the client leaves. This
change reports transport completion and does not cancel or retry
mutations.
- This PR does not close #18753. The original missing-write cause and a
received-200-without-commit scenario remain unconfirmed.
- Local full-suite failures remain unresolved despite green CI. Keep
this PR in draft until that verification gap is accepted or resolved.

> This is a targeted bug fix. ROADMAP.md does not list overlapping
feature work.

## Model Used

- OpenAI GPT-6 through Codex, with reasoning, tool use, code execution,
and parallel subagent investigation.
- The exact runtime model ID and context window size were not exposed to
the agent.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [ ] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge
2026-11-03 09:26:03 -06:00
DottaandPaperclip 27d35314be feat: connect experimental Cursor Cloud to native Runner (#18069)
## Thinking Path

> - Paperclip helps people manage AI agents at work.
> - Native Runner owns execution and task authority.
> - Operators need an explicit native Cursor Cloud choice.
> - Tools/files need company, task, session and run scope.
> - This connects setup and lifecycle to Runner Core.
> - Legacy agents retain their selected runtime/history.

## Linked Issues or Issue Description

Depends on #18068. Managed-only API placement and API-owned Stop
containment are in #18310. Refs #5664, #11803, #17299, and #17471. No
duplicate native integration was found.

## What Changed

- Add explicit native setup, Test, saved settings and dispatch with
no-prompt account/catalog/callback checks.
- Scope MCP to company, agent, task, session and run through
authenticated HTTPS. Register verified artifacts/work products; preserve
unpriced accounting, completion and budgets.
- Connect Question/Plan continuation, cold handoff and admitted fresh
sessions; recheck waiting admission under the task lock.
- Present audited Stop intent while preserving physical containment
proof; bound typed Stop metadata. Preserve title edits through
preference/navigation readiness.
- Select one live/history representation per run to prevent duplicate
settled cards. Integrate upstream task/read-state/retry/schema contracts
and repair two test fixtures.

## Verification

Core #18068 merged normally as `3910352c923a`. Product head
`5a60f541c982` has passing repository CI/build/browser checks, including
required `ci / verify` and `ci / e2e`, plus a fresh exact-head Greptile
5/5 review with zero unresolved threads. Two non-required external app
checks remain pending: Paperclip Review is queued waiting for its
assigned agent; Superagent Security Scan is reviewing the diff. These
are not claimed completed. The normal merge endpoint enforces the
repository rules; no policy or status override is used. The dependent
Top PR is addressing its fresh deferred-Stop review finding.

Product `5a60f541c982` passed 270 UI and 20 selected
Cloud/native/disk-full tests, UI/server typechecks, tokens and maps.
Top’s clean normal merge passed tokens/maps/diff checks; its matrix is
pending. Earlier receipts retain Core’s 47 Rust, 4 compiled-native
retirement and 19 settlement controls, normal Runner build, and runtime
9af build/recursive typecheck. Reproduced loading/shutdown CI fixture
failures passed 7/7 browser and 121/121 unit tests, including a
broken-unsubscribe negative control. No timeout/oracle was relaxed.

The actual-parent restack adopts 75 mainline paths, including
schema/permission and generic runtime changes, beyond the prior two
fixture repairs and client projection fix. The full difference from
hosted `9af1a5de42a2` is therefore larger than four files; backend
source also differs. The merged config-read checks match Core391, and
the Cursor placement/runtime/relay/selection modules match frozen9af;
selected affected regressions pass. Paid proof keeps its actual 9af
source/image identity. The separately tested live/history card fix is
not claimed deployed on that host.

On 9af, original saved settings survived reopening and fresh no-prompt
Test authenticated catalog/model/binding; no new Save is claimed. Fresh
Linux admission passed. With `claude-sonnet-4-5`, Artifact downloaded
exact 27 bytes and matched 121 canonical identities in complete
S3-served history. First Stop’s strict quiet FAILED on a 10.373-second
gap despite independent cancellation/no late artifact. One targeted
unchanged-sampler retry retained all 24 samples: 35.076 seconds quiet /
maximum gap 4.584 seconds, 67 ordered canonical identities, and separate
CANCELLED/artifact-absent status after the original 180-second deadline.
The pre-reload duplicate settled card is disclosed. Exact-ID cleanup for
the three new provider agents is awaiting authorization, so full grades
and the fresh Artifact’s post-cleanup download remain blocked. A
historical deleted-provider artifact was newly downloaded on 9af;
original F226 Question/Plan/reconnect and cleanup proof keep their
identities.

Retained failures: stale fixture 132/133; E8 48/135 with 87 reset-hook
failures; Mac 17 SQL failures/skips; earlier
title/startup/custody/Stop/deletion failures with unknown causes/bounded
recoveries; `dea023` Stop’s warning/50 duplicate identities; pre-repair
ACK and original Core CI failures. WSP’s 251-assertion, 7/8-suite
attempt failed physical PostgreSQL teardown; the unchanged wrapper rerun
passed 128. Retry alone establishes no cause or full local-suite pass.
All paid attempts remain recorded; vendor dollar cost is UNKNOWN.

## Risks

Experimental: Cursor independently acknowledges neither the effective
model nor instruction digest before the first prompt. Explicit values
are submitted and checked through behavior; unpriced provider dollar
cost stays UNKNOWN.

Callbacks need authenticated reachable HTTPS. Scope, revocation,
budgets, completion and cleanup remain authoritative. Operator Stop
intent is separate from physical containment. Saved legacy agents retain
their selected runtime. Prior shared title-save/deletion failures remain
disclosed.

## Model Used

OpenAI Codex, GPT-6 family, assisted with reasoning, repository tools,
code execution, browser verification, and delegated code review. The
serving model’s exact API ID and context window are not exposed. No
value is inferred from the model catalog.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used and the limits of available model
metadata
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have linked public issues or described the issue in this PR
- [x] I have not referenced internal or instance-local Paperclip issues
or links
- [x] My branch describes the change and contains no internal ticket ID
- [x] I have run the scoped local tests reported above; pending gates
are stated separately
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation
- [x] I have considered and documented the risks
- [x] All required CI gates pass on the current head
- [x] Fresh Greptile is 5/5 with no unresolved findings on the current
head
- [x] I will address all reviewer comments before requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-03 09:01:39 -06:00
DottaandPaperclip 1e3ab690b0 fix(routines): prevent recursive Sentry run repairs (#18714)
Validate Sentry failed-run alerts before routine dispatch. Reject repair-generated provenance and private sources. Deduplicate source runs through stable primary-key receipts, including setup retries and suppressed deliveries.

Verified with 166 focused tests, server typecheck and build, generated-reference checks, passing required CI, and Greptile 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-02 21:37:11 -06:00
DottaandPaperclip a92f616cd0 fix(db): attribute bounded captures to execution owners (#18634)
## Thinking Path

> - Paperclip manages agent work through shared server and database
services.
> - High query volume can spend main-thread time on SQL construction and
result decoding.
> - The existing bounded response capture counts rows and bytes, but it
cannot name the operation that executed a query.
> - Bundled callers can be missing, and measuring every parsed JSON
value adds work to the diagnostic.
> - This pull request carries a semantic execution owner into each
measured query and bounds response sizing.
> - Operators can rank query volume by owner before choosing a
performance fix.

## Linked Issues or Issue Description

Refs #17504, #17522, #17537, #17564.

**What existing behavior does this improve?**

The opt-in `PAPERCLIP_DB_RESPONSE_CAPTURE_WINDOW` diagnostic.

**Subsystem affected**

Database diagnostics and existing server execution boundaries.

**Current behavior**

The capture groups by table, operation, and builder caller. It cannot
distinguish execution owners that reuse a prepared query. Bundled
deployments need source maps for caller locations. The capture visits
every returned value and serializes parsed JSON again to estimate its
size.

**Proposed behavior**

Version 2 captures use fixed semantic owners for HTTP requests, delivery
queues, execution phases, and budget work. They count completed calls,
returned rows, errors, and completion wall time. They sample successful
results 1, 17, 33, and so on, with fixed cell and string/byte limits.
Complex values remain explicitly unmeasured. Semantic ownership works
without source maps.

**Reason and benefit**

A short census can identify which operations issue the most queries or
return the most rows. It preserves each query's execution owner through
pooled connections, concurrent operations, and prepared-query reuse. It
reduces diagnostic work without claiming per-query CPU attribution.

**Breaking changes**

Opt-in capture output moves from `entry.v1`/`summary.v1` to
`entry.v2`/`summary.v2`. Sampled primitive byte lower bounds replace
all-result byte estimates. Consumers must use the new fields and must
not extrapolate them into a total. There is no database migration or
application API change.

## What Changed

- Added `withDatabaseOperation(owner, action)` to the DB package. The
server owns semantic labels. Queries retain execution ownership in their
completion closure.
- Added fixed owners at existing HTTP, delivery, budget, and execution
timing boundaries. Delivery work replaces the context of the producer
that woke it.
- Added complete call/row/error counters and completion wall durations
within the existing bounded window. Durations include waits and are not
CPU measurements.
- Sampled and bounded primitive-value sizing. Complex and oversized
values remain unmeasured. Known owners skip repeated stack extraction.
- Preserved the deadline, group limit, paced output, query outcomes, and
existing coverage exclusions. Updated the operator documentation.
- Added real PostgreSQL owner-isolation and prepared-query tests, server
scope tests, and plain bundled-runtime proof.
- Moved projection-test and benchmark full-response sizing into an
explicit test helper. Existing payload-size thresholds remain unchanged;
these synthetic checks do not use the production sampler.

## Verification

- Final-commit DB capture suites: 43 tests passed in 13.86 seconds
against the actual PostgreSQL driver, including concurrent owners,
prepared reuse, transaction failure, a dedicated client, sampling, and a
query crossing the deadline. DB typecheck passed.
- Server attribution and existing delivery/execution/budget suites: 53
tests passed. Final concurrent HTTP GET/JSON POST scope refinement: 5
tests passed.
- Runtime bundle suite: 8 existing tests passed. The new no-source-map
owner test passed after its fixture was corrected to use a driver
thenable. An ordinary native Promise can bypass an overridden `.then`;
the real PostgreSQL tests separately cover driver integration.
- Observer-only experiment: three alternating pairs per case on Node
24.22.2, macOS arm64, 3,000 calls per run after warm-up. For an
already-materialized synthetic JSON result, median capture CPU fell from
82.37 to 1.87 microseconds/call. For a 100-row primitive result, it fell
from 8.10 to 4.40 microseconds/call. Every result retained identity and
all calls were counted. The new diagnostic intentionally measures less
byte data; these are observer costs, not database latency or a hosted
application speedup. Disabled-case timings were below one
microsecond/call and too small for a performance conclusion.
- Final lifecycle regression: 28 unit tests and DB typecheck passed.
Closing the last capture disables and replaces owner storage;
overlapping captures retain it.
- Final-commit full recursive typecheck passed in 38.38 seconds and full
build passed in 40.81 seconds. The build stamp matches
`83ed4ba9cebd68d48929c46f15ab32d5c0b9687b`. Focused server attribution
tests also passed 5/5 before the test-helper-only migration. Full test
acceptance will use the repository's parallel CI gates; the unsharded
local command takes hours.
- Verification failures retained: initial sandbox runs could not open
local IPC/PostgreSQL listeners; permitted reruns exercised the real
services. One old test expected JSON serialization and was updated to
assert explicit unmeasured complex values. The first bundle fixture used
the wrong promise boundary, as described above. A retained-callback
regression also demonstrated that disabling and reusing the same
AsyncLocalStorage instance could resurrect an old owner; replacing the
instance after the last capture closes fixes that case.
- The first CI run found older projection tests consuming the previous
full-response fields. Those consumers now use the full-response test
helper, preserving their byte thresholds. The two complete affected
route suites plus helper tests passed 115/115 in 73.08 seconds. A
temporary full-row query negative control then failed the unchanged
2,048-byte limit at 5,000,812 bytes and was restored; five other service
suites passed 34/34 and the affected chat cases passed 3/3. All three
previously failing CI shards passed on the corrected head. A local
skill-list benchmark smoke check completed 255 captured queries with no
in-flight queries or overflow; this was integration verification under
concurrent test load, not a timing benchmark.
- Two final-head runner jobs lost communication with GitHub before
reporting a test result. One targeted retry of those jobs and their
dependent aggregate passed. All 53 final-head checks passed, with 3
intentional skips; previously passing suites were preserved.
- Hosted capture remains pending until the reviewed revision is deployed
and a bounded window is explicitly configured.

Reproduce the focused checks with Node 24 and the repository pnpm
version:

```sh
pnpm exec vitest run packages/db/src/response-capture.test.ts packages/db/src/response-capture-postgres.test.ts
pnpm --dir server exec vitest run --config vitest.config.ts src/__tests__/database-operation-attribution.test.ts scripts/bundle-runtime.test.mjs
pnpm -r typecheck
pnpm build
```

## Risks

- Owner context is active while a valid future capture window is armed,
up to 30 minutes before accounting starts. No-window and expired calls
take the direct path. Diagnostics still add bounded work while enabled.
- A nearest nested owner is reported. Broad `http.request` and
unattributed groups remain visible; this does not identify every service
callsite or SQL shape.
- The existing census excludes tagged/raw explicit-execute/streaming
paths and internal protocol commands. Its counts are not PostgreSQL
transaction totals.
- Completion wall time includes queue/database/network/scheduling waits.
Concurrent durations overlap. Sampled bytes exclude complex values,
large strings, and values beyond the sizing limits.
- Activation is startup-only. Operators must choose the window after
artifact preparation, allow rollout/warm-up time, and remove the expired
setting afterward.

## Model Used

OpenAI Codex, GPT-6 family, with code execution, repository inspection,
and parallel agents. The exact deployment model ID and context-window
size are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-02 12:12:21 -06:00
DottaandPaperclip 006d5691a6 feat(server): support volume-free web server handoff (#18518)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Its web server also owns controller admission and user commands.
> - A rolling deployment starts a replacement before the old server
exits.
> - Shared database state alone does not preserve local agent files,
browser chunks, or in-process commands.
> - This pull request adds an opt-in ephemeral server contract and a
local overlap test.
> - The benefit is a web server that can relinquish traffic and local
files without losing acknowledged work.

## Linked Issues or Issue Description

Refs #18403, #18327, #18335, #18341.

**Current behavior**

The control-plane web server still has local agent-file authority and
local command targets. A new container can miss saved bytes or send a
command to the wrong process. An existing browser tab also needs chunks
from its original build.

**Proposed behavior**

Eligible servers share only PostgreSQL, object storage and stable
secrets. A replacement waits for readiness. The old server drains
admitted requests, keeps owner commands available through handoff, and
releases controller leases. Explicit maintenance adopts existing files
before the volume is removed.

**Reason and benefit**

This is the application prerequisite for rolling app-only deployments.
Schema migrations and permanent multi-replica operation are outside this
change.

## What Changed

- Store agent-file manifests in PostgreSQL and immutable bytes in S3.
Treat local roots as disposable projections. Preserve existing
permission checks and edit conflicts.
- Add an explicit migration, eligibility and object-retention command to
the production runtime bundle.
- Add `/api/ready`, admitted-request drain, browser restart signaling
and retained immutable UI chunks.
- Carry transaction-aware work hints across controllers during handoff
and active execution. Keep listeners closed when idle.
- Deliver durable Stop and runtime responses to the execution owner.
Recheck request lifecycle and access before response dispatch.
- Reject local-only configuration in ephemeral mode. Keep the feature
disabled for existing installations by default.
- Add an isolated PostgreSQL/S3/proxy/browser deployment harness with
independent server homes and retained failure evidence.

## Verification

- All CI checks pass on `ce17b95019d3cf1cabd357d19b6d37f96782cc5c`:
workspace typecheck, build, Runner checks, server/general tests and
browser shards. Greptile reviewed the runtime changes at 5/5 with all
findings resolved; the final commit updates maps and the plan.
- Local targeted checks passed: coordinator 27; Stop/recovery 33;
PostgreSQL command delivery 6; route/workspace regressions 8; bootstrap
Stop 1; restart recovery 75. PostgreSQL command tests simulate the
provider edge.
- Local browser proof passed 91 reads and 19 saves; reconnect took
1,013ms. The tab opened Audit from S3 after deletion of A's whole
home/build. This uses compiled workspace code and local-trusted auth,
not released containers or live agents.
- Local workspace typecheck/build passed; current server and runtime
bundle passed. The earlier broad local suite was stopped after three
hours and is not a passing receipt. Its retained failures and isolated
reruns are documented in the plan.
- Stacked #18526 adds stricter graceful-exit proof, an unready
replacement control, live Runner qualification and a Stop handoff
correction. Hosted enablement remains gated on those results and Cloud
#1096. No hosted stack has been enabled.

## Risks

- Apply migration 0422 before enabling the feature. Existing data must
pass each owner's migration and eligibility checks. Do not discard a
volume based on a single migration page.
- Durable manifests remain authoritative after the opt-in is disabled.
Rolling back to a binary that ignores them is unsupported.
- Cold directory restoration and complete shutdown timing still need
qualification before enabling a hosted stack.
- Retained UI assets must cover the supported browser and rollback
window. Cleanup is explicit; no new idle maintenance polling is
introduced.
- This change does not claim production zero downtime or authorize a
fleet-wide enablement.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window limit are not exposed in this session. Used reasoning,
code editing, shell execution and browser automation.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-01 01:58:54 -06:00
DottaandPaperclip eb12cfc96b perf(github): activate connection work from events and deadlines (#18395)
Reviewed and verified at f77620b9a4.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-31 15:38:43 -05:00
ec3b558277 fix(server): schedule accepted native response recovery (#17696)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native runs store accepted results and their finalization decisions.
> - A completed run can still owe its public answer after an
interruption.
> - Periodic recovery scans for that answer even when no new work
exists.
> - This pull request wakes recovery from committed changes and saved
retry deadlines.
> - The existing owner still verifies the result, source, and current
publication authority.
> - Settled or dormant response work no longer needs its own periodic
scan.

## Linked Issues or Issue Description

Refs #17690 (merged). This is slice 48 of the sleep migration stack, now
based on master. Related: #17661 handles GitHub terminal publication;
this change handles native accepted-response presentation.
Governed-review presentation remains a separate duty.

**What happened?**

Native finalization recovery repeatedly checks already accepted results
for a missing answer. Removing that check without complete notifications
would lose responses when evidence arrives late, an agent resumes, or
admission settles after execution.

**Expected behavior**

Recover the answer from its saved result when a supported change makes
it eligible. Retain real retry deadlines. Stop querying when the answer
is selected or cannot currently be published. Reconstruct work after
restart.

**Steps to reproduce**

1. Commit a native accepted result with no selected response comment.
2. Run finalization recovery while its source is unavailable or locked.
3. Restore that source or release the lock. The answer must appear once
without repeating provider execution.

## What Changed

- Honor recorded non-create decisions and skip-comment policy in both
discovery and recovery. Apply durable conversation/result/continuation
guards before recovery locks. Preserve genuine unanswered and legacy
recovery.
- Scope routine response wakes to the affected issue/run. Coalesced
input names its active run; pending-effect settlement names its issue.
Sixteen existing global authority/configuration hooks use one explicit
company invalidation boundary. No new transaction wrappers or tables are
added.
- Add an accepted-response worker using the existing delivery
coordinator and scoped deadline helper. Preserve saved retry deadlines
and pending-effect fences under the owner's lock.
- Share the finalization notification boundary. Wake on finalization
settlement, exact accepted-result/final-message evidence, agent
availability, retry admission, and existing chat authority changes.
- Keep lifecycle policy in services. A narrow pre-write status hook lets
lifecycle retain ownership of its transaction and status transitions.
- Register the worker during enabled maintenance startup. Remove its
authority subscription on shutdown. Periodic replay skips this duty on
both committed and fresh-commit paths; ordinary finalization and
explicit recovery retain their behavior.
- Add a partial index and expression statistics for unpresented native
runs. Migration 0412 supports concurrent replay and adds no queue table.
- Add real PostgreSQL and lifecycle coverage. Preserve the
implementation and producer audit in the plan document.

## Verification

- Additional real Photon callback acceptance on the published production
code disproves the security warning: ten active assertions cause zero
response-recovery hints/SELECTs, while three concurrent assertions
during an actual attention-to-active recovery cause exactly one
wake/SELECT. The existing locked status guard already enforces this
boundary. No production change was needed; the resolved finding now has
a successful current-head security check. The regression will be
retained with the next slice.
- Eligibility repair source `892f36e5f66ba1af123034936a7160c64c3ab883`
passes **1489 distinct server cases**: 1139 in six complete
recovery/finalizer/monitor/summary/workspace suites (81.02 seconds),
plus 350 selected chat cases (297.68s). Another 1463 chat cases are
outside the pattern; the complete 1813-case file is not claimed. All
eleven new regressions are included in that count. Five dedicated fresh
PostgreSQL 17 databases migrate through unchanged 0412, with no
fixture-source adaptations. Database build, server types/build,
boundaries and full-SHA migration ordering pass. All seven test source
hashes match.
- The exact published baseline first fails eight behavioral assertions
while the three genuine-answer cases pass. The same corrected test
source passes all eleven after repair: recorded none/reuse remain
suppressed; five ineligible history families take zero recovery row
locks across repeated issue hints; missing/legacy/create-null answers
still publish once; fresh periodic finalization defers presentation to
its worker. Earlier fixture setup and variable/compiler failures remain
explained in the audit, rather than erased by later success.
- Repair source `2fc8b0614aaf9ac7337045de4ced7325f95edc45` passes **1821
server cases**: 1496 in 13 complete producer/lifecycle/authority suites
(110.63 seconds), plus 325 selected chat cases (278.99 seconds).
Coverage includes all native-session cases, queue admission, status
decisions, membership/board access, the complete GitHub workflow,
accepted-response lifecycle and real authority restoration. The other
1477 chat cases are outside this selection; the complete 1802-case file
is not claimed for this repair run.
- Actual PostgreSQL regressions first reproduce a coalesced-input hint
missing its run scope and two routine hint paths producing 12
response-history SELECTs each after three notifications. The same
assertions pass with run-scoped coalescing and zero response-history
SELECTs for unscoped routine hints. Actual membership restoration still
publishes the waiting response once. Assertion time limits are
unchanged; this measures real driver SQL and persisted producer work.
- Eleven dedicated fresh PostgreSQL 17 databases migrate through 0412.
One embedded-only wake-queue fixture import temporarily uses its own
external database and restores byte for byte. All 14 selected source
files match pre-run hashes. This qualifies external-database behavior
rather than reproducing embedded CI locally.
- Master advanced independently with #18126's Pi archive retry fix.
Normal merge `2a535bc61259ef99387369e8b32498f774e5ba2c` preserves both
complete binary patches. All accepted server/database/UI/test/CLI trees
stay identical; the three independent runner installer/manifest files
are retained. All 13 installer tests pass, including stalled header/body
cancellation. Combined server types/build, boundaries and full-SHA
migration order pass. A subsequent normal merge of #18128 at
`e8b78cce5bb557b086bee9ca930eeaab1d09b244` retains its three test-only
file changes with both complete patches preserved. The changed HTTPS
backfill case passes on a fresh PostgreSQL fixture (1 selected case; 178
outside the pattern), and combined types/build, boundaries and migration
order pass again. All 14 repair source hashes remain unchanged. The
independent browser fixture changes are preserved; no local browser run
is claimed.
- The earlier source `9feec170ef8174011fc3c5d3d6d76f0a0f39fea0` passed
the 4171-case full baseline before this scope repair. That scope is
preserved in the audit rather than represented as a current full rerun.
All 0412 database bytes remain unchanged: original concurrent
index/statistics/ANALYZE, prior executable history, journal entries and
retained snapshots. Five snapshots 0408 through 0412 remain. The
populated 0411-to0412 upgrade preserved 410 prior receipts and 38
companies, appended one receipt and none on replay; index/statistic
checks passed.
- The earlier P2 security failure, Greptile's former conditional
dependency on #17702, and the subsequent actual Ponytail eligibility
findings are recorded in
`doc/plans/2026-10-25-native-response-presentation-sleep-audit.md`. This
PR now includes the issue-scoped pending-effect notification fix itself,
so it is independently correct before #17702. Fresh published-head CI,
Greptile 5/5 and a clean security scan remain required. No
deployed-sleep, external-wake or fleet-savings claim is made.


## Risks

- The writer contract is process-local. Independent SQL writers and
other replicas are not covered by these notifications.
- Unpresented backlog reconstruction still has memory and time cost
proportional to candidate count. Pages contain at most 100 rows;
execution uses the shared fair batch. Selected history is excluded.
Health-only signals do not trigger reconstruction.
- Publication policy remains with its current owner. Digest, company/run
binding, source authorization, selected-answer suppression, and
ownership fences remain required.
- The rest of native recovery and governed-review presentation still run
separately. This PR does not relax Cloud sleep eligibility or add
external wake coverage.
- The stack parent has merged and its actual ancestry is reconciled.
This publication still needs fresh current-head CI and review.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, shell
execution, and PostgreSQL tests. The runtime does not expose a more
specific model identifier or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Codex <noreply@openai.com>
2026-10-29 02:31:20 -05:00
DottaandPaperclip 5f0da562db fix(server): wake chat lifecycle milestones from committed changes (#17606)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Connected chats show safe notices when a run starts, waits,
finishes, or fails.
> - A recurring scan projected these notices even when no eligible work
existed.
> - Eligibility depends on committed run state, conversation lineage,
and current authority.
> - Existing database rows can reconstruct that work after restart.
> - This PR wakes a scoped worker from those writes and removes
lifecycle polling.
> - Native progress events keep their separate existing owner until the
next migration.

## Linked Issues or Issue Description

**What happened?**

The chat maintenance timer repeatedly scanned run lifecycle state. Empty
and settled instances still queried the database. A run-status event
alone cannot replace the scan: prompt receipts, continuation receipts,
identity changes, and restored authority can make a run eligible later.

**What did you expect?**

Committed changes wake projection. Startup reconstructs eligible work. A
completed pass stops until another relevant change arrives. Failed work
retains recovery without scanning settled history forever.

**Steps to reproduce**

Configure an automatic chat endpoint. Finish its work, or leave it
unused. Observe recurring lifecycle projection queries from the chat
maintenance callback.

Stacked on #17592. Related foundation: #15625. This PR owns lifecycle
milestones only.

## What Changed

- Preserve upstream coalesced Cloud membership writes together with
locked membership restoration and its committed chat notification.

- Signal milestone eligibility from the real Slack account-link
transaction.

- Register the lifecycle worker with the existing delivery coordinator.
Remove lifecycle projection from the periodic chat callback. Keep native
progress in a separately named lane and preserve the explicit combined
API.
- Reconstruct company/issue scopes from the existing selector. Drain
bounded batches with company fairness. Retry failures, retain writes
through drain/resume, and disarm settled work.
- Signal from supported run, presentation, interaction, receipt,
lineage, conversation, recovery, and authority writers. Preserve their
transaction ownership, including nested savepoints and uncertain
settlement. No new table is added.
- Preserve provider authorization, authored-final suppression,
responsible-user continuity, deduplication, the 24-hour selection
window, and raw-output restrictions. Keep provider interpretation in its
owner.
- Preserve PostgreSQL timestamp precision in lifecycle pagination.
Rejected submillisecond candidates must not repeat the same page and
starve later notices.
- Document producer coverage and alternate-writer dispositions in
`doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md`.

## Verification

Published head: `1d056f68c5b1005933b9e12de8745bf1c4cbeeca`, targeting
master after #17592 merged at
`7d75ae0569e41786640cf09647ae8849411d2f22`. A normal merge integrates
the actual landed parent. The entire 77-path milestone patch is
binary-identical to the locally verified preparation across that
ancestry reconciliation; both unrelated master paths are preserved. The
four earlier producer/test integration conflicts retain scoped milestone
and voice notices, current onboarding completion admission, and expanded
identity query/read-scope coverage.

- **330 distinct selected cases pass on the verified unchanged patch:**
305 across 14 complete suites (all five milestone worker suites, run
identity, board claim, Cloud provisioning, auth session routes, Cloud
actor authentication, stale-lock recovery, GitHub credentials, project
Git authorization, and app admission wiring), plus 25 selected real chat
producer cases. The remaining 1,590 chat integration cases were skipped,
not passed.
- The real producer cases cover saved milestone discovery, GitHub and
Slack identity changes, account-link rollback/reconfirmation, ordinary
ingress reactivation, post-close action/delivery receipt settlement, all
ten authority-restoration paths including Cloud synchronization, and
changed native review metadata.
- Worker coverage verifies outer commit, rollback, uncertain-write
recovery, restart, bounded batching/fairness, timestamp precision,
retries, drain/resume, and settled SQL quiet. Authorization and existing
publication deduplication remain with their owning services.
- Server production typechecking, build, and module-boundary checks
pass. The tests above were executed separately. Production typechecking
excludes `src/__tests__` and does not replace those runs. Each database
owner uses its own newly created and app-migrated PostgreSQL fixture.
- [Current-parent acceptance and preserved failure
ledger](https://github.com/paperclipai/paperclip/blob/1d056f68c5b1005933b9e12de8745bf1c4cbeeca/doc/plans/2026-10-25-chat-run-milestone-sleep-audit.md#current-parent-integration-and-acceptance-october-28)
records commands, counts, producer scope, conflict dispositions, and
limits. Earlier combined-stack counts and failures keep their original
revision scope; local acceptance does not erase old CI failures.
- No new migration, queue table, dependency, workflow, or shared
scheduler contract is introduced. Parent migration 0397, bounded consent
recovery, voice cleanup/retention repairs, and runner checkpoint
restoration remain present.
- **Exact-head Greptile is 5/5 and all 55 current checks are clean.** No
Ponytail review is present, which is an explicitly authorized
nonblocker; no failing review is waived. Older reviews and parent checks
are not current passes. Actual failing Ponytail findings must be fixed;
an absent/stopped/paused turn without a verdict is an authorized
nonblocker. Active unfinished Ponytail uses its original request/start
timer, with a 25-minute monitor cutoff respecting the user's 30-minute
maximum.
- No complete local repository matrix, live provider tenant, external
alarm, physical host/database suspension, or fleet savings claim is
made. Native progress remains a separate following slice.

## Risks

- A future eligibility writer must notify the existing projection
boundary. Independent SQL writers or another active server process are
outside this process-local signal contract.
- Authority and lineage remain sensitive. The worker uses the existing
selectors and publication policy; it does not grant execution or
disclosure authority.
- Native progress and other publication maintenance still have recurring
work. This PR removes one reason, not the entire chat maintenance timer.
- Worker quiet does not prove database suspension, host sleep, or
external alarm coverage. Cloud sleep blockers remain unchanged.
- Merge eligibility requires passing current-head CI and Greptile 5/5
after this actual-master reconciliation. Subsequent stack slices retain
their own gates.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window size are not exposed. Used repository inspection,
reasoning, shell execution, PostgreSQL tests, browser E2Es and GitHub
tools.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details available in this session)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run the documented focused tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-28 11:07:54 -05:00
DottaandPaperclip a26d217e94 fix(server): coalesce Cloud membership synchronization writes (#17890)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Cloud requests synchronize a human user's profile and company
membership.
> - Concurrent requests repeated the same writes before the debounce
could record success.
> - Upserts also updated unchanged rows and caused avoidable tuple work.
> - This pull request shares identical pending writes and guards
unchanged updates.
> - Each request still reads current authority. Bounded profile lookup
also reaches older memberships.
> - The benefit is less database work during request bursts with the
same actor results.

## Linked Issues or Issue Description

**What happened?**

A burst of 20 trusted Cloud requests issued 20 membership upserts and 40
profile plus membership tuple updates for one unchanged user. The
completed-write debounce did not share in-flight work.

**Expected behavior**

Identical in-flight requests share persistence work. Distinct contexts
retain their order. Failures remain retryable. Each request resolves its
own current company and role authority.

**Steps to reproduce**

Run the synthetic PostgreSQL comparison below at baseline
`db70c5acd92f50fa37fb4f0dea3da0e7685089a2`. The harness imports the real
resolvers and counts actual writes with database triggers.

Related work: #17663 covers adjacent Cloud authorization and projection
behavior. This change preserves fresh imported memberships,
owner-elevation settings and stale administrator removal.

## What Changed

- Serialize synchronization per database client and user. Share only the
last identical queued context. Preserve A/B/A transitions and retry
after partial failure.
- Guard user and membership updates with null-safe value comparisons.
Retain default-grant repair when `RETURNING` is empty.
- Keep authority reads outside the write debounce.
- Resolve exact principal IDs through the existing index, then use at
most one company-scoped alias query. This prevents both the old 200-row
profile cutoff and repeated membership-page scans. Exact IDs take
precedence over namesake aliases.
- Add concurrency tests, PostgreSQL tests, a reproducible benchmark and
behavior documentation.
- Accept both Git 2.55 and 2.56 diagnostic messages in an existing
workspace safety assertion, after full CI exposed that portability
failure. No expiration behavior changes.

## Verification

- Five focused suites passed, 63 tests total. These include Cloud actor
concurrency, transient database retry, company provisioning, real
PostgreSQL synchronization and the user profile route.
- The auth-to-profile regression returned 404 before the lookup repair
and 200 after it. The final fixture has 601 memberships and tests raw
and normalized non-UUID IDs, missing slugs, alias ordering and exact-ID
priority. A counterfactual with the old paging repair performs four
membership queries for a missing slug; the final code performs two, each
returning at most one row. Cross-company access returns 403 before
lookup.
- Local PostgreSQL 17.11 benchmark uses two warmup pairs and five
alternating measured pairs. Each pair starts a fresh debounce scope. All
20 complete actor results match on every sample.
- Burst median: 51.07 ms before, 25.71 ms after. Ranges: 47.21–63.25 ms
and 24.17–29.32 ms. Membership statements: 20 to one. Tuple updates: 40
to zero.
- Reproduce: `pnpm --filter @paperclipai/server exec tsx
../scripts/benchmark-cloud-membership-sync.ts
--baseline-ref=db70c5acd92f50fa37fb4f0dea3da0e7685089a2
--docker-postgres=17`.
- Real PostgreSQL tests compare all four alias fields with the route's
own JavaScript canonical slug output across 76 fixtures. Exhaustive
Unicode lowercase inspection validates the special SQL mappings.
- A real SQL_ASCII database reproduces the previous en-dash alias 404.
The final whole-character replacements precede ASCII-only translation.
Four alias fields and ten multibyte edge cases pass, including rejection
of a false euro-to-k alias. The disposable database is migrated
normally, then removed.
- Server TypeScript passed with no diagnostics.
- PostgreSQL tests used the normal migrations with a temporary Docker
startup override because this host's native PostgreSQL semaphore
allocation is exhausted. No tracked test infrastructure changed.
- The first full CI run had an unrelated CLI company-import/export
health-start timeout. Review also identified repeated profile scans;
this follow-up repairs that finding. A later run passed 3,882 tests in
server shard 4/12 and failed one existing assertion because Git 2.56
changed its error wording for a missing gitfile target. The safety
behavior still refused expiration and preserved the workspace. The
assertion now accepts both valid messages; the actual Git/PostgreSQL
regression passes locally. Final head
`3a644e600ab0ae7315f9828b1ea9dffdef407e4a` passed [full
CI](https://github.com/paperclipai/paperclip/actions/runs/40537153522),
including the previously failing shard. There are 55 successful
checks/statuses, two skipped and one neutral. Fresh Greptile and
Paperclip reviews are both 5/5 on this exact head, with zero unresolved
threads. GitHub reports mergeable and CLEAN.

- Full repository typecheck and build passed on the integrated final
source. The local full test command was stopped after native PostgreSQL
initialization failed repeatedly, producing 16 failures and skipped
database suites. It is incomplete, not a pass. Focused real-database
suites used disposable Docker PostgreSQL. Current-head CI runs the full
gates separately.

## Risks

- Coalescing is local to one process and database client. Other replicas
can still take conflict locks.
- `updated_at` now records actual profile and membership changes. Access
lists sorted by it can show a different order. Profile resolution no
longer depends on the first 200 entries. Exact principal IDs take
precedence over namesake aliases. The single alias fallback still scans
company candidates; no constant-CPU claim is made.
- Default role grants retain their existing additive semantics. This
change does not define a new demotion policy.
- Local synthetic service timings exclude HTTP and network latency. No
hosted deployment or production speedup is claimed.
- No migration, startup job, polling change or deployment is included.

## Model Used

OpenAI Codex, configured model `gpt-6.1-sol` with xhigh reasoning, tool
use and code execution. Independent blast-radius review also used
`gpt-6-astra` and `gpt-6.1-sol`. The runtime context window was not
exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-27 11:21:44 -05:00
Devin FoleyandPaperclip 9f3a5456ac perf(server): bound run reads in authentication and conversation recovery (#17663)
Read compact run identity during authentication and compact turn state during conversation recovery. Preserve fresh checks, durable steering and reply evidence, locking, and complete stored payloads.

Validate 184 targeted PostgreSQL and lifecycle tests, workspace typecheck, server build, full CI, and Apex 5/5.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-25 15:53:18 -07:00
Nicky LeachandPaperclip 79b2bf78dd Create agent records when onboarding names are submitted (#17005)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core onboarding selects an agent and connects its provider account.
> - The current flow creates the agent record after authentication.
> - Setup needs a stable agent identity as soon as the user submits its
name.
> - This PR saves a paused setup record on name submission and finishes
the hire on that same record.
> - The existing sandbox authentication and approval paths continue to
apply.

## Linked Issues or Issue Description

**Subsystem affected**

Core agent lifecycle, shared validators, and onboarding UI.

**Current behavior**

Onboarding creates the database record at the final hire step. Earlier
authentication and saved setup have no agent record.

**Proposed behavior**

Create a paused agent with a setup hold when the user submits its name.
Do this before adapter selection or authentication. Update that same
record after adapter selection. Reuse its ID on retries and saved setup.
Finish that record through the existing hire path.

**Reason and benefit**

Give setup a stable identity without coupling this refactor to computer
provisioning or a new execution backend.

**Breaking changes**

Incomplete setup now leaves an inactive agent record. Existing clients
can still create and hire agents without the new optional fields. The
sandbox authentication flow and API key support remain in place.

**Additional context**

Requested Core-only refactor. Related work: #15422 changes runner
defaults. This PR preserves those choices. No duplicate early-record
refactor was found in the issue and PR search.

## What Changed

- Save the agent when the user submits its name in both onboarding entry
points. Use a provisional adapter until the user chooses one. Reopen
name-only drafts at adapter selection.
- Add retry-safe setup creation and same-record hire completion. Keep
approval checks at final hire.
- Save and restore setup configuration in both onboarding entry points.
Show Finish setup for paused setup agents.
- Restrict setup changes to authorized board users. Block generic resume
until setup finishes.
- Add lifecycle, authorization, connection, and UI coverage. Update the
specification and feature maps.

## Verification

- Latest focused server run: 126 lifecycle and AI-connection tests
passed after the Apex fixes. An earlier broader run passed 301
lifecycle, instruction, adapter-validation, and AI-connection tests.
Earlier authentication, retry, cross-company, and direct-create
regressions also passed.
- Name-save verification: 336 tests across the wizard, naming dialog,
New agent, action-button, and wizard-step suites passed. Coverage
includes retry identity, repeated submission, closing during a save,
saved-model restoration, and a disabled process adapter.
- Shared, server, and UI TypeScript checks passed. The latest UI package
check and direct server TypeScript check passed. The server package
preparation command could not finish locally because Cargo is
unavailable; full build preparation runs in CI. Map checks, token gates,
and `git diff --check` passed; each pushed diff was scanned for
credentials and PII.
- All six onboarding browser tests passed together with installed Chrome
against a fresh throwaway instance after the Apex fixes. They include
numbered duplicate names, first-time Copilot without a connected
account, and Grok’s saved model. They verify early persistence in both
entry points, closing and reopening a name-only draft, same-record
adapter selection, sandbox sign-in, login reload, and failed-connection
gating. Provider responses are simulated; Core persistence is real.
- Regression coverage includes incomplete Copilot creation versus final
validation, Kimi settings through redacted responses, stale local
drafts, company switches during creation, saved environments, and
provider changes with personal/shared old bindings.
- Apex found five name-only setup gaps: numbered names, early Copilot
account validation, the Grok model default, same-name draft ownership,
and provisional runner restoration. The fixes preserve server names,
defer implicit account binding until hire, set Grok’s model, avoid
name-based reuse of unfinished records, and restore Auto for pending
adapter choices. UI, database, and browser regressions cover these
cases.
- Standard Greptile found a Grok provider mismatch in the new adapter
save. The save now uses ACPX with the Grok profile. Both the 129-test
dialog/setup run and a real-API browser regression passed after the fix.
The first browser attempt stopped on an ambiguous empty-roster button
selector; it now scopes the click to the toolbar.
- Moving the save point exposed a saved-model reset and timing
assumptions in the same-name and portal fixtures. The source choice now
preserves the restored model, and tests cover same-name recovery without
a second hire.
- Initial CI exposed direct-create compatibility, concurrent setup
effects, fixture assumptions, and a missing-fields assumption in the
provider-validation guard. These were fixed and verified locally. Review
findings were addressed with authorization, recovery,
credential-persistence, and restored-configuration regressions.
- The final desktop capture passed after host resource cleanup. A repeat
first failed before Core started because macOS had exhausted
shared-memory IDs. Standalone PostgreSQL initialization reproduced the
same error. Releasing only unattached control segments from this task’s
stopped test databases allowed the unchanged revision to pass. No Core
or system setting changed.
- Local database tests required port access outside the sandbox. Missing
declared development dependencies were supplied outside the branch. No
live provider sign-in was tested locally.
- [Full monorepo
CI](https://github.com/paperclipai/paperclip/actions/runs/39522434236)
is green on `82b82fec189a172a885d4dd565ae2f4a18c73efc`. Standard
Greptile scored 5/5 with zero new comments (check `118969424903`). The
subsequent Apex review also scored 5/5 with zero new comments (check
`118972866331`) on that same commit. All review threads are resolved.
- The separate Superagent Security scan returned neutral/inconclusive
because its pinned merge-base checkout failed or timed out. GitHub
returned 404 for check and suite rerun requests. This is not a
successful security scan; the CI workflow and other security checks
passed.

## Risks

- Users can leave inactive setup records behind, including after name
submission alone. These records cannot run until hire completes.
- Setup completion touches hiring and approval boundaries. Tests cover
retries, company isolation, creator permissions, and approval snapshots.
- An uncertain credential-save result retains its new secret reference
for retry. A confirmed client rejection removes only credentials staged
by that attempt.
- No schema migration or VM provisioning change is required.
Agent-computer integration is out of scope.

## Model Used

OpenAI Codex, based on GPT-6, with repository search, code editing, and
tool execution. The exact deployment ID and context window are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] Apex Greptile is also 5/5 on the same commit as standard review
and green CI
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 18:08:57 -07:00
DottaandPaperclip 505e1a5582 feat(cases): add persistent workflows with same-task agent handoffs (#16959)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - An agent workflow needs persistent facts and a safe way to transfer
work.
> - Creating a task for each phase splits the objective, discussion,
workspace, and review policy.
> - A case can retain the facts while one primary task moves between
agents.
> - Transfers must survive process failure and preserve private access,
budgets, and completion gates.
> - This pull request adds durable case handoffs, native case tools, and
read-only case pages.
> - Prompts describe the workflow. Existing task controls decide when
work can run and finish.

## Linked Issues or Issue Description

**Problem or motivation**

Prompt-driven agent teams need a persistent record that can move through
triage, implementation, verification, and correction. Ordinary
reassignment does not provide a durable instruction, safe release
barrier, or recoverable delivery receipt for an agent handing off its
own running task.

**Proposed solution**

Extend standalone cases with one primary task, a descriptive phase,
retained access sources, and a handoff ledger. Apply assignments after
source execution and workspace settlement. Deliver each transfer through
the existing durable wake queue. Provide `create_case`, `read_case`,
`update_case`, and `handoff_case` in the native runner. Creation
requires kickoff and explicit workflow text. An explicit handoff to
`done` closes the case when normal task completion is accepted.

**Alternatives considered**

Do not add a workflow engine, filesystem queue, configured state graph,
or one task per phase. The workflow document combines transition prose
with an optional Mermaid `stateDiagram-v2` map. The server does not
execute the diagram. The Pipelines experiment removed on master stays
removed.

**Roadmap alignment**

This extends Agent Reviews and Approvals, Enforced Outcomes, Bounded Run
Recovery, and Activity Log attribution. Existing approval, review,
budget, privacy, workspace, and responsible-user rules remain
authoritative.

Related work: [ordinary handoff wake
recovery](https://github.com/paperclipai/paperclip/pull/15019). This
feature adds a separate persisted case-handoff receipt and release
barrier.

## What Changed

- Add case routing fields, retained access sources, durable handoff
history, delivery leases, and replay-safe migrations.
- Keep the same primary task across agent phases. Preserve each
returning agent's own session and the task's workspace.
- Fence outgoing mutations, wait for provider and workspace release, and
recover delivery without duplicate task comments or runs. Recheck
current assignment permissions as the original caller before applying a
delayed transfer. Scope a run Stop to that run’s own handoff intents, so
repeated old Stop requests preserve newer work. Cancelling terminal
intent without a source run also requires fenced current task ownership,
including native retries and preparation with an old checkout. For Cloud
callers, revalidate current stack membership with an uncached Cloud RPC
and derive owner elevation from the current managed setting.
- Keep case reads, documents, assets, Audit, and live events inside
retained task/project permissions.
- Add native case creation/read/update/handoff tools. Store required
workflow text and initial documents on the first task. Page oversized
case facts and referenced document bodies through the dedicated tool
with bounded UTF-8 JSON, current routing versions, snapshot conflicts,
and current source authorization on every page.
- Add read-only case lists and detail pages, task-style list controls,
compact dotted field tables, linked handoff tasks, and case filters in
Audit.
- Add native/legacy integration tests, migration/restart tests, browser
coverage, and bounded live receipt fixtures. Extend the trusted sandbox
condition to cover the new subscription fixture profile.
- Hide obsolete paused-agent warnings on completed or cancelled task
threads.
- Synchronize the company-switch settings test with React effects after
a CI failure. Preserve its security assertions and production remount
behavior.

## Verification

The branch directly includes master `9ef434fe79`. The required `ci /
verify` and `ci / e2e` gates pass on head `765fccd67c`. [Current-head
CI](https://github.com/paperclipai/paperclip/actions/runs/39464348614)
passes typecheck, build, tests, and all eight browser shards. Security
checks pass. [Greptile gives this exact head
5/5](https://github.com/paperclipai/paperclip/pull/16959#issuecomment-6291156549),
with no unresolved review threads. The optional Paperclip Review check
remains queued because its assigned agent is paused. It is not a
required branch check. No check result was overridden.

Local workspace typecheck and build pass after the master merge, along
with database migration checks, token gates, map checks, Storybook
catalog checks, and the Copilot profile identity check. Standard builds
refresh Runner vendor output. Focused checks pass 2,008
API/paging/context tests and 7 strict case protocol tests. The final
commit changes one test fixture only; its ten tests, UI typecheck, and
token gates pass. No production source changed in that commit.

On current head `765fccd67c`, all **194 critical workflow tests**
executed and passed with no skipped tests in those files:

- 38 Stop/ownership tests, including board completion intent, native
retries, stale Stop, and preparation with an old checkout.
- 2 actual legacy process/HTTP handoff tests, including scratch failure,
a closed-stdio child that continues writing, and recovery through a
fresh worker after the process group stops.
- 5 migration tests, including SQL column metadata, preserved null
authority context, and all six prior-numbering receipts.
- 3 process-settlement and 5 scratch tests.
- 88 handoff-service, 18 heartbeat handoff, and 21 native case-tool
integration tests.
- 7 bounded case/document-read projection tests and 7 strict
protocol-contract tests.

The current CI checkout was `fed65d49`, merging `765fccd67c` into master
`9ef434fe79`. Its new document regressions reconstruct exact
Unicode/escaped bodies and revisions through real native authority and
authenticated HTTP with generic API tools disabled. They cover
case-owned, primary-task, and linked-task documents; routing on every
page; stale document snapshots; access loss and unlinking;
unrelated/foreign/private sources; system documents; and quarantined
content. Active-task document tools retain their existing behavior.
These are current-head test results from the first CI attempt; the
browser-only retry did not rerun these suites.

The full `64fdd3403f` CI failed one separate settings test: after a
company switch, it observed an unsettled API-key draft. The production
component and test were unchanged from prior green source. Production
switches remount both keyed settings layers; the test directly reused an
unkeyed view and used a timer tick instead of waiting for React effects.
The final commit replaces that fixture synchronization with React `act`.
The exact secret-clearing assertions and all timeouts remain unchanged.
The original assertion failure remains recorded; no unchanged retry was
used. The narrow unchanged baseline and fixed fixture each pass 10
tests.

The upstream result-paging merge preserves case capture namespaces and
current asset privacy. Its API integration suite executed 39 of 41 tests
successfully, with two skipped; the optional assigned-result PRP
integration test was skipped. Neither skipped suite is counted in the
194 critical tests or as live provider proof.

All 361 upstream migration entries and SQL bodies remain unchanged. Six
case migrations follow master as 0363–0368 with identical SQL hashes.
Official Drizzle generators produced the combined metadata and five
retained snapshots. Generation reports no schema changes, and the
snapshot drift test passes. The earlier `ce361ae5d0` CI passed on
attempt 1, including its then-current 179 critical tests. Fresh review
later identified unusable document references with generic API tools
disabled; the bounded document reads and ten additional authenticated
HTTP tests address that finding.

The full local `pnpm test:run` attempt was cancelled after this Mac
exhausted PostgreSQL semaphore capacity; it is **not** counted as a
pass. Later database attempts also stopped before assertions or skipped,
including one sandbox socket denial. Two original stale-Stop regressions
did reproduce the bug against the old source. A REST serialization
assertion passed after two preserved database-startup failures. No test
timeout was increased. All 194 assertions listed above executed and
passed in current-head CI. The complete 385-file, 3,963-test UI shard
also passed after the fixture repair.

A configured-provider receipt workflow passed 13 backend checks across
triage → implementation → verification → correction → done, using one
task and four work runs. The saved receipt is revision 2 with total 6
USD; case and task both closed. The original canonical attempt remains
failed because its browser assertion confused the terminal destination
with completed-task tooltips. The corrected full browser verifier passed
on the same retained database and original runs, with no additional
provider calls. The last clean-build browser readback is on
`6a86c54a53`; later changes affect recovery, authorization, bounded
reads, and settlement. Accounted work cost is a $1.010574 rate-card
estimate; provider billing and setup-probe usage are unavailable.

Copilot profile 49 passed the unchanged seven-fixture private protocol
pack (46/46 checks, no retries) and the product file-edit canary after
master's completion-replay change. All 60 bound implementation hashes
remain unchanged on the current source. Historical profile-47/48
identities and results remain intact. See the [profile-49 compatibility
record](https://github.com/paperclipai/paperclip/blob/codex/persistent-case-handoffs/doc/plans/2026-10-20-copilot-completion-recovery-compatibility-v49.json).
This is bounded compatibility evidence, not a fresh full matrix,
installed-package product test, or Daytona rollout.

Preserved CI failures and retries:

- On `20662930ac`, Slack managed setup failed a five-second success
assertion after reload. Its test/application path was unchanged from a
prior green head. One unchanged-source retry passed all 25 tests in that
shard; no source or timeout changed.
- On `9402b4f17a`, two CI workers received shutdown signals at 15:24:19
UTC. Neither log had an assertion failure before interruption. One
unchanged-source retry of each passed all Runner static checks and all
32 tests in browser shard 6; both aggregates passed. The next head,
`7b4374edaf`, passed all CI without retries.
- On `295f289b5a`, serialized shard 9 timed out during its first
device-login test, then the next test found an active login lease.
Direct login/test paths were unchanged from a prior green head. Cold
import plus a timed-out async body using the next test's mutable
actor/store is the supported fixture-race explanation; no request trace
proves that sequence. The original failure remains recorded. One
unchanged-source retry passed all 294 tests in 19 files, including all
41 device-login tests. No login source or timeout changed. The
subsequent `ce361ae5d0` CI passed without retries.
- On `64fdd3403f`, one company-switch settings assertion observed an
unsettled draft. The final test-only commit uses React `act` without
changing assertions, timeouts, or production code. On `765fccd67c`, all
ten settings tests and the complete 385-file, 3,963-test UI shard
passed.
- On `765fccd67c`, browser shard 3 completed the split-handoff
assertions, then cleanup received HTTP 500 from run cancellation after a
one-second PostgreSQL row-lock timeout. The extra chat run was reporting
completed work while teardown began. Cleanup stopped before restoring a
global experiment flag, causing a later assertion to fail. Relevant
production and test paths are unchanged from an earlier green commit.
Stop's lock order is consistent, and no nested self-lock was found; the
exact PostgreSQL blocker was not recorded. One unchanged-shard retry
passed. The original failure, trace, and downstream flag failure remain
recorded. No Stop implementation, cleanup assertion, or timeout changed.
The retry supports intermittent cleanup contention; it does not identify
the original blocking transaction.


## Risks

- Local process transfers require host-owned PID/group absence. Missing
or uncertain termination evidence keeps delivery pending, independently
of scratch setup.
- Handoff admission and task cancellation share execution ownership. New
regression tests cover temporary holds, late source writes, and
uncertain provider termination.
- Retained private sources use an intersection of existing permissions.
This intentionally fails closed when a source is missing or an existing
task cannot preserve that boundary.
- Migrations add tables and indexes. They preserve master history and
can replay over prior feature installs. Migration 0368 adds nullable
internal authorization provenance with no default or authority backfill.
Historical transfers still awaiting release without this context fail
closed; an operator must cancel them and request a new handoff.
- Delayed Cloud handoffs depend on the current Cloud membership RPC. An
unavailable, ambiguous, expired, or mismatched response holds the
handoff for retry; it does not reuse cached authority or grant access.
- Bounded reads reload and reauthorize the selected projection per page.
Underlying work grows with the data size and number of pages. Changed
data or routing requires restarting from offset zero without the old
snapshot.
- Phase names and workflow diagrams remain advisory. They do not satisfy
review or approval gates.
- This is a large, cross-cutting PR. The maintainer requested one PR for
the whole feature; the commits separate data, runtime, UI, and
qualification work.

## Model Used

OpenAI Codex, GPT-6 family, with tool use, code execution, and delegated
code review. The exact deployment model ID and context-window size were
not exposed in the session. Live fixture agents used the explicitly
configured provider model recorded in their test evidence.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 12:46:14 -05:00
DottaandPaperclip 92b238137b Connect and configure GitHub Copilot through Apps (#15705)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Apps and AI Connections store credentials and govern access to agent
runtimes.
> - Users need a supported saved-token connection and config values that
survive save and reopen.
> - The release must preserve saved-token access and task lifecycle
guarantees.
> - This pull request will connect Copilot through the normal Apps and
AI Connections path.
> - The benefit is a reviewable part of the Copilot production release.

## Linked Issues or Issue Description

Refs #15701 and #15703. Supersedes setup work in #15562. Base: #15703.
Next: #15706.

## What Changed

- Require configuration update authority before Copilot
credential-bearing model discovery; default to the saved environment and
preserve explicit selection for editors.

- Keep Copilot discovery passive during reconnect reconciliation;
refresh on explicit opening, account/environment changes and reconnect.

- Add GitHub AI token connections with encrypted storage, ownership,
grants and audit logging.
- Keep repository access separate from Copilot runtime authentication.
- Probe the selected environment without sending a model prompt and
preserve authenticated model selection through save/reopen.
- Keep fresh execution admission consistent with normal Copilot setup;
persist task model choices without changing provider or permission
policy.
- Refresh authenticated models after reconnecting the same saved Copilot
account, including recovery from a previous authentication error.
- Apply the additive provider-default constraint migration through the
normal generated migration workflow.

## Verification

Runtime #15701 is merged at `d855e550ad8d2a267312983f26457aabf5393c3a`.
History #15703 is merged at `df80616d98119cbf5abd40549794d9980a5fabf6`.
Remaining source
[`42bcbd61299655f7e37aabe6bb5f19bba770ad1d`](https://github.com/paperclipai/paperclip/commit/42bcbd61299655f7e37aabe6bb5f19bba770ad1d)
contains the four-file authorization repair. The remaining PR trees
preserve six nonoverlapping files from the newer public mainline
managed-settings fix and the CI-owned lockfile. No unrelated provider is
enabled.

The security finding is repaired: existing-agent discovery requires
`agent_config:update` before connection selection or token resolution.
The saved environment is the default. Authorized editors can select a
prospective environment. Both denied board-viewer and agent-reader
regressions fail on the original source. All49 route tests and server
typecheck pass with the fix. Denied requests perform no connection
selection, credential resolution or probe. Current-head security scans
pass on both remaining PRs; the repaired thread is resolved.

Copilot CLI1.0.88 and profile47 retain digest
`sha256:55044aa235e88b3fe36ec8a556ee1e3382590870aab907e4462ef7d64af49cea`.
The [qualification
record](https://github.com/paperclipai/paperclip/blob/42bcbd61299655f7e37aabe6bb5f19bba770ad1d/doc/plans/2026-10-20-copilot-qualification-v47.json)
reports20/20 Product cells,154 assertions and7/7 protocol cases,46
assertions against frozen normal installed source
`c2616370800f5f6df45246b0890ba35a7cd2faab`. The exact qualification
model is `gpt-5.6-luna`. Every passing Product case confirms cleanup.
Original failures remain recorded; automatic behavior retries are
disabled. The route-only authorization repair changes no bound
runtime/profile bytes and does not relabel the frozen campaign.

The ordinary installed remote journey passed saved-token selection,
model discovery, normal UI save/reopen, metadata-only Test, completed
task, accepted receipt and exact19-byte browser download. An open model
catalog caused no additional discovery requests during117seconds. Actual
installed runtime smokes passed on macOS ARM64, macOS x64 under Rosetta
and native Linux x64 on Daytona. Raw token browser entry was not
exercised; encrypted connection creation used the normal API.

The first Daytona file attempt timed out during delayed semantic
dispatch; its precise stalled layer remains unresolved. Independent
ordinary remote delivery and the other19 Product cells passed before one
instrumented recheck passed8/8 with unchanged source,prompt,assertions
and deadlines. No causal latency repair is claimed.

Before the authorization repair, all four final PR heads passed required
verify/E2E and fresh Greptile5/5 reviews with zero unresolved threads.
Current-head CI/review is running for the two repaired heads. Local
focused checks retain129 connection,59 runner,72 shared-eval and88
recovery/workspace passes, plus migration safety and token gates. The
additive Copilot migration is361; released master history is preserved.

[Cloud normal-package
run39414239704](https://github.com/paperclipai/paperclip/actions/runs/39414239704)
builds the pre-security source4f3f. All four native producers and Linux
package assembly pass; the final clean installed consumers are
finishing. All18 retained archives match their hashes;37,863 regular
files contain zero selected credentials. That package does not contain
the later route repair. The actual final published package must include
it and pass the local/Daytona live canaries. The frozen Daytona image
remains published and independently verified. No separate full-image
scan is claimed. Docker builds run in the cloud.

**Remaining:** current-head CI/review, setup and qualification merges,
final publication, and ordinary local/Daytona canaries against the
actual shipped package. The user explicitly authorizes the maintainer
review override; CI remains a merge gate. No production deployment is
claimed. Provider USD remains unknown and GitHub-attributed.

## Risks

Bad credentials, missing entitlement, runtime installation and
unavailable models must remain distinct failures. A connection must not
adopt ambient credentials. The constraint migration changes accepted
provider values, with no credential-storage migration.

Current-master NewAgent fixture extraction is retained. Copilot cases
use that shared support and test both passive discovery and explicit
reconnect under the same setup lifecycle.

The onboarding retry regression shares the component suite’s
saved-connection, authenticated-model, draft, agent-list, hire and
render lifecycle fixtures. Keeping it beside the existing onboarding
failures makes adoption versus duplicate hire observable under the same
lifecycle; this is the concrete cohesion reason for the existing large
test file.

## Model Used

OpenAI Codex, based on GPT-6. The exact serving model ID and context
window are not exposed in this task. Used for code changes, review and
tool-assisted verification.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-20 07:02:22 -05:00
DottaandPaperclip d9bcf4dfe8 fix(auth): let Cloud stack admins manage experimental settings (#16907)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People use experimental settings to opt into product features.
> - Cloud stack Admins can manage their company, but cannot save an
experiment.
> - Requiring an Owner role adds unrelated privileges to this workflow.
> - This pull request gives trusted Cloud stack Admins permission to
update experiments.
> - Other instance administration and platform settings keep their
existing checks.

## Linked Issues or Issue Description

**What happened?**

A person with the Cloud stack Admin role receives “Instance admin access
required” when they enable an experimental setting, such as Fast
responses.

**Expected behavior**

Cloud stack Admins can save editable experimental settings. This
permission does not grant general instance-admin authority. Members,
Support users, and agents cannot change experiments. Cloud-managed and
operator-hidden settings remain protected.

**Steps to reproduce**

1. Sign into a Cloud staging workspace as a stack Admin.
2. Open Settings → Experimental.
3. Enable Experimental fast responses.
4. Observe the permission error instead of a saved setting.

**Additional context**

Related: #10343 introduced computed Owner elevation. #15748 added Fast
responses and the company Connections settings page. The roadmap and
existing PRs were checked. This is a small authorization fix for the
existing settings path.

## What Changed

- Derive experiment-management authority from the current trusted Cloud
stack Admin role on each request.
- Apply that authority only to the experimental settings PATCH route.
Keep general settings, environment defaults, and task drain restricted
to instance admins.
- Preserve existing hidden-setting checks, managed feature restrictions,
and activity records.
- Add role and role-downgrade regressions, allowed and denied route
cases, staging coverage, and updated OpenAPI and access documentation.

## Verification

- Before the fix, the Admin fast-response request returned 403 and the
new role assertions failed. The first sandboxed HTTP run could not bind
local sockets; the regression run outside that sandbox reached the
expected authorization failures.
- Focused Cloud actor, settings route, managed-overlay, and health
tests: 149 passed.
- `node scripts/check-maps.mjs` and `git diff --check` pass.
- `pnpm -r typecheck` and `pnpm build` pass.
- Greptile reviewed commit `7c33b90066121f03183579d2fae2650755e025e4` at
5/5. Its one style finding is fixed and resolved.
- The full local `pnpm test:run` did not pass. It reported two early
collection failures before workspace build outputs were ready and a
15-second timeout in the unchanged AgentMail recovery test. The run was
stopped after those failures while the full CI suite continued. No full
local-suite pass is claimed.
- The first current-head CI attempt failed the unchanged Runner test
“preserves a real listening process and its marker when a health RPC
times out” with `runner_remote_process_identity_unavailable` /
`process_identity_changed`. The test passes in a local isolated run.
This recovery does not establish the CI failure's cause. [Failed
job](https://github.com/paperclipai/paperclip/actions/runs/39332989655/job/118336142261).
The failed shard passed on its single rerun. The failure cause remains
unconfirmed. All CI checks on commit
`7c33b90066121f03183579d2fae2650755e025e4` are now green. [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/39332989655).
- No deployed staging behavior is claimed. Manual acceptance after
deployment: as a stack Admin, enable Fast responses, reload, and confirm
Connections appears. Confirm Members and Support users still cannot save
experiments.

## Risks

- This intentionally extends experimental settings write access to Cloud
stack Admins. It does not persist an instance-admin role or expand other
instance controls.
- Production retains its existing managed feature policy. Staging
retains its editable experiment allowlist and protected execution and
owner-elevation settings.
- No database migration or UI change is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution, and browser inspection. The runtime does not expose an exact
model snapshot or context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 21:48:00 -05:00
DottaandPaperclip ac9459bae3 feat: configure company agent observability (#16874)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Companies use several agent harnesses and execution environments.
> - Operators need one place to select where native agent logs and
traces go.
> - An inference proxy would add a dependency to the agent request path.
> - This pull request supplies native OpenTelemetry settings to managed
Codex and Claude Code processes.
> - Company secrets protect destination credentials. Export failure does
not change the task result.

## Linked Issues or Issue Description

**Subsystem affected**

Company settings, shared contracts, database, server, adapters, and
runner.

**Problem or motivation**

An operator cannot configure a company destination for native agent logs
and traces. Manual harness configuration is hard to apply across local
and remote execution.

**Proposed solution**

Add one company destination: Braintrust or a custom OTLP HTTP/protobuf
endpoint. Apply a transient snapshot when execution preparation starts.
Keep credentials out of API reads, task environment manifests, recovery
files, and command arguments. Replace a warm process when its export
settings change and preserve its conversation.

**Alternatives considered**

Multiple destinations can use a user-operated collector for fan-out.
This change does not add a collector, an inference proxy, or transcript
scraping.

**Roadmap alignment**

This extends company settings, secret management, and support for
multiple harnesses. The maintainer requested this implementation.
Searches for observability and Braintrust found no duplicate company
destination feature. Related infrastructure tracing work in #16236 has a
separate content-free contract.

## What Changed

- Add strict schemas, revision checks, encrypted secret storage,
activity records, and board-only settings routes.
- Add explicit synthetic logs and traces tests with a bounded HTTP
request and sanitized results.
- Supply native Codex configuration and Claude Code environment values
through local and remote launch paths.
- Preserve Codex login refresh through the existing managed credential
link. Retain unsaved drafts across background reads. Require complete
custom header re-entry when the endpoint origin changes.
- Separate process export compatibility from conversation identity.
Isolate generated Codex configuration and preserve resumable sessions,
including remote home refresh after export changes.
- Advance the immutable Pi/Copilot source attestations for shared
environment and sidecar code changes. Keep their provider payloads
unchanged and retain prior identities for decoding.
- Redact destination credentials throughout launch diagnostics and HTTP
logs.
- Add the production settings form, Storybook states and journey,
documentation, and feature/access map entries.

## Verification

- Real managed local Codex 0.160.0 and Claude Code 2.1.286 tasks
exported to Braintrust. API readback and the Braintrust UI showed the
test records. A real remote Daytona Codex task exported through the
legacy `exec` path.
- Live tests exercised content capture off, export disabled, warm
conversation reuse, and a destination that accepted connections but
never replied. Agent work succeeded and export requests stopped at the
configured three-second timeout. The UI connection test returned “Test
payload accepted”; the saved API key stayed masked after reload.
- All current-head CI test lanes, typechecks, packaging/build checks,
and required aggregates passed on
`71e994b2be9c8fb6d6e938a4ad7cf4c95fdafe31` ([CI
run](https://github.com/paperclipai/paperclip/actions/runs/39319194318)).
Greptile completed on that exact commit at 5/5, with zero unresolved
review threads. The additional optional Paperclip Review check remains
queued waiting for its assigned agent.
- Local workspace typecheck and build passed. Focused verification
included 239 combined runner tests, 789 API/native-session tests, 107
Codex home/ACP/sandbox-site tests, 204 feature/privacy tests, and the
affected Rust suites. Migration, token, map, Storybook catalog, and
Storybook build checks passed during acceptance. These checks ran at the
relevant implementation or merge boundary; the current-head CI is the
complete final gate.
- Managed AI credential projections now receive enabled and disabled
company export settings; genuine caller-owned Codex homes remain
untouched. Two regressions reproduced the skip before the repair. All
276 engine/lifecycle tests and adapter-utils typecheck pass afterward.
The launched TOML retains provider/MCP settings, credential continuity,
and redaction.
- Settings saves, removal, and the real company-deletion owner acquire
secret mutation ownership before database locks. PostgreSQL NOWAIT
probes cover all three paths; concurrent save/rotation/removal/deletion
requests settle correctly. The affected suites passed 137 tests,
followed by all 32 service/connection cases after expanded deletion
coverage. Server and Codex adapter typechecks pass.
- Codex source profiles use the effective CLI argument list. A spawned
fixture CLI reproduced both ignored-argument failures; all three
precedence/fallback cases now pass.
- Real tar staging reproduced remote conversation loss in both API-key
and subscription modes before the repair. Both now pass: changing
managed settings preserves sessions, archives, native indexes, and
refreshed subscription credentials. A corrupt upload preserves the prior
config and conversation.
- Preserved failure evidence: initial CI caught stale Pi/Copilot source
attestations, a missing test-harness helper, and packaging fixture/probe
mismatches; those were corrected and covered by focused checks.
Superseded heads had intermittent attachment-composer timeouts despite
successful uploads; [the original trace and screenshot remain
available](https://github.com/paperclipai/paperclip/actions/runs/39309701089/artifacts/12215109203).
The same attachment case passed on the final head: [25/25 tests in shard
3](https://github.com/paperclipai/paperclip/actions/runs/39319194318/job/118288637051).
- A connection-fault fixture also exposed a stale pooled socket after
its hang case; closing fixture responses isolated scenarios and the full
affected suite passed.
- The broad local test run was not clean: database resource failures,
timeouts, an older cached module graph, and a stale optional-resource
manifest assumption were recorded. Focused checks passed after the
documented fixes. This does not establish a clean full local pass.
- Native remote Codex, remote Claude, and the full ACPX Codex bridge
remain unqualified. The direct Codex resume test does not establish full
bridge acceptance.

## Risks

- Native exporters perform network requests from each execution
environment. A server connection test does not prove remote
reachability.
- Prompts, assistant responses, and tool content are enabled by default
when supported. Native caps and harness-specific controls still apply.
- Caller-owned Codex homes are unsupported for automatic configuration
and remain untouched.
- Native harnesses and their tool processes can inspect export
credentials. The settings form and documentation advise using
ingestion-scoped credentials. Retained headers remain bound to their
saved scheme, host, and port; changing origin requires fresh values or
removal.
- Migration 0357 adds a company-scoped row and secret reference. It is
safe to reapply. Secret removal and company deletion use the existing
secret lifecycle.
- Slow or unavailable destinations must not block task settlement.
Export settings take effect at the next execution preparation boundary.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, code execution, and
browser verification. The exact serving model ID and context window size
are not exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 15:36:04 -05:00
DottaandPaperclip 51c81d0129 feat(sentry): link run errors to task and run pages (#16803)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Operators use Sentry to inspect run and server failures.
> - Reports identify the run and task, but the existing links open API
records.
> - Operators need a direct path from an error to the normal task and
run pages.
> - This change adds navigation links through the shared server Sentry
hook.
> - Saved company and task records determine the links. A failed lookup
preserves the error report.

## Linked Issues or Issue Description

Related work: #16300 added saved run API references.

**What existing behavior does this improve?**

Sentry event context for run failures, Stop timeouts, and server errors
from an authenticated agent run.

**Current behavior**

Reports show task and run UUIDs. The existing run URL opens an API
record. The operator must find the normal task page separately.

**Proposed behavior**

Add `task_url`, `run_url`, and `task_identifier` tags when saved run
metadata is available. Add `taskUrl` and `runPageUrl` to the event
context. Use the company prefix and saved task identifier in normal
board URLs.

**Reason and benefit**

An operator can open the task or run directly from the error report.

## What Changed

- Add a shared Sentry hook for events with a known run identity.
- Read only the saved company prefix, task identifier, and run identity.
Keep the lookup within the recorded company and agent.
- Include authenticated run identity on request errors. Use the existing
run identity on Stop errors.
- Bound optional lookups to one second and eight unfinished reads.
Preserve the original event on failure.
- Keep the existing API references and error grouping.
- Document the fields and add SDK, timeout, and company-boundary
coverage.

## Verification

- Passed: `pnpm -r typecheck` and the final server TypeScript check.
- Passed: `pnpm build`.
- Passed: 332 tests across the affected server suites, plus 25 final
link and real Sentry SDK tests after review fixes. The SDK test checks
outgoing event envelopes for run failures, request errors, and Stop
errors. It checks that unrelated errors receive no links. The database
test verifies saved metadata and company boundaries. It passes with
embedded PostgreSQL and with an explicitly configured disposable test
database.
- Passed in CI on `bec0cdac853efdef71041d59a66261941827653e`: the full
test matrix, typecheck, build, Runner checks, browser suites, and the
real Sentry SDK contract. All 56 active checks pass. Two Storybook jobs
are skipped.
- The full local `pnpm test:run` attempt first failed because PostgreSQL
library links were missing. The package setup script repaired the
install. The affected database suites then passed. The repaired full
local run was stopped after all CI test jobs passed; it did not complete
locally.
- Greptile: 5/5 on the final commit. All three review threads are
resolved.
- Read-only browser inspection confirms that Sentry displays context URL
values with an Open link control.
- No live test event was posted. Validation uses the real SDK with an
in-memory transport.

## Risks

- A slow or unavailable database can prevent optional links. It does not
discard the error. At most eight unfinished lookups can remain active.
- Missing or unsafe public origins omit the links. URLs longer than the
tag limit remain in context.
- This change applies to newly captured events. It does not update
existing Sentry events.
- The change adds no schema, permission, or first-party Telemetry
contract change.

## Model Used

OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with tool use and code
execution. The exact context window is not exposed in the session
metadata.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-19 06:47:47 -05:00
DottaandPaperclip e6dcc6b095 Connect personal Muse through secure native execution and Vault OAuth (#16448)
Add company-scoped Muse native control, OAuth, wake receiver and recovery.

Recheck current authority after waits and before effects or receipt replay. Preserve task-read scope, assignment expiry, independent holds, nonblocking reassignment and truthful cleanup provenance.

Retain source trust on progress comments and verify current Muse V8 authority through real signed-JWT HTTP requests. Preserve the upstream trimmed ordinary authentication lookup.

Refs #16448.

Preserve captured canonical queue recovery and concurrent environment preparation while retaining the scoped Muse V8 exclusion. Normal migration 0354 and existing Muse SQL remain unchanged. Fresh current-head Greptile review is 5/5 with the project-write finding addressed. Required CI and independent technical review gate the merge; maintainer override is reserved for formal approval rules.

Recheck current Muse project-write authority after all receipt/task/run/admission waits, before creation or replay. Real PostgreSQL regressions reproduce four late effects/replays on the old route; the repaired three-file treatment passes 354 cases with no skips, and the scoped server typecheck passes. Ordinary/Dot behavior is preserved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-10-17 22:44:36 -05:00
Devin FoleyandPaperclip 63fbb714e6 fix(server): release idle tracking after aborted static UI files (#16685)
## Thinking Path

> - Paperclip runs work for AI agents.
> - Hosts must prove that accepted work has finished before idle sleep.
> - HTTP request tracking keeps unfinished writes active after a client
leaves.
> - Static file middleware has no returned completion promise.
> - An aborted UI download can leave its request active forever.
> - This PR completes only a matched file response and keeps other work
tracked.

## Linked Issues or Issue Description

Refs #15522. Found while qualifying #16679. A search found no duplicate
static-download idle fix.

**What happened?**

Closing a client during a static UI download leaves one tracked request
active. The response has closed, but the file middleware never calls the
wrapped response end method. The instance cannot report idle after that
request.

**Expected behavior**

A closed, matched static file response must not leave a permanent work
hold. Unfinished application writes must remain tracked. A missing file
must keep the normal fallback and error paths.

**Steps to reproduce**

1. Serve a large local file through the static UI middleware with idle
admission enabled.
2. Start the download over HTTP.
3. Disconnect the client after its first data chunk.
4. Observe that the tracked request remains active with the original
middleware.

**Paperclip version or commit**

Reproduced on the behavior in
`d30648dee89720358271c276b9fc28cf83f18be7`. This branch starts at
`cfa9dce295970a85b3d8ee14cab0eccf50bc72e2`.

## What Changed

- Mark the completion of a known read-only response without dropping
pending middleware promises.
- Wrap static UI file serving. A successful file match is required
before a closed response can complete its tracking. A late match
rechecks an already closed response. Missing files keep the normal
fallback.
- Use the wrapper for hashed assets, other static UI files, and public
development files. Preserve caller headers and cache options.
- Test both HTTP abort timings, nested routers, conservative
mounted-application holds, parent writes, mutating GET fallbacks, and
callback-only work.
- Update the idle contract and its verification map. Correct the
paused-message recipe from hosted qualification: an explicit Pause
cancels older queued work, so test new accepted input after the pause
and await the normal resume lifecycle.

## Verification

- The original two real stream-abort regressions fail with the original
Express static behavior.
- Three review regressions fail on
`e260f4694afab8447b0cb203f439cb3ccad254e8`: a client closing before file
lookup, an unfinished parent in a mounted application, and an unfinished
parent in a root without installed handler tracking. Fixed sources were
restored byte for byte after the comparison.
- Two focused suites: 16 passed. They cover nested routers, both abort
timings, hidden parent writes, late missing-file fallbacks, contents,
HEAD, conditional responses, and caller headers.
- Module and map checks pass. Map checks are structural only.
- `pnpm -r typecheck` and `pnpm build`: passed on
`f242ee739344484995d3addf81635512370778c8`.
- Full CI matrix: 54 successful checks and 2 intentional Storybook
skips.
- Apex: 5/5 on the exact current head, with zero unresolved threads.
- Full local maintained test groups are still running before final
handoff.
- The initial local broad run was stopped before changing source for the
P1 review findings. Two file-mode fixture failures in that run came from
the scratch runner's inherited `0077` mask; the same suite passes with
the normal child test mask `0022`, without source or host configuration
changes. The replacement run uses `0022` for test children and
command-scoped `init.defaultBranch=master` for existing Git fixture
assumptions.
- This PR has not been deployed to a hosted environment.

## Risks

Only a matched file served by the admitted application with installed
handler tracking can complete this way. Routers share that scope.
Mounted Express applications and roots without handler tracking keep the
existing conservative response hold; this change does not claim their
hidden writes are tracked. An unfinished tracked parent promise still
blocks sleep after a file client disconnects. Redirects, missing files,
application callbacks, and other GET requests retain their guards. No
authentication, company scope, provider cleanup, or scheduling gate is
removed. No schema or API response changes.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, source editing, and shell
tools. The session does not expose a more specific serving model ID or
its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-17 20:28:57 -07:00
Devin FoleyandPaperclip 4de6335c3b perf(server): trim saved run JSON from agent authentication (#16577)
## Thinking Path

> - Paperclip coordinates AI agents that work for companies.
> - Agent JWT authentication checks each request against its saved run.
> - Saved runs can contain large prompts and result payloads.
> - The first run lookup needs only identity fields, cancellation state,
and the conversation flag.
> - Reading all saved JSON transfers data that this lookup does not use.
> - This PR selects the required JSON fields in PostgreSQL and preserves
the existing authorization decisions.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

The initial saved-run lookup in agent JWT authentication.

**Subsystem affected**

Server authentication and database queries.

**Current behavior**

The lookup transfers the full context and result JSON on each request.
It uses only the cancellation state and conversation flag from those
values.

**Proposed behavior**

Select the required JSON fields in PostgreSQL. Keep the run, company,
and agent predicates. Read current state on every request.

**Reason and benefit**

A synthetic lookup with a 96 KiB prompt and 4 KiB result returns 102,520
JSON bytes before this change and 134 bytes after it. This reduces data
transferred by this query.

**Breaking changes**

None intended. Identity fields, write revocation, read access, and
cancellation response codes retain their current behavior.

Related: #9280 addresses authentication round-trip counts. This PR only
reduces the data returned by the existing initial run lookup.

## What Changed

- Extract the initial run lookup and select only the JSON fields that
authentication consumes.
- Add real PostgreSQL tests for legacy JSON shapes, fresh cancellation
state, payload size, and run/company/agent scope.
- Add a reproducible benchmark that loads the original query from an
explicit baseline commit.
- Document the lookup and benchmark.

## Verification

- Baseline: `06689c1b3410000bcd65ff40cb469e600209c4db`.
- `pnpm -r typecheck`: passed.
- `pnpm build`: passed.
- Final server typecheck: passed.
- Final-head GitHub checks: all green at
`2be1c38e6bf51616b69993cf84cee1eef9657517`. Apex: 5/5, zero unresolved
threads, conflict-free.
- `pnpm check:module-boundaries`: passed. The standalone synthetic
benchmark now uses the existing root benchmark directory and resolves
Drizzle through the server workspace.
- Focused HTTP middleware and database tests: 29 passed on real Homebrew
PostgreSQL 16.14. The SQL test compares 270 combinations of status,
cancellation JSON, and conversation JSON. It also checks fresh state and
company/agent/run scope.
- Broad native verification: 873 passed across the HTTP middleware, SQL
projection, run diagnostics, and native executor suites.
- The default local full command completed its general server lane with
12,090 passed and 8,885 skipped tests. It could not finish the root
workflow: the host could not start the bundled PostgreSQL; a diagnostics
assertion matched a directory name; and Git fixtures retained the host
default branch. These failures remain in the verification record. All 68
failed files now pass on replay with disposable Homebrew PostgreSQL, an
isolated checkout, and command-scoped Git defaults. The 68-file replay
passed 2,588 tests and left two email skill-resolution failures. Moving
the checkout outside an unrelated ancestor skills directory resolved
those last failures: all 44 tests in that file passed. Product code and
test assertions were unchanged for these host adjustments.

Benchmark command:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-agent-authentication.ts --baseline-ref=06689c1b3410000bcd65ff40cb469e600209c4db
```

The final benchmark ran at treatment commit
`2be1c38e6bf51616b69993cf84cee1eef9657517` with a clean worktree. The
local run used PostgreSQL 16.14 and the maintained test constructor seam
to start disposable Homebrew clusters. The bundled PostgreSQL binary
could not start on this host. Both versions use the same schema and
predicates. Each case has two warmup rounds and ten measured rounds with
alternating order. Random content prevents unrealistic compression.
Complete authentication decisions match.

| Context / result | Before JSON bytes | After JSON bytes | Before
median | After median |
| --- | ---: | ---: | ---: | ---: |
| 1 KiB / 4 KiB | 5,240 | 134 | 0.157 ms | 0.157 ms |
| 96 KiB / 4 KiB | 102,520 | 134 | 0.331 ms | 0.140 ms |

These are local query round trips, including result transfer and
decoding. The small control has no material latency gain. The large case
improves by about 2.4x. Both plans read the same single row. The primary
result is reduced response size. This does not measure HTTP latency,
server CPU savings, or fleet-wide cost savings. Full samples and plans
are retained with the verification evidence.

## Risks

- JSON types must retain their meaning. The real database tests compare
the consumed authorization contract with the original projection,
including SQL null, JSON null, scalars, arrays, objects, string
booleans, and cancellation states.
- The change covers the first JWT run lookup. Later run identity capture
keeps its existing full-run reads and transaction locks. Those reads
remain a separate cost.
- The change adds no schema migration or authorization cache.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, code editing, and shell
tools. This session did not expose a more specific serving model ID or
its context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-16 11:44:14 -07:00
514d8738cf fix: keep the JWT run-mismatch audit when the claimed run is missing (#13183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Rejected agent JWTs whose run header disagrees with the signed claim
are audited into `activity_log` so operators can see spoof attempts
> - The audit insert writes the claimed run id into `run_id`, which is a
real foreign key to `heartbeat_runs`; a bogus claim violates the FK and
the catch swallows it, so the audit row is silently lost exactly when it
matters most
> - This pull request verifies the claimed run exists before trusting it
into the FK column; the claim itself stays in `entityId` and `details`
> - The benefit is a reliable audit trail for rejected credentials

## Linked Issues or Issue Description

Refs #10498 — the anonymous fall-through half was fixed on master
already (every failed-credential branch now returns an explicit
401/403/422); this closes the residual FK-swallowed audit defect.

## What Changed

- `server/src/middleware/auth.ts`: `auditAgentJwtRunHeaderMismatch`
looks up the claimed run id and populates `runId` only when the row
exists; `entityId` and `details.claimRunId` carry the claim either way.
- `server/src/__tests__/agent-auth-middleware.test.ts`: a mismatch whose
claimed run does not exist still produces the audit row with `runId`
omitted.

## Verification

- `pnpm --filter @paperclipai/server vitest run
src/__tests__/agent-auth-middleware.test.ts` — 17 tests pass.
- Before: a run-header mismatch whose claimed run id was absent from
`heartbeat_runs` violated the `activity_log.run_id` FK, and the catch
swallowed it — no audit row. After: the row lands with `runId` null and
the claim preserved in `entityId`/`details`.

## Risks

- Low risk. One extra indexed select on a rare rejection path; the audit
payload is unchanged apart from `runId` now being conditional on
existence.

## Model Used

- Anthropic Claude — SWE-2 Max agent via Devin CLI, tool use and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Waseem Ilyas <1478353+Waseemilyas@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-10-15 15:43:56 -07:00
3a440d28c4 fix(server): read existing grants before seeding role defaults (#16365)
`insertMissingPrincipalGrants` now reads the existing permission keys first and inserts only missing grants. In the steady state no INSERT runs, so cloud-tenant auth resync no longer waits on row locks held by concurrent grant edits until the statement timeout (Sentry PAPERCLIP-BACKEND-3S).

Co-Authored-By: Paperclip <Paperclip-Paperclip@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-14 20:01:34 -07:00