Commit Graph
4348 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 92dbf39e6f feat(analytics): attribute agent work and onboarding steps to a person on Cloud (#22260)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud sends product events straight to PostHog: server
events from `server/src/services/posthog-direct/` and browser events
from `ui/src/lib/analytics/`
> - A person-level funnel goes from sign-up to the first finished task
> - Agent-driven events (`agent.task_run`, `agent.task_completed`,
`agent.first_heartbeat`) used `workspace:<stackId>` as the distinct id,
so the funnel broke at those steps
> - The in-workspace onboarding wizard sent no step events, so we could
not see where people stop before their first agent
> - This pull request attributes agent work to the responsible person on
the Cloud direct path, and adds consent-gated onboarding step events in
the browser
> - The benefit is one PostHog person per customer across sign-up,
onboarding, agent runs, and task completion. Self-hosted output does not
change.

## Linked Issues or Issue Description

**Subsystem affected**
Cloud product analytics: Cloud direct PostHog delivery (server) and
browser product analytics (UI onboarding wizard).

**Problem or motivation**
A PostHog funnel from sign-up to the first finished task cannot follow
one person. Agent-driven events run in the background, with no signed-in
request, so they use the workspace id as the distinct id. The onboarding
wizard also sends no events, so we cannot see which step people leave.

**Proposed solution**
Give background emit sites a narrow scope that names the person
responsible for the work. Use the task creator first, then the task's
responsible user, then the run owner. Keep the signed-in request user
first. Add `attributed_via` to every Cloud direct event. Add four
browser events to the onboarding wizard. Send them through the existing
analytics module, so the consent rules stay the same.

**Alternatives considered**
A lookup in the transport at send time. We did not use it: it adds
database reads to the telemetry path and loses the emit site's context.
A new contract event for onboarding steps. We did not use it: the server
does not see the wizard steps, and the UI must never send
`PAPERCLIP_EVENTS` names.

**Additional context**
Self-hosted installs keep the collector path. The collector envelope
stays byte-identical (a test checks it).

## What Changed

- `server/src/telemetry-company-scope.ts`: add `withTelemetryActor({
companyId, person }, fn)`, `scopedTelemetryPerson()`, and
`resolveTelemetryPerson()`. The resolver picks the task creator, then
the task's responsible user, then the run owner. It ignores
`local-board`. `withTelemetryCompany` now calls `withTelemetryActor`.
- `server/src/services/posthog-direct/request-context.ts`: a signed-in
request user wins (`attributed_via: request`). Without one, the scoped
person is used.
- `server/src/services/posthog-direct/event-shape.ts`: every Cloud
direct event (including `cloud.workspace_started`) has `attributed_via`
(`request`, `task_creator`, `task_assigner`, `run_owner`, or `none`). An
event with a person keeps today's person-event shape (no
`$process_person_profile: false`).
- `server/src/services/agent-task-run-telemetry.ts`: `agent.task_run`
reads the task creator and responsible user in the existing agent query
(one left join, no new query), and the run's `responsibleUserId`.
- `server/src/routes/issues.ts`: `agent.task_completed` uses the issue
row that the route already has, and the agent actor's `onBehalfOfUserId`
as the run owner.
- `server/src/services/heartbeat/agent-status.ts`:
`agent.first_heartbeat` uses the agent's `createdByUserId` (already
loaded) as the run owner.
- `ui/src/lib/analytics/product-analytics.ts`: add
`captureProductAnalyticsEvent()`. It captures only with a browser
analytics config and granted consent. Events from before posthog-js
loads wait in a queue (at most 50) and keep their time. Withdrawn
consent or a change of signed-in person drops the queue.
- `ui/src/lib/analytics/onboarding-analytics.ts` and
`ui/src/components/OnboardingWizard.tsx`: capture `onboarding_opened`,
`onboarding_step_viewed`, `onboarding_step_completed` (with `step` and,
for `connect_model` and `launch`, `outcome`), and
`onboarding_dismissed`. `step` is `create_agent`, `connect_model`,
`review`, or `launch`. No event has free text.
- Docs: `packages/shared/src/telemetry/README.md` (Cloud Direct
Delivery) and `doc/cloud-browser-analytics.md` (new "Onboarding events"
section).

## Verification

- `server/src/__tests__/posthog-direct-telemetry.test.ts`: attribution
matrix (creator, assigner, run owner, none stays on
`workspace:<stackId>` with `$process_person_profile: false`), the
request user wins over the scope, and the self-hosted collector envelope
is byte-identical with a person scope.
- `server/src/__tests__/agent-task-run-telemetry.test.ts` (embedded
Postgres): `agent.task_run` scope for creator, assigner, run owner, and
nobody; another company's task is never read.
- `server/src/__tests__/issue-telemetry-routes.test.ts`:
`agent.task_completed` scope for creator and assigner.
- `server/src/__tests__/heartbeat-process-recovery.test.ts`:
`agent.first_heartbeat` scope is the agent owner.
- `ui/src/components/OnboardingWizard.test.tsx`: a launch for the old
company that settles after a switch to another company does not end the
new analytics session.
- `ui/src/lib/analytics/onboarding-analytics.test.tsx`: each transition
fires once, held events drop on a person change, a stale opening step is
not a view, dismissal fires once, nothing is captured when analytics is
off or consent is `denied` or `ask`, and queued events keep their time.
- `ui/src/lib/analytics/product-analytics-events.test.ts` still passes
(the UI never captures a `PAPERCLIP_EVENTS` name).
- Local Cloud-mode run: an isolated instance (own Postgres, trusted
Cloud tenant headers, `PAPERCLIP_POSTHOG_HOST` set to a local capture
server, never real PostHog). A stub stood in for posthog-js and sent
through the `/_ph` proxy.
- A user created a task and assigned it to the agent. `agent.task_run`
arrived with `distinct_id: paperclip-id:local-user-1`, `attributed_via:
task_creator`, the raw `task_id`, and no `$process_person_profile`. (The
run itself failed locally because the Cloud runner is not available. The
event still shows the attribution.)
- `agent.first_heartbeat` arrived with the same person and
`attributed_via: run_owner`. `server.started` kept
`workspace:stack-local` with `attributed_via: none`.
- Headless Chromium walked the wizard: `onboarding_opened`
(create_agent), `onboarding_step_viewed` (create_agent),
`onboarding_step_completed` (create_agent), `onboarding_step_viewed`
(connect_model), `onboarding_step_completed` (connect_model, `api_key`),
`onboarding_step_viewed` (review), `onboarding_step_completed` (review),
`onboarding_step_completed` (launch, `succeeded`). Each fired once, with
the signed-in user as the distinct id. Events from before posthog-js
loaded arrived with their original time.
- A second walk left the onboarding page on the connect step and sent
one `onboarding_dismissed` (connect_model).

## Risks

- Low risk for self-hosted: the collector path never reads the new
scope, and a test checks the envelope bytes.
- Cloud: agent events move from the workspace person to a real person.
Insights that count `workspace:` distinct ids for these events will
change. `attributed_via` lets a query filter or split them.
- `task_assigner` uses the task's responsible user. Paperclip does not
store who assigned a task, so this is the person who owns or delegated
the task.
- Privacy: the new data is Cloud-only and uses ids that Cloud events
already carry. No new free text, email, or name goes on an event.
**Privacy review requested** for the Cloud attribution change (per
`rules/data-paths.mdc`). The generated contract does not change:
`attributed_via` is a Cloud-only property, like `stack_id` and
`user_id`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, headless browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:44:45 -06:00
e2cee049f9 fix: show instruction saves in history, credit the real writer, and speed up the agent tool-rule list (#19515)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Operators and agents change agent instructions (AGENTS.md), and they
must see who changed them and when
> - Agent files replaced instruction revisions. Saves no longer write
revision rows, so instruction history is empty
> - The current revision is stamped with the reader and the date
1970-01-01, so it gives false audit data
> - Separately, an agent that lists tool rules gets a timeout when many
rules name the same resources
> - This pull request reads history from the save records that Paperclip
already writes, credits the real writer, and checks each tool-rule
resource one time
> - The benefit is a correct audit trail for instruction changes and a
tool-rule list that agents can read

## Linked Issues or Issue Description

No public issue exists. I searched open and closed issues and PRs for
these problems and found none.

**What happened?**
1. `get_agent_instruction_history` (and the history API) returns 0
revisions for an agent whose AGENTS.md was saved many times. The
activity log shows each save as `agent.files_updated` with the path,
content hash, actor and run.
2. `read_agent_instructions` returns a current revision with `createdAt`
1970-01-01. Its actor is the agent that reads it, and its `sourceRunId`
is the run that reads it.
3. `GET /api/companies/{companyId}/tools/policies` from an agent fails
with a transport timeout most of the time when the company has many tool
rules. A board user gets the list without a problem.

**Expected behavior**
1. History lists each save, newest first, with its writer, run, source
and time.
2. The current revision names the agent or user that wrote those bytes,
and the time of that save. If no save is recorded, it names no writer.
3. The agent gets the tool-rule list in normal time, with the same rules
hidden as before.

**Steps to reproduce**
1. Save an agent's instruction entry file (board editor,
`update_agent_instructions`, or a run that edits the file). Read the
history. It is empty.
2. As a different agent, call `read_agent_instructions` on that agent.
The revision shows the reader and 1970-01-01.
3. Create about 40 tool rules that name the same connection and agent.
As an agent, list `/tools/policies`. Each rule does a new set of access
checks for each resource, one after another.

**Paperclip version or commit**
`master` at `819420a60a`.

**Deployment mode**
Hosted. The new tests reproduce all 3 problems on `master`.

## What Changed

- `agent-instruction-revisions.ts`: history merges the legacy revision
rows with the `agent.files_updated` save records for the entry file. The
order is `(created_at, id)` descending, and cursors keep PostgreSQL
timestamp precision.
- `agent-instruction-revisions.ts`: the current revision takes its
writer, run, source and time from the latest save of the same content
hash. A reader is never the writer. Bytes with no save record have no
writer.
- `agent-instruction-revisions.ts`: an identical re-save over bytes with
no record is marked `source: "adopted"`. It repairs the runtime journal
but is not listed as a save.
- `agent-instruction-revisions.ts`: a history ID can be a legacy
revision, a save record, or the current content token. A save record
keeps only metadata. If its bytes are no longer current, a read returns
404 with `INSTRUCTION_REVISION_CONTENT_NOT_RETAINED`.
- `agent-file-store.ts`: saves record `byteLength`. A run collection
that changes the entry file records its path, hash and size, so history
shows saves from runs.
- `connection-management-authz.ts`: `filterConnectionManagementPolicies`
keeps one read decision for each distinct resource in a request, and
loads each application or catalog entry one time. A rule is still hidden
when any resource that it names is hidden. Board callers do not use this
path.
- Tests: new tests for save history, paging, writer credit, adoption,
and the tool-rule list. Old assertions that expected empty history now
check the exact saves.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/agent-instruction-revisions.test.ts
src/__tests__/agent-instruction-working-copies.test.ts
src/__tests__/agent-directory-working-copies.test.ts
src/__tests__/built-in-agents.test.ts
src/services/native-runtime/agent-instruction-tools.integration.test.ts
src/__tests__/tool-policy-list-authz.integration.test.ts`
- With the fix: all pass (revisions: 28 passed, 1 skipped on Linux;
other 5 files: 230 passed).
  - With the new tests on `master` code: 13 tests fail, as expected.
- `src/__tests__/connection-management.integration.test.ts`: 90 passed.
- The new tool-rule test lists 40 rules that name 1 connection and
counts the connection reads. It needs fewer than 10.
- I could not run the full server typecheck in my sandbox because of a
memory limit. CI runs it.
- CI on head `a2f0b825`: all checks pass, including `ci / verify` and
`ci / e2e`.

## Risks

- History now shows entries whose ID is an activity row ID, not a
revision row ID. Clients that read or restore an older save get a clear
404 because Paperclip no longer keeps those bytes. The latest save and
legacy revisions still read and restore as before.
- Saves recorded before this change have no `byteLength`. For those,
history shows the current size when the hash matches, and 0 otherwise.
- Stock updates of built-in agents do not record a file path, so they do
not show in history.
- No migration. No change to the stored data model.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits, test runs). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: lucktastic <13757243+lucktastic@users.noreply.github.com>
2026-11-18 16:41:53 -08:00
Devin FoleyandPaperclip 75ae377438 feat(ui): offer to cancel active subtasks (#22228)
Offer to cancel readable active subtasks when a task is canceled. Show a counted heading, status icons and titles, with choices to cancel only the parent or also its listed subtasks. Include waiting chat descendants and skip terminal or moved tasks observed before each write.

Verified with focused UI and database tests, final CI, Greptile 5/5 with no unresolved comments, and manual acceptance.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-18 16:41:06 -08:00
DottaandClaude Opus 5.5 c600d69242 Let plugin connectors declare their tool timeout (#22249)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Plugins can serve connector tools from their worker (`transport:
"plugin"`, added in #22004, #22019 and #22201).
> - The tool gateway runs these tools with a fixed 10 s default timeout.
> - Some plugin tools call backends that take 12–20 s. These calls fail
every time with "Tool execution timed out", even though the plugin's own
HTTP client waits much longer.
> - The plugin connector design said that the connector, not a fixed
gateway constant, sets the timeout.
> - This pull request lets a plugin method or a single static tool
declare `timeoutMs`, and gives plugin-served tools a 30 s default.
> - The benefit is that a plugin connector can serve slow tools like
other connectors can, within the same 60 s gateway ceiling.

## Linked Issues or Issue Description

Refs #22004, #22019, #22201. Refs #20746.

**Relationship to #20746.** This PR builds alongside #20746. It does not
supersede it. #20746 changes the default for legacy top-level plugin
tools (provider `paperclip_plugin`) to 60 s. This PR changes only
plugin-served connector tools (provider `paperclip_plugin_connection`,
`transport: "plugin"`). The two PRs touch different branches of the
gateway and can merge in either order. This PR does not pin the legacy
default in its tests, so #20746 does not break them.

**Current behavior**
A plugin connector tool always times out after 10 s, unless the caller
sends `timeoutMs`. Agents and runners do not send it. The worker RPC
also stops at 30 s.

**Proposed behavior**
A plugin can declare `timeoutMs` (1000–60000 ms) on a `transport:
"plugin"` method or on one static tool. The gateway uses that value. If
nothing is declared, the gateway waits 30 s.

**Reason and benefit**
Plugin tools whose backends take longer than 10 s can now work. Plugin
authors control the value in the manifest, and the gateway keeps its 60
s ceiling.

**Breaking changes**
None. Existing manifests stay valid. Only plugin-served tools get the
new 30 s default.

## What Changed

- `packages/shared`: `timeoutMs` is a new optional field on
`ConnectionMethodDef` and on `PluginConnectorToolDef`. The validator
accepts integers from 1000 to 60000 only. It rejects `timeoutMs` on
methods that do not use `transport: "plugin"`. New exported constants:
`PLUGIN_CONNECTOR_TOOL_TIMEOUT_MIN_MS`,
`PLUGIN_CONNECTOR_TOOL_TIMEOUT_MAX_MS`, and
`PLUGIN_CONNECTOR_TOOL_DEFAULT_TIMEOUT_MS` (30000).
- `server/src/services/plugin-connector-runtime.ts`:
`pluginConnectionToolTimeoutMs()` resolves the timeout in this order:
the tool's value, then the method's value, then 30 s. The worker
`executeTool` RPC now gets the gateway timeout plus 1 s. Thus the
gateway's `tool_timeout` error is what callers see, and the 30 s RPC
default no longer cuts off a longer declared timeout.
- `server/src/services/tool-gateway.ts`: `executePluginConnectionTool`
now applies the timeout itself, after it loads the connection. All three
plugin paths use it: session, assigned-tool and direct calls
(`executeTool`), test calls (`executeTestCall`), and approved action
execution. If a caller sends `timeoutMs`, that value wins, as before. It
is still clamped to 60 s. Approved actions keep their fixed 60 s budget.
- Other transports do not change. Remote MCP, local stdio, built-in and
legacy plugin tools keep their timeouts. (#20746 separately proposes a
60 s default for legacy plugin tools.)
- Docs: `doc/plugins/PLUGIN_SPEC.md` §11A and the plugin SDK README
describe the field, its range, and the default. The SDK manifest type
gets the field from `@paperclipai/shared`.

### Why 30 s

30 s is the same as the plugin worker manager's default RPC timeout. So,
before this change, the worker already allowed 30 s, and only the
gateway's 10 s limit stopped the call. With 30 s, the common 12–20 s
backend case works without a declaration. It is also well under the 60 s
ceiling. Tools that need more time can declare up to 60 s.

## Verification

- New `server/src/__tests__/plugin-connector-tool-timeout.test.ts` runs
the real gateway invocation path against an embedded Postgres and a fake
worker. Fake timers make the 12 s backend free in wall-clock time:
- With method `timeoutMs: 30000`, a 12 s tool succeeds through
`executeTool` and through `executeTestCall`. The worker RPC timeout is
31000.
- With tool `timeoutMs: 5000`, a 12 s tool fails at 5 s with `504
tool_timeout`.
- With no declaration, a 12 s tool succeeds (it failed at the old 10 s),
and a 45 s tool times out at 30 s.
- Manifest values above 60000, below 1000, or not integers are rejected.
A caller `timeoutMs: 120000` is clamped to 60 s.
- A legacy plugin tool (`paperclip_plugin`, a different provider path)
never reaches the connector runtime, and a caller deadline still
applies. This test does not pin the legacy default, because #20746
changes it.
- Remote MCP, local stdio and REST timeouts do not change in this PR.
Their existing timeout tests in `tool-gateway.test.ts` and
`google-workspace-rest-transport.test.ts` still apply.
- Red/green: with the source change reverted, 4 of these 5 tests fail.
The legacy-tool test passes both ways, as expected.
- `packages/shared/src/plugin-apps.test.ts`: validation cases for the
range, non-integer values, and non-plugin methods.
- Commands run:
- `pnpm --filter @paperclipai/shared typecheck && pnpm --filter
@paperclipai/server typecheck && pnpm --filter @paperclipai/plugin-sdk
typecheck`
- `cd server && npx vitest run
src/__tests__/plugin-connector-tool-timeout.test.ts
src/__tests__/plugin-connectors-plugin-transport.test.ts
src/__tests__/plugin-connector-agent-grant.test.ts` (20 passed)
- `cd packages/shared && npx vitest run src/plugin-apps.test.ts
src/app-definitions.test.ts` (76 passed)
  - `pnpm check:module-boundaries`
- Full suite not run locally. CI supplies it.

## Risks

- Low risk. Plugin-served tools now wait up to 30 s instead of 10 s by
default. An agent turn can wait longer for a slow or stuck plugin tool,
but never longer than the existing 60 s ceiling.
- The connection row lookup now runs before the timeout timer starts.
This is one indexed query.
- Dynamic tool lists (`kind: "dynamic"`) use the method's `timeoutMs`.
Per-tool values apply to static tools only.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), through Claude Code with tool use
(shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:24:03 -06:00
Devin FoleyandPaperclip d4b7ae5532 fix(tasks): require merged PR deliverables before native completion (#22244)
Keep native tasks with declared pull request deliverables active until
GitHub verifies every deliverable is merged. Preserve delivery guidance
in follow-up contracts and recheck the current outputs under task locks.

Use bounded incomplete-work feedback during reassessment, including for
saved non-Done reports. Serialize work-product restoration and identity
changes with completion. Preserve configured reviews and conversations.

Add regression coverage and repair the CI fixtures exposed during
qualification. All CI checks pass on afee630120. Greptile is 5/5 with no
unresolved review threads.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-18 16:20:17 -08:00
DottaandClaude Opus 5.5 11d777b80f feat(server): fit PostHog proxy limits to busy Cloud stacks and make them configurable (#22243)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Paperclip Cloud, the board UI sends browser analytics to PostHog
through a same-origin proxy at `/_ph`
> - The proxy limits requests in flight: 16 per client address and 256
in total
> - Each open tab runs posthog-js, and one tab can have a few requests
in flight at once
> - On Railway without `TRUST_PROXY`, `req.ip` is the private edge hop,
so all visitors share one 16-request limit
> - This pull request raises the limits, makes them configurable, and
stops the per-client limit when `req.ip` is a shared private hop
> - The benefit is that busy stacks with many users and many tabs do not
lose analytics to 429 answers, and ops can see when a limit is hit

## Linked Issues or Issue Description

**Subsystem affected**
Server: the PostHog reverse proxy for Cloud browser analytics
(`server/src/routes/posthog-proxy.ts`).

**Problem or motivation**
The proxy limits are too low for busy Cloud stacks. One user with many
tabs can use 16 requests in flight. Behind the Railway edge without
`TRUST_PROXY`, `req.ip` is a private `100.64.x.x` hop, so all visitors
share one 16-request limit. The proxy then answers 429 and posthog-js
drops or delays events and replay uploads. Ops cannot change the limits
and cannot see when the proxy refuses requests.

**Proposed solution**
Raise the defaults to 128 per public client address and 1024 in total.
Read both limits from two environment variables with a strict parse and
a ceiling. When `req.ip` is not a public address, apply only the total
limit. Give each request a 60 second deadline. Log a warning at most
once a minute when the proxy refuses requests at a limit.

**Alternatives considered**
Find the visitor for the per-client limit from `X-Forwarded-For`. A
client can write that header, and an earlier security review asked the
proxy to keep `req.ip` as the key, so this change does not do that.
Scale the per-client limit for a shared hop. All visitors share the hop,
so a scaled limit is only a second total limit. The proxy applies the
total limit alone instead.

**Additional context**
Fleet operators can set the variables with an `env_config` campaign.
`doc/cloud-browser-analytics.md` tells how. Refs #22012 (the proxy) and
#22052 (visitor address behind a private hop).

## What Changed

- Raised `DEFAULT_POSTHOG_PROXY_LIMITS` to 128 requests in flight per
client address and 1024 in total. The 64 MiB body limit did not change.
- Added `PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT` and
`PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT_PER_CLIENT`.
`resolvePostHogProxyLimits` reads them where `resolvePostHogProxyMount`
builds the proxy. A value must be a whole number from 1 to 65536. An
invalid value logs a warning and uses the default. The server does not
stop.
- Added `posthogProxyClientKey`. It returns `req.ip` only when it is a
public address. For a private, carrier-grade NAT, loopback, link-local
or other reserved address, or no address, the proxy applies only the
total limit. The key never comes from a client header.
- Added `createPostHogProxyLimitReporter`. When the proxy answers 429,
it logs `PostHog proxy refused requests at its in-flight limit` at most
once a minute. The fields are the refusal counts per limit since the
last warning, the requests in flight, the limits and the time since the
last warning. The warning has no addresses. It uses no timer: it checks
the time when a refusal occurs.
- Added a request deadline of 60 seconds from the time the proxy accepts
a request. At the deadline, the proxy stops the request to PostHog and
closes the browser response, also when PostHog already finished and the
browser does not read the response. The slot is then free again. Before,
the 30 second upstream timeout was an idle timeout, so a slow body that
sent a byte at a time could hold a slot until the server request timeout
(300 seconds).
- The mount object now also returns the limits in use.
- Updated `doc/cloud-browser-analytics.md`: the limits, why these
defaults, the variables, the shared-hop behavior, the warning fields and
how to tune a fleet with an `env_config` campaign.

Why these defaults:

- 128 per address covers about 10 people behind one office network, each
with 4 tabs and 3 requests in flight per tab, with space left over.
- 1024 in total keeps memory bounded. The proxy streams bodies, so a
request holds two sockets and the stream buffers between them, not the
full body. In a local test on this branch, 1024 held requests added
about 96 MiB of resident memory with 4 KiB bodies and about 173 MiB with
256 KiB bodies (256 held requests with 4 KiB bodies: about 40 MiB).
These numbers include the test client and the test upstream in the same
process, so the proxy share is smaller. Operators can lower the limit
for small stacks.

## Verification

- `cd server && npx vitest run
src/__tests__/cloud-posthog-browser.test.ts`: 87 tests pass (5 runs in a
row).
- New tests:
- Defaults, values from the environment, an empty value, and invalid
values (`0`, `-5`, `1.5`, `1e3`, `12abc`, `abc`, `0x10`, `+8`, `65537`,
`1024000`, a value larger than a safe integer). `65536` is accepted.
Each invalid value keeps the default and warns once per variable.
  - The mount passes the configured limits to the proxy.
- Per-client limit for a public address (with `trust proxy` set): a
second request from the same address gets 429, a request from another
address gets 200.
- Shared private hop (no trusted proxy, peer is loopback): the
per-client limit does not apply, and the total limit still answers 429.
- A forged `X-Forwarded-For` does not select the bucket when no proxy is
trusted: three held requests with the same forged public address and a
per-client limit of 1 all get 200.
- Request deadline: an upstream that does not answer gives 502 at the
deadline and frees the slot, a body that trickles in is cut off at the
deadline, and a 16 MiB response that the browser does not read is closed
at the deadline.
- The tests wait until the held upstream receives each request, not for
a fixed time.
- Total limit across public addresses, and the reported reason (`total`
or `per_client`) for each 429.
- The warning rate limit: one warning for the first refusal, none in the
next 60 seconds, then one warning with the counts of the refusals in
between, and no addresses in any warning.
- Red/green: with the shared-hop rule removed (`posthogProxyClientKey`
returns `req.ip`), 9 of the new tests fail. With the deadline removed,
the 2 deadline tests fail. With both, all pass.
- `cd server && npx tsc --noEmit -p tsconfig.json`: passes.
- `pnpm verify`: exit 2, because `doc/cloud-browser-analytics.md` is a
global CI input. Required CI supplies the broad coverage.
- Memory numbers come from a local script that held 1024 requests
through the proxy against a local upstream (Node 26, macOS).

## Risks

- Higher limits let more requests use memory and sockets at once. The
worst case is bounded by the total limit, and operators can lower it
with `PAPERCLIP_POSTHOG_PROXY_MAX_IN_FLIGHT`.
- Behind a shared private hop, one client can use all of the total
limit, and other visitors then get 429. The request deadline frees each
slot after at most 60 seconds, but a client that sends again can keep
the limit full. Before this change, the same visitors shared a limit of
16, which one client could also fill. A separate limit for the hop would
not help the visitors behind it, because one client that fills it blocks
all of them. Setting `TRUST_PROXY` brings back the per-client limit.
- A PostHog request that takes longer than 60 seconds now fails with
502. posthog-js bodies are small and PostHog answers in well under a
second, so this does not affect normal traffic.
- If an operator sets `TRUST_PROXY` to trust every hop, `req.ip` comes
from the client's `X-Forwarded-For`. This was already true before this
change. The total limit still applies.
- The proxy is mounted only on Cloud with browser analytics on.
Self-hosted installs do not change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 18:05:52 -06:00
4434714824 Fix sidebar Recents task order and archive visibility (#22251)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The sidebar gives people quick access to tasks, chats, and projects
they recently opened.
> - Expanding a parent or project in Recents can show tasks the person
never opened.
> - Archiving a task from the inbox leaves it in Recents.
> - This change gives each recent task its own row and removes tasks
archived for the current viewer.
> - People can reopen an individual task and keep saved shortcuts above
More.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Recents in the Standard and Classic sidebars.

**Subsystem affected**

Board navigation and task summaries.

**Current behavior**

Opening a parent or project can show other tasks beneath it. A recent
subtask moves its parent group when reopened. Archiving a task leaves
its recent link visible.

**Proposed behavior**

Show each visited task in its own row, in recent activity order. Keep
canceled and completed tasks eligible. Remove tasks archived for the
current viewer from Recents. Preserve saved pins.

**Reason and benefit**

People can find the task they opened by its own position in the list.
Archiving removes its recent link without removing a saved shortcut.

**Breaking changes**

The Recents rows no longer expand into task lists. Project links still
open the project. The navigation summary adds an optional personal
archive flag.

Related work: #22219, #22024, #21904, and #20450.

## What Changed

- Render task, agent chat, and project rows directly in Standard
Recents. Keep the existing controls to pin each kind of row.
- Render standalone task rows with the existing Classic action menus.
Remove sidebar child loading and refresh code.
- Derive the viewer’s archive state using the existing inbox rule.
Filter archived tasks out of Recents while retaining authorized pinned
shortcuts.
- Cover independent parent and child order, canceled and archived tasks,
pin retention, and the Classic Archive action.

## Verification

- Reproduced the empty Recents row after cancellation before the fix.
- On the rebased branch, 424 focused UI and live-update tests and the UI
typecheck passed on macOS.
- Ran 77 issue-summary and OpenAPI integration tests with embedded
PostgreSQL on macOS. The Runner source SDK registry check passed on the
pushed revision.
- The updated sidebar test renders 402 recent rows. A deliberate
repeated task lookup made the input-read guard fail. The restored
implementation passed.
- The paused-assignee browser test checks the task detail heading in
each interface. Both variants reproduced the CI locator failure locally
and passed with the scoped locator.
- All 17 focused Storybook browser checks passed in Chromium on macOS.
- Run `pnpm verify` for local selection. The product documentation
change requests broader GitHub CI coverage.
- Open a parent task and a child task in either sidebar. Reopen the
child and check that its row moves ahead of the parent. Cancel the child
and check that it stays visible. Archive it and check that its recent
row disappears.

- On commit `8051038099`, [GitHub
CI](https://github.com/paperclipai/paperclip/actions/runs/44421716811)
passed, including the required verification and end-to-end checks.
-
[Greptile](https://github.com/paperclipai/paperclip/pull/22251#issuecomment-6994494812)
rated the same commit 5/5. Its review thread is resolved.

## Risks

- Recents uses the same viewer-specific archive rule as the inbox.
Archiving prunes the local recent entry. A restored task can enter
Recents again when opened.
- Task summaries still return authorized tasks for saved pins. Child and
unread counts remain in the existing API response for older clients.
- The change requires no database migration.

## Model Used

- OpenAI GPT-6 via Codex with shell, GitHub CLI, Vitest, and Chromium.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <1504818+scotttong@users.noreply.github.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 16:04:42 -08:00
Devin FoleyandPaperclip d702f567bc fix(recovery): keep a task's lock while its run's executor still settles it (#22248)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When an agent's provider account recently reported a usage limit,
Paperclip holds new work instead of starting a run that would fail the
same way. The provider limit wait cancels the run, then schedules a
retry at the expected reset, and the retry takes over the task's
execution lock
> - The cancellation and the retry are two separate writes. Cancelling
the run wakes task recovery for that task, and its stale-lock sweep
treats a cancelled run's lock as stale and clears it
> - When the sweep wins that race, the retry scheduler finds the lock
gone and refuses to schedule ("Subscription retry suppressed because the
task execution lock changed"). The wait then releases the task, and
stranded-work recovery escalates it to a person with a Retry button
> - The stranded-work reconciler already waits while a terminal run's
own executor is still settling it (`terminalRunSettlingUntil`). The
stale-lock sweep did not
> - This pull request makes the stale-lock sweep wait for the same
executor before it clears an open task's lock, and look again at the
executor's deadline
> - The benefit is that a task waiting for a provider usage limit
resumes by itself at the reset, instead of stopping on "Couldn't finish
this attempt" until someone presses Retry

## Linked Issues or Issue Description

Refs #19246, which added the provider usage limit wait.

**What happened?**
An agent's provider account hit a usage limit. A task's next wake was
held for the reset: its run was cancelled with `provider_limit_wait`.
About 200 ms later the stale-lock sweep cleared the task's execution
lock, because the run was now cancelled. The wait then tried to schedule
its retry, found the lock gone, and released the task. The continuation
that followed hit the same race. After two lost retries, stranded-work
recovery blocked the task and asked for Retry.

On one instance, 80 of about 600 provider-limit waits in two days lost
their retry this way, across 43 tasks. 15 of those tasks were blocked
behind a Retry button.

**Steps to reproduce**
1. Give an agent a recent provider rate-limit failure with no reported
reset.
2. Wake the agent on an assigned task.
3. Let task recovery sweep the task's locks between the cancellation and
the retry (the new integration test does this from the cancellation
write).

**Expected behavior**
The task keeps its lock for a retry scheduled at the reset.

**Actual behavior**
The sweep clears the lock, no retry is scheduled, and the task waits for
a manual Retry.

## What Changed

- `recovery/service.ts`: `sweepStaleIssueLocks` keeps an open task's
lock while a run that holds it is still being settled by its executor,
using the existing `terminalRunSettlingUntil`. That is a live execution
in this process, or an unexpired controller lease in another one. The
sweep reports the deadline through `onDeferredDeadline`, so it looks
again once the executor is done. A closed task's lock is still cleared
at once.
- Tests:
- The stale-lock sweep keeps an open task's lock while the executor is
live, and clears it once the executor is gone. A done task's lock is
cleared either way.
- A provider-limit wait whose cancellation is swept straight away still
schedules its retry, and the retry holds the task's lock.

## Verification

- `npx vitest run` on the new cases: pass. Both fail without the source
change; the integration test then reproduces the incident (the sweep
clears the lock and no retry is scheduled).
- `npx vitest run` on 58 related server test files (stale-lock sweep,
stranded and assigned-work recovery, provider limit wait, native resume
locks, retry admission, recovery hints): all pass, except 6 cases in
`runtime-selection.test.ts` that read a skill file from a fixed local
path. They fail the same way on master on this machine.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low. The sweep waits only while the run's executor is provably still
running: a live execution in this process, or another process's
unexpired controller lease. A crashed executor has neither, so the sweep
clears the lock as before.
- An executor that keeps running after its run is terminal now delays
the sweep's backstop clear of an open task until it finishes. The
executor releases or hands off the lock itself when it finishes, and the
stranded-work reconciler already waits for the same executor.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 15:39:14 -08:00
DottaandClaude Opus 5.5 28bdb4d778 feat(cloud): migrate durable data online and name volume-free config blockers (#22240)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Hosted Paperclip servers can move from a server with a volume to a
volume-free server, which allows deploys without downtime
> - A server with history can move online only after its durable data is
off the local disk
> - Today only an offline maintenance process moves that data, so each
existing server takes an outage of many minutes
> - Most data owners already work safely while the server keeps serving,
but nothing drives them online
> - This pull request adds a signed, bounded pass that drives those
owners online and reports progress, and it names configuration blockers
> - The benefit is that existing servers can move to volume-free with no
maintenance window

## Linked Issues or Issue Description

**What existing behavior does this improve?**
An existing volume-backed server can move to volume-free only through
offline maintenance, because only the offline process migrates its local
data.

**Subsystem affected**
Server: ephemeral enrollment, durable data owners (agent files, skill
packages and installations, runtime skill assets, native harness backups
and controllers), Cloud control endpoints.

**Current behavior**
- Online preparation refuses while local data remains, and no online
path moves that data.
- When the volume-free configuration is incomplete, preparation fails
with a server error that does not say why.

**Proposed behavior**
- `POST /api/instance/ephemeral-enrollment/migrate` (Cloud control
action `ephemeral-enrollment:migrate`) runs one pass with a 60-second
budget. The pass drives the online-safe owners, defers items that are
busy, and returns counts, deferral reasons and fresh eligibility.
- Preparation and the eligibility read report an incomplete
configuration as a named blocker: `volume_free_storage_flags_required`,
`volume_free_master_key_required`, `volume_free_decision_key_required`,
`volume_free_database_listener_required` or
`volume_free_configuration_required`. Preparation returns 409 with these
blockers.

**Reason and benefit**
Cloud can call the pass until a server is eligible, and then move it
online with no outage.

**Breaking changes**
None. The offline maintenance behavior is unchanged.

## What Changed

- New `services/online-durable-data-migration.ts`. The pass:
- requires S3 storage and `PAPERCLIP_DURABLE_AGENT_FILES=true`, so the
server does not keep creating local data;
- runs these owners in order, with live writers: `migrateAgentFiles`
(`writersStopped: false`), the local skill object copy, skill snapshots
and versions, skill installations, runtime skill assets, and the native
harness and controller prestage;
  - stops between pages when the budget is spent;
  - runs one pass at a time in each process.
- `skill-package-storage-migration.ts`: a new `online` mode. In this
mode a row that another owner holds is skipped and counted as `busy`.
The row claim and the publication already use compare-and-swap. The
offline mode is unchanged and still fails closed on a busy row.
- `services/ephemeral-enrollment.ts`: a configuration check maps the
volume-free contract errors to blocker codes. Preparation and the
eligibility read both use this check.
- `routes/ephemeral-enrollment.ts`, `middleware/cloud-control.ts`,
`services/cloud-runtime-identity.ts`: the new route and action. The
route accepts only an empty body.

## Verification

- `npx vitest run src/routes/ephemeral-enrollment.test.ts
src/services/ephemeral-enrollment.test.ts
src/services/online-durable-data-migration.test.ts
src/services/skill-package-storage-migration.test.ts
src/__tests__/openapi-routes.test.ts
src/services/cloud-runtime-identity.test.ts`: 102/102 pass.
- New tests:
- the pass drives every owner with live writers, aggregates counts and
deferrals, and reports eligibility;
  - the pass does nothing without S3 or durable agent files;
  - the pass stops at the budget;
  - one pass runs at a time;
- the skill object copy refuses outside maintenance unless the mode is
online, skips a busy row online, and still fails closed offline;
- each missing setting becomes a named blocker in preparation and in the
eligibility read;
- the migrate route accepts only a signed migrate assertion and an empty
body.
- `tsc --noEmit` reports no errors in the touched files. A full local
`pnpm verify` needs the Runner build, so CI covers it.

## Risks

- The pass runs real migrations while the server serves. Each owner
keeps its own locks and compare-and-swap rules. Busy items are deferred,
not forced. The budget limits each call.
- The skill object copy can now run online. It copies immutable objects,
verifies the hashes, and publishes only if the row did not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:36:30 -06:00
DottaandClaude Opus 5.5 45c3f4caa2 fix(connectors): require an install before an agent reaches a plugin connector (#22237)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can add connectors that serve their own tools (`transport:
"plugin"`, #22004, #22019, #22201)
> - The activation rule is that an agent gets a connector's tools only
through an install of an active connection for that agent
> - An agent with no install could still list and call a plugin
connector's tools, and the plugin worker ran them with that agent's run
context
> - This pull request makes the tool access policy require the install
for plugin-served connections
> - The benefit is that a plugin connector reaches only the agents it is
installed for

## Linked Issues or Issue Description

Refs #22004
Refs #22019
Refs #22201

**Subsystem affected**
Tool access policy and tool gateway (plugin-served connectors).

**Problem or motivation**
A plugin connection was installed for agent A only. Agent B, a standard
agent with no install, called one of its tools. The plugin worker got a
run context with B's agent id.

**Root cause**
An install gives access only through the connection's app profile
binding. The policy does not check installs. For an agent outside that
profile, the policy goes on to its other allow routes. The common route
is the unscoped `tools:use` grant that every new standard agent receives
(`NEW_STANDARD_AGENT_DEFAULT_GRANT_KEYS`). `governedByConnectionProfile`
is false for B, because B is not bound to the connection's profile. So
`explicitGrant` allowed the tool for discovery and invocation.
Organization-grant connections ran the worker. Per-agent connections
listed the tools and reached grant resolution.

**Expected behavior**
An agent without an install for the connection (or a company install)
sees no tools and cannot call them. An installed agent is not affected.

## What Changed

- `server/src/services/tool-access-policy.ts`: for an agent caller on a
connection whose stored transport is `plugin`, deny with the new reason
`deny_connection_not_installed` unless an install exists for that agent
or the company. The check runs before profiles, grants, trust rules and
allow policies, so none of them can replace the install. It reads the
transport from the stored connection, not from the caller. The install
read uses the decision cache, so a listing batch reads it once per
connection.
- `packages/shared/src/types/tool-access.ts`: add
`deny_connection_not_installed` to `ToolAccessReasonCode`.
- `doc/connections/CONNECTOR-PLAYBOOK.md`: document the rule.
- Legacy plugin tools (`paperclip_plugin`) keep their own
implicit-install gate and are not changed.

## Verification

- New `server/src/__tests__/plugin-connector-install-gate.test.ts`
(embedded Postgres, real routes). Both agents get the default
standard-agent grants, including the unscoped `tools:use`. For an
organization-grant connection and a per-agent connection installed for A
only:
- A lists the tool through its gateway session and its Paperclip Runner
assigned-tool gateway, and a call reaches the worker.
- B (with its own install of an unrelated connection) lists none of them
in either path.
- B's direct call by name through its gateway session, its runner
gateway, and the runner's `paperclip_call_assigned_tool` is refused. The
worker is never called.
  - Before the fix, both cases failed: B listed the tool.
- Regression suites pass: `tool-access-policy-service`,
`tool-access-policy-index`, `tool-gateway`, `tool-gateway-service`,
`tool-gateway-*`, `prompt-injection-gateway`,
`plugin-connectors-plugin-transport` (legacy plugin tools),
`plugin-connectors-circleback-parity`, `plugin-connector-agent-grant`,
`heartbeat-runtime-mcp-servers`, `run-preparation`,
`tool-access-service`, `generic-mcp-connection`,
`remote-mcp-connectors`, `railway-connection`, `aws-mcp-connector`,
`mcp-protocol-versions`, `google-workspace-*`.
- `tsc --noEmit` passes for `server`, `packages/shared` and `ui` source.

## Risks

- Behavior change: an uninstalled agent loses access to plugin-served
connection tools it reached before through a general grant, a trust rule
or an allow policy. That access broke the activation rule. Installed
agents and company installs are not affected.
- The board's per-agent access check now shows
`deny_connection_not_installed` for agents without an install.
- Built-in (non-plugin) connections are not changed by this pull
request. They have the same policy path. Whether to require installs for
them is a separate decision, because existing setups may depend on it.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 17:34:09 -06:00
DottaandClaude Opus 5.5 d1ca0cc218 fix(native): stop retrying a run that filled its sandbox disk (#22239)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native agent runs execute in a managed sandbox with a fixed disk
size
> - An agent can fill that disk. The runner then exits with "No space
left on device". The disk reserve restarts the runner, but the turn in
progress can still be lost, and the run fails
> - The run failure did not carry the full-disk fact. The run was
classified as a generic provider or runner failure. Automatic recovery
then started the same work on a fresh sandbox of the same size, which
filled the disk again. After three attempts the task stopped in
"Recovery needed" with no useful explanation
> - This pull request names the cause `sandbox_disk_full` when the
runner reported a full disk during the run, and stops automatic retries
for that cause
> - The benefit is that Paperclip no longer spends sandboxes and time on
retries that cannot succeed, and the task tells the user what to do in
plain language

## Linked Issues or Issue Description

**What happened?**
A native run in a managed sandbox filled the 10 GiB disk. The runner
exited with "No space left on device (os error 28)". The runner was
restored, but the in-flight turn was lost and the run failed as
`native_provider_terminal_failed`. Automatic recovery repeated the same
work on a fresh sandbox of the same size three times. Each attempt
filled the disk again. The task ended in "Recovery needed" with an
unrelated cause.

**Expected behavior**
When a run ends because its sandbox disk is full, the run failure names
that cause. Automatic recovery does not repeat the same work on a
sandbox of the same size. The task shows a plain-language explanation
and a working Retry.

**Steps to reproduce**
1. Run a native agent task in a managed sandbox whose work installs more
than the free disk space.
2. Wait until the runner exits with "No space left on device".
3. See the run fail as a generic provider failure, and see automatic
retries start fresh sandboxes that fail the same way.

**Relevant logs or output**
`paperclip-runner` exited with "No space left on device (os error 28)".
The run then failed with `provider_turn_lost_on_restore`, classified as
`native_provider_terminal_failed`.

Refs #22068 (disk reserve that restarts the runner). Refs #22235
(continues a turn that a runner restore lost, once, in the same
sandbox). This PR covers the case where the run still ends.

## What Changed

- Server: each run's diagnostic lease now records when a runner
diagnostic reports a full disk (`ENOSPC`, `EDQUOT`, "No space left on
device", "Disk quota exceeded"). The runner's exit detail already
reaches the server as a diagnostic, so no runner change is needed. The
fact belongs to that run's diagnostic epoch only. A later run on the
same warm session, or a diagnostic after the epoch retires, does not see
it.
- Server: native failure classification returns `sandbox_disk_full` when
either (a) the run's lease saw a full-disk report, or (b) the error
chain carries the runner's existing `sandbox_disk_full` launch code (a
relaunch that cannot stage on the full disk). A fixed list of failures
keeps its own meaning: permission, approval, usage limit, model,
runtime, integrity, cleanup quarantine, ownership, saved-state and
admission failures.
- Server: `sandbox_disk_full` is a permanent native failure. The run
does not take another same-run attempt. Native safe replacement does not
pick it up, because it is not one of its failure codes. The task becomes
blocked with a board-owned recovery action and a plain-language next
action.
- Server: the bounded retry scheduler (`scheduleBoundedRetryForRun`)
refuses `sandbox_disk_full`. It does not schedule a `transient_failure`
retry on a fresh sandbox of the same size.
- UI: the task chat shows "Disk full" and a plain explanation instead of
the raw failure code.
- Retry stays available. `sandbox_disk_full` is not a reconciliation
cause, so the board's Retry starts a new run when the user chooses.

## Verification

- Red before the fix (captured on the unfixed code):
- `heartbeat-retry-scheduling.test.ts` "does not repeat a run that
filled its sandbox disk on a fresh sandbox of the same size": expected
`not_scheduled`, received `scheduled`.
- `native-sandbox-disk-full.test.ts` "names the full disk and stops
instead of retrying the same work": expected `sandbox_disk_full`,
received `native_provider_terminal_failed`.
- `native-sandbox-disk-full.test.ts` "names the full disk when the
runner could not be relaunched on it": expected `sandbox_disk_full`,
received `native_session_interrupted`.
- Green after the fix:
- `cd server && npx vitest run
src/services/native-runtime/native-sandbox-disk-full.test.ts
src/__tests__/heartbeat-retry-scheduling.test.ts
src/services/native-runtime/remote-runner-disk-reserve.test.ts` (90
passed).
- `cd ui && npx vitest run src/components/TaskChatThread.test.tsx` (426
passed).
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-session-diagnostics.test.ts`.
- Safety cases in the tests:
- A runner that exited for another reason still gets its bounded retry.
- A turn lost on restore without a full-disk report keeps
`native_provider_terminal_failed`.
- Integrity and approval failures keep their codes when a full disk was
reported.
- A full-disk report belongs only to its own run's diagnostic epoch; run
B on the same session does not see run A's report.
- After a full-disk report, other runner and provider loss codes
(session.open timeout, initialize timeout, protocol error, process
status failure) also become `sandbox_disk_full`.
- Worst-case automatic run starts per hour across a company: before,
each disk-full task started up to 3 automatic runs, each on a fresh
sandbox (3 × the number of affected tasks). After, a disk-full failure
starts 0 automatic runs. Only a person's Retry starts a new run. Real
transient failures keep their existing bounded lanes, so their worst
case is unchanged.
- Not run locally: the full server suite, the full typecheck and build.
Required CI covers them.
- Not verified live on staging yet. A live check needs a run that fills
its sandbox disk after this change deploys.

## Risks

- A run whose runner reported a full disk, and which later fails with a
runner or provider loss for an unrelated reason, is now labeled
`sandbox_disk_full` and is not retried automatically. The board's Retry
still works. This is the safer error: a full disk is the most likely
cause, and an automatic retry would repeat the same disk use.
- The full-disk report lives on the run's in-memory diagnostic lease. A
server restart between the report and the failure loses it, and the run
falls back to the previous classification. That is the previous
behavior, not a new failure.
- Low risk otherwise. No schema or API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 23:19:12 +00:00
eaac8ed0f7 feat(adapters): add Claude Fable 5.2 and GPT-6.2 with CLI floors, and refresh sandbox CLI pins (Nov 18 audit) (#22222)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents run on coding harnesses. Each adapter has a model catalog,
and sandboxes contain pinned CLIs.
> - Anthropic released Claude Fable 5.2 and OpenAI released GPT-6.2 Sol
and GPT-6.2 Astra for Codex. Both need newer CLIs than the Paperclip
Runner provider pack ships.
> - A stale catalog hides valid models. A model that needs a newer CLI
fails with an opaque backend error.
> - This pull request adds the verified model IDs with their CLI floors,
names the version gap in Paperclip Runner, and refreshes the sandbox CLI
pins.
> - The benefit is that operators can select current models and get a
named version error instead of a backend failure.

## Linked Issues or Issue Description

Refs #20774 and #21636 (both add parts of Fable 5.2 support; see
"Overlap" below).
Refs #18486 (adds GPT-6.1 Terra; this PR does not duplicate it).
Refs #16708 (adds Claude Haiku 5.5 to `claude_local`; this PR does not
duplicate it).

**What happened?**

Claude Fable 5.2 (`claude-fable-5-2`) and GPT-6.2 Sol / Astra
(`gpt-6.2-sol`, `gpt-6.2-astra`) are generally available. The
`claude_local`, `codex_local`, and OpenCode pickers do not list them.
Claude Code 2.1.321 is the first release with Fable 5.2. Codex 0.169.1
is the first release whose bundled catalog lists GPT-6.2. The Paperclip
Runner provider pack bundles Claude Code 2.1.286 and Codex 0.160.0.

**Expected behavior**

The pickers list the new models. A run on a CLI below the model minimum
stops with the detected and required versions, or falls back to a
compatible model with a visible message.

**Proposed change**

Add the models and their per-model CLI floors. Gate ACPX Claude in
Paperclip Runner on the Claude Code version that its Agent SDK bundles.
Refresh the sandbox CLI pins with verified digests. Record the audit.

## What Changed

- Claude Code (`claude_local`):
- Add `claude-fable-5-2` (label "Claude Fable 5.2") at the top of the
picker. `claude-fable-5-1` stays selectable after the current-generation
models.
- Add Bedrock `global.anthropic.claude-fable-5-2`. Anthropic documents
only this global ID for Bedrock.
  - Fable 5.2 offers all five efforts (`low` to `max`).
- `minimumClaudeCliVersionForModel("claude-fable-5-2")` returns
`2.1.321`. The existing CLI-lane Test and launch checks use this floor.
On a managed sandbox, the CLI lane installs a private Claude Code at
this version, as it does for the other floors.
- Paperclip Runner (ACPX Codex): `resolvePaperclipRunnerProviderProfile`
rejects a model whose Codex floor is above the qualified ACPX Codex
runtime (`0.160.0`). The error code is
`paperclip_runner_codex_cli_version_incompatible`, and the message gives
both versions.
- Paperclip Runner (ACPX Claude):
`resolvePaperclipRunnerProviderProfile` rejects a model whose Claude
Code floor is above the version that the qualified Agent SDK bundles.
The error code is `paperclip_runner_claude_cli_version_incompatible`.
The message gives both versions. A new helper,
`claudeCodeVersionForAgentSdk`, maps SDK `0.3.N` to Claude Code `2.1.N`
(the SDK's `claudeCodeVersion` field) and returns `null` for any other
line.
- Codex (`codex_local`):
- Add `gpt-6.2-sol` and `gpt-6.2-astra` at the top of the picker, in the
bundled catalog order.
  - Both offer Fast mode and the `low` to `ultra` efforts.
- Both have a CLI floor of `0.169.1`. On the native Codex provider with
the pinned Codex 0.160.0, the existing fallback moves `gpt-6.2-sol` to
`gpt-6.1-sol` and `gpt-6.2-astra` to `gpt-6-astra` with a visible
message.
  - The default stays `gpt-6.1-sol`.
- OpenCode fallback catalog: add `openai/gpt-6.2-sol`,
`openai/gpt-6.2-astra`, `anthropic/claude-fable-5-2`,
`google/gemini-4-carbon`, and `google/gemini-4-hydrogen`.
- Evaluation image sandbox layer (`docker/daytona-runner`):

  | CLI | Before | After |
  | --- | --- | --- |
  | Grok CLI | 1.0.70 | 1.0.89 |
  | Gemini CLI | 0.65.0 | 0.68.0 |
  | Cursor CLI | 2026.10.22-27dcf5f | 2026.11.17-f532c45 |
  | GitHub CLI | 2.103.3 | 2.105.0 |

- Add `doc/adapter-model-audit-2026-11-18.md` with sources,
compatibility findings, and deferred items. Add a note on above-window
Codex floors to `doc/architecture/paperclip-runner-compatibility.md`.

No saved agent changes model, effort, or spending settings. No picker
entry is removed.

### Managed sandboxes

On Paperclip Runner, Fable 5.2 and GPT-6.2 stay unusable until the
provider pack moves to Claude Code 2.1.321 or later and Codex 0.169.1 or
later. A sandbox image that carries that provider pack must also be
promoted. This PR does not change the provider pack (see "Deferred
items" in the audit). Native remote Codex accepts only `>=0.149.0
<0.161.0`, so GPT-6.2 also needs that window to move.

### Overlap

- #21636 sets the Fable 5.2 floor to `2.1.290`. The [Claude Code
changelog](https://github.com/anthropics/claude-code/blob/main/CHANGELOG.md)
adds Fable 5.2 in 2.1.321. `git merge-tree` merges the two without a
text conflict, but #21636's line becomes unreachable after this PR's
line, and its test then fails. Rule: keep `2.1.321`. Merge this PR
first. Then close #21636 or change its floor and test to `2.1.321`.
- #20774 (draft) lists `claude-fable-5-2`. It conflicts in
`packages/adapters/claude-local/src/index.ts` and
`server/src/__tests__/adapter-models.test.ts`. Rule: keep this PR's
catalog order and Bedrock ID.
- #18486 adds `gpt-6.1-terra`. It conflicts in
`packages/adapters/codex-local/src/index.test.ts` and
`src/server/codex-args.test.ts` (the ordered ID list and the Fast-mode
message). Rule: keep both sets of models in the bundled catalog order
(`gpt-6.2-sol`, `gpt-6.2-astra`, `gpt-6-astra`, `gpt-6.1-sol`,
`gpt-6.1-terra`, ...). Either PR can merge first; the second rebases.
- #16708 adds Claude Haiku 5.5. It conflicts in
`packages/adapters/claude-local/src/index.test.ts` (adjacent test cases)
and `doc/DEVELOPING.md`. Rule: keep both test blocks.

## Verification

Research date: November 18, 2026. Sources are listed in the audit doc.

- `vitest run src/index.test.ts src/server/model-order.test.ts
src/server/cli-capabilities.test.ts src/server/test.remote.test.ts` in
`packages/adapters/claude-local`: 40 passed. `execute.remote.test.ts`
(29), `test.probe.test.ts` (21), `sandbox-cli-version.test.ts` (15),
`acp.auth.test.ts` (25), and `acp.test.ts` (47, with
`--testTimeout=60000`) also passed when run one file at a time.
- `vitest run src/index.test.ts src/server/codex-args.test.ts
src/ui/build-config.test.ts src/server/test.remote.test.ts` in
`packages/adapters/codex-local`: all passed (155). `acp.test.ts` (49),
`acp.cli-version.test.ts` (4), and `execute.test.ts` (10) also passed
one file at a time.
- `vitest run` in `packages/adapters/opencode-local`: 94 passed.
- `vitest run src/services/native-runtime/acpx-qualification.test.ts
src/services/native-runtime/codex-model-fallback.test.ts
src/__tests__/adapter-models.test.ts
src/__tests__/heartbeat-model-fallback-warning.test.ts` in `server`: all
passed. The new cases cover the ACPX Claude and ACPX Codex version
errors and the native GPT-6.2 fallback on Codex 0.160.0.
- `vitest run src/lib/agent-setup-fields.test.ts
src/lib/codex-reasoning-effort.test.ts` in `ui`: 11 passed.
- `tsc --noEmit` in the three adapters and `pnpm typecheck` in `server`:
passed.
- Cursor archive SHA-256
`189ed41d8963cd22dcdafbbd62f072cc55ac08374c922dbd04aee525f4018ab6` is
computed from the versioned Linux x64 tarball. The extracted binary
reports `2026.11.17-f532c45`, the version that
`https://cursor.com/install` resolves.
- GitHub CLI SHA-256
`79e7ab5db738c48777b696e374f8b428f0647debfb424f44d2f426370407fe2f`
matches `gh_2.105.0_checksums.txt`.
- Grok CLI 1.0.89 and Gemini CLI 0.68.0 installed from npm report those
versions. The flags that Paperclip uses are in `--help` (`-m`,
`--effort`, `grok models`; `-m`, `-p`, `--output-format`,
`--approval-mode`, `--yolo`).
- Host baseline: tests that create an AI provider need the host's empty
`ANTHROPIC_BASE_URL` unset. Under full-package parallel runs, SSH and
sandbox fixture tests time out at 5 seconds on this host; each file
passes alone. The audit doc records both.
- Not run locally: the full `pnpm -r typecheck`, `pnpm build`, browser
suites, and the image build. CI runs these. No live model inference ran.

## Risks

- Low risk for saved agents: no default, saved model, effort, or
spending setting changes.
- The new Paperclip Runner checks reject `claude-fable-5-2` on ACPX
Claude and `gpt-6.2-*` on ACPX Codex (and any later model whose floor is
above the bundled runtime) before launch. Without this PR, the turn
would start and fail at the backend.
- The `0.3.N` → `2.1.N` mapping covers only the current SDK line. Any
other line returns `null`, and the check is skipped.
- The sandbox CLI pins change only the evaluation image. The fleet image
change is separate.
- Rollback: revert this PR. No data migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, web fetch, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 14:27:00 -08:00
DottaandClaude Opus 5.5 bb258a5429 fix(connections): explain failed app sign-ins and guide GitHub app install (#22234)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents can ask a person to connect an app, such as GitHub, through a
sign-in window
> - When that sign-in fails, the server knows the reason and records it
in the activity log
> - The sign-in window tells the opener only "failed", so the person
sees "Authorization did not complete" for every cause
> - People cannot fix the problem themselves, and support cannot help
without reading server data
> - This pull request sends the safe failure code to the opener and
shows plain copy for the common causes
> - The benefit is that a person knows what to do next, such as to
finish faster or to stay in the same session

## Linked Issues or Issue Description

**What happened?**
A user tried to connect GitHub from an agent request. The dialog showed
"GitHub couldn't connect. Authorization did not complete. Try again when
you're ready." The same text shows for every failure cause.

**Expected behavior**
The dialog should tell the user why sign-in failed and what to do next.

**Steps to reproduce**
1. Start an app connection from an agent request.
2. Let the sign-in take longer than the state lifetime, or finish it in
a different Paperclip session.
3. See the generic "Authorization did not complete" message.

**Paperclip version or commit**
`master` at 84a529df4f.

**Deployment mode**
Paperclip Cloud. The connection-intent sign-in path is the same in
self-hosted deployments.

## What Changed

- The connection-intent sign-in page now posts a `code` field with a
`failed` outcome. Only codes that match `^[a-z][a-z0-9_]{0,63}$` are
sent. Connected and declined outcomes send no code.
- Both OAuth callback catch blocks pass their failure code to that page.
- OAuth state errors now carry codes: `oauth_state_expired`,
`oauth_state_missing` and `oauth_session_mismatch`.
- The setup flow maps common codes to plain copy. Unknown codes keep the
old generic text, and raw codes never show to the user.
- The managed-connector callback now records `tool_app.oauth_failed`
with the failure code. Before, it answered 200 and recorded nothing.
- A callback whose state is already gone now answers an HTML request
with a page that posts `oauth_state_missing`. The dialog accepts it only
from its own active sign-in window.
- GitHub: before sign-in, the setup explains that Paperclip connects
through the Paperclip for GitHub app and which account to install it on.
- GitHub: when sign-in finds no usable installation, the sign-in window
passes the GitHub install and manage links (github.com only) to the
dialog. The dialog shows "Install Paperclip on GitHub" and "Manage
repositories on GitHub" next to "Try again".

## Verification

- `server`: `npx vitest run
src/routes/tool-access-connection-intent.test.ts
src/__tests__/generic-mcp-connection.test.ts` (126 passed). A new test
checks that a safe code is posted, an unsafe code is dropped, and a
connected outcome has no code.
- `server`: `npx vitest run
src/__tests__/tool-access-oauth-protocol.integration.test.ts -t "missing
sign-in state"` covers the missing-state page on both callbacks.
- `ui`: `npx vitest run src/features/connections/` (130 passed). Tests
cover the GitHub install links, which drop non-github.com links, and
missing-state messages from an old window.
- `ui`: earlier run `npx vitest run
src/features/connections/ConnectionSetupFlow.oauth.test.ts` (21 passed).
New tests check the copy for expired and cross-session sign-in, and that
unknown codes show the generic text.

## Risks

- Low risk. The postMessage adds one optional field. Old openers ignore
it.
- The state errors now include `details.code`. The status codes and
messages do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:56:16 -06:00
DottaandClaude Opus 5.5 46e38e1387 fix(native): let the outgoing server checkpoint its idle warm session while its follow-up has not started (#21887)
**HOLD: do not merge until the coordinator confirms (merging resets the
zero-downtime gates).**

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A deploy swaps servers with an overlap. A warm agent session stays
with the outgoing server that ran its last turn.
> - When that turn finishes during the overlap, the outgoing server
saves its idle session and closes it. A checkpoint fence decides whether
the save is still allowed.
> - The task's follow-up run usually starts on the incoming server at
the same moment. It waits for the outgoing owner (#21774), and #21791
lets the outgoing server save while the follow-up is parked on it.
> - The fence still refused the save
(`native_runner_checkpoint_owner_lost`) in every other state of the same
unstarted follow-up. In release-swap waves the close then fails at least
once and retries, and the follow-up waits longer. If every attempt hits
such a window, the owner stays until the outgoing server stops, up to
the 510 s overlap. Speed is a requirement.
> - This pull request makes the fence accept the outgoing server's final
save in every state in which the follow-up has not started: no process,
no own process start, no own checkpoint, `attempt = 0`, same session. It
still refuses when the follow-up names another owner or has started.
> - The benefit is that the outgoing server closes its session on the
first attempt, and the follow-up continues at its next owner check.

## Linked Issues or Issue Description

Refs #21791, #21774, #21782

**What happened?**

Release-swap waves logged `idle warm native session close failed …
native_runner_checkpoint_owner_lost`. The close worked on a later
attempt. Two recent waves, traced with the wave's 2-second run samples:

| Wave | Outgoing run | Attempt | Follow-up at that moment | Old fence |
| --- | --- | --- | --- | --- |
| 13:48Z | `480c86e4` (finished 14:29:04.7) | 1, 14:29:07.7 |
`5396e5ab`, started 14:29:05.6 on the incoming server: `preparing`
(legacy, incoming boot) at 06.2, native `observed` with no owner at 08.2
| refused: a preparing follow-up was accepted only on the outgoing
server's own boot, and a native follow-up only when parked or re-claimed
with `waitingForOwner` |
| 15:58Z | `87de6f54` (finished 16:16:14.1) | 1, 16:16:19 | `2009f6bb`,
started 16:16:18.7: `preparing` (legacy, incoming boot) at 19.8 |
refused: foreign preparation |
| 15:58Z | `87de6f54` | 2, 16:16:27 | `2009f6bb`: native `observed`, no
owner from 21.8 to 25.8, owner stamped at 27.8 | refused: selected
follow-up before its first owner check, or a park or re-claim without
`waitingForOwner` |

In each case the follow-up was the task's current run and had no
process, no process-start event and no checkpoint rows. No successor
claim or stale epoch held the session: the outgoing server's own run had
committed and released its lease. The fence refused only because it did
not recognize these states of the same unstarted follow-up.

**Expected behavior**

The outgoing server saves and closes the session on the first attempt
while the follow-up is in any unstarted state. The follow-up then sees
the saved session at its owner check and continues without parking (or
within one recheck if it already parked).

**Steps to reproduce**

1. Run a warm native session on a remote sandbox, and let a run finish
on server A during a swap to server B.
2. Let B claim the follow-up right away.
3. While B prepares it, selects native execution, parks it, or re-claims
it, A saves and closes its idle session. The fence refuses the save.

**Paperclip version or commit**

master `f61b919701` (merged in). The waves ran on master builds without
this pull request.

## What Changed

- `native-runner-checkpoint.ts`: the parked-follow-up check is now
`unstartedSuccessorOfOwner`. It takes `FOR SHARE` on the follow-up's
finalization row (if any), then on its run row, in that order, inside a
savepoint. The claim and defer paths use the same order. If native
selection created the finalization row while the fence waited for the
run lock, the fence rolls the savepoint back, which releases the run
lock, and takes both locks again in the same order (at most twice). A
last statement then reads the current state again and accepts one of
these states:
- **Preparing on any server** (new): the run is still `legacy`,
`preparing`, on this task, has a live controller lease and no process,
and has no finalization row. Before, only the outgoing server's own boot
was accepted.
- **Selected, before its first owner check** (new): native, finalization
`observed`, `attempt = 0`, no lease, no failure, empty recovery history.
  - **Parked**: as before, but `waitingForOwner` may be absent.
- **Re-claimed**: as before (live `same_run_resume` claim by the current
controller), but `waitingForOwner` may be absent, as in claims from a
previous release.
- These conditions apply to every state and are unchanged: running; no
own process-start event; no own checkpoint rows. For the native states,
also: same company, task and native session; `attempt = 0`; no result;
no process, unless it is the identity inherited from the outgoing run
(`inheritedProcessIdentityRunId`). The fence still requires an accepted
result and no new work.
- A `waitingForOwner` that names another run or boot is still refused.
- `SAME_RUN_RESUME_DISPOSITION` moves to the leaf module
`same-run-resume-disposition.ts`. The fence no longer imports
`scheduled-resume-continuation.ts`, so the runner E2E server does not
load the executor and `@paperclipai/plugin-sdk` through it
(`server-startup-import-graph.test.ts`).
- The run-row share lock makes native selection (`runtime_selection`
locks the run `FOR UPDATE`) and process start wait for the save to
commit. A save that waited reads the committed state in its last
statement and is refused if the follow-up has started.
- Tests, real Postgres:
  - `native-controller-recovery.integration.test.ts`, new:
- The outgoing server saves and closes while the incoming server
prepares the follow-up.
    - Native selection of that follow-up waits for the save.
- A save that waited while native selection held the run lock and
created the coordinator is accepted (Greptile review).
- It saves and closes while the follow-up is selected but before its
owner check, also with an inherited process identity, and the
follow-up's check resolves at once.
- It saves and closes while the follow-up is parked or re-claimed
without `waitingForOwner`, including a claim entry in the previous
release's shape.
- Still refused, preparing: expired controller lease, `dispatching`, own
process, own process-start event, other task.
- Still refused, selected: own process, a process inherited from another
run, own process-start event, own checkpoint, other session, admitted
(`attempt = 1` with a lease).
    - The outgoing-server fixtures share one helper.
- `native-harness-backup-storage.integration.test.ts`: the harness
backup fence shares this check. A released owner whose follow-up is
still preparing on another server now replaces its selected backup.
`expired_preparation`, `dispatching_successor` and `foreign_native`
still keep the old backup.
- `native-runner-restart-recovery.integration.test.ts`: the
`foreign_preparation` early-retirement case now expects the save to
succeed. `expired_preparation`, `dispatching_successor`, `wrong_issue`,
`unclaimed_successor` (another session) and `foreign_native_preparation`
still expect a refusal.
- From the earlier commits of this pull request: the re-claimed state,
`sameRunResumeClaimEntry` recording `waitingForOwner`, the run-row lock
and the post-lock recheck.

## Verification

- Red/green for this round (only the fence source changed between runs):
- With the fence at `ed7227eeda` (this pull request before this round),
9 tests fail with `native_runner_checkpoint_owner_lost`: preparing close
(+ `foreign_preparation` ×2), native-selection ordering,
selected-before-check ×2, waiting-without-owner ×3. The 11 new refusal
cases pass both before and after.
- With the fence change:
`native-controller-recovery.integration.test.ts` and
`native-runner-restart-recovery.integration.test.ts` 306 passed, 14
skipped. `native-harness-backup-storage.integration.test.ts` 73 passed.
`scheduled-resume-continuation.test.ts`,
`native-runner-checkpoint-save.test.ts` and
`native-controller-recovery.integration.test.ts` 110 passed.
- Savepoint restart: the new waited-on-native-selection test fails with
`native_runner_checkpoint_owner_lost` without it, and passes with it.
- `tests/runner-e2e/server-startup-import-graph.test.ts` failed in CI at
`cc513d9434` (import path through `scheduled-resume-continuation.ts`).
It passes locally after the move.
- Earlier rounds (unchanged): red on master for the re-claimed close,
the process-start ordering and the claim ordering; red without the
run-row lock; red without the recheck.
- `tsc --noEmit` for `server` passes.
- `pnpm verify` (base `f61b919701`) selected 2842 of 3683 files, because
the checkpoint module has a wide import graph. The reusable job passed:
2 files, 12 tests. The fresh job was cut off locally (exit 144, no test
failure printed). Required CI supplies that coverage.
- `node scripts/agent-preflight.mjs` stops at `pnpm install
--frozen-lockfile`: on master, `server/package.json` lists
`posthog-node` (#22006) and the lockfile does not have it yet. CI
resolves the lockfile inline. Locally, dependencies were installed
without the frozen lockfile, and `pnpm-lock.yaml` was not committed.
- Not run: a release-swap wave. The coordinator runs the gates.

## Risks

- The fence is wider. It now accepts the outgoing server's final save
while the follow-up is in any state before it starts. The save needs an
accepted result and no new work. The follow-up must have no process
(except the identity inherited from this run), no process-start event,
no checkpoint of its own, and `attempt = 0`. A follow-up that names
another owner, holds an expired claim, has been admitted, or is in
another session is still refused.
- A follow-up that another server is preparing is now accepted. It has
not touched the session: it runs the owner check after native selection,
and native selection waits for the save to commit. The earlier
same-boot-only rule was stricter than needed.
- The fence takes share locks on the follow-up's rows whenever it
reaches this check, also when it then refuses. The save transaction is
short. The lock order is the finalization row, then the run row, as in
the claim and defer paths.
- Previous-release state: a claim or park written by a release without
`waitingForOwner` is now accepted, not refused. No stored shape changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:37:16 -06:00
DottaandClaude Opus 5.5 397f163c7f fix(issues): show tasks held by a resolved no-replay hold as Blocked (#22215)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - Agents can schedule a follow-up check (a monitor) on a task. The task then shows as "Monitoring" until the check is due.
> - A failed run can leave a recovery record that blocks replay of its work. The record can have the status `resolved` and still block, because its evidence keeps `automaticRecovery.replay = "blocked"`.
> - Wake admission treats that record as a hold. It refuses every wake for the task, including the due monitor check, and records the skip with `execution_reconciliation_required`.
> - The task display status counts only `active` and `escalated` recovery records. Thus the task shows "Monitoring" with an overdue check, not "Blocked". The board does not see that a person must act. Agents re-arm the check, and the check stays overdue for days.
> - This pull request makes the display status show these held tasks as "Blocked". It uses the part of the wake-admission hold predicate that is always effective and cheap to query.
> - The benefit is that a task whose wakes are held shows as "Blocked" in the list, in search, in counts and in summaries, and the person who must act can find it.

## Linked Issues or Issue Description

(B) No issue exists. Bug report:

**What happened?**
Scheduled follow-up checks on agent tasks stayed overdue for many days. Each due check was skipped with the reason `execution_reconciliation_required`. One skip record had more than 25,000 merged attempts. The tasks did not show as "Blocked". They showed as "Monitoring" (or as in progress) with an overdue check. Agents re-armed the check, but the check never started.

**What did you expect to happen?**
When wake admission holds a task, the task shows as "Blocked", so a person can see it and use the recovery controls.

**Steps to reproduce**
1. Give an agent an `in_progress` task with a monitor.
2. Add a recovery record for the task with status `resolved`, a reconciliation cause (for example `provider_failure_meaning_unverified`) and evidence `automaticRecovery.replay = "blocked"`.
3. Read the task. `getExecutionBlocker` returns the hold and wake admission refuses the monitor wake.
4. See that `displayStatus` is `monitoring`, not `blocked`.

**Paperclip version or commit**
`master` at 8282d386da.

Related open PRs: #14590 and #14156 change how no-replay holds are released. This PR does not change hold release. It only makes the held task visible.

## What Changed

- `server/src/services/issue-display-status.ts`: An agent task with lifecycle status `in_progress` now shows `blocked` when it has a recovery record with these properties:
  - The cause is a reconciliation cause other than `legacy_execution_requires_reconciliation`.
  - The evidence has `automaticRecovery.replay = "blocked"`.
  - The evidence does not have the legacy workspace-restore schema.
  - The task is not a conversation.
- For these records, `executionBlockerPredicate` is always true and `restoreHoldReleasedForCurrentOwnerPredicate` is always false. Thus "Blocked" never shows for a task that wake admission would wake. The exceptions of the full predicate apply only to legacy holds, legacy workspace-restore holds and conversation boundaries. Those keep their current display.
- The full admission predicate is not used in the projection. I tried it first, and list queries became 3 to 4 times slower, because every list, search and count query includes this projection.
- `server/src/__tests__/issue-display-status.test.ts`: New regression test. It also checks that `getExecutionBlocker` agrees with the display status before and after the hold allows replay.

## Verification

- Failing before the fix: `pnpm --dir server exec vitest run src/__tests__/issue-display-status.test.ts`. The new test "shows an overdue monitor behind a resolved no-replay execution hold as Blocked, not Monitoring" failed with `AssertionError: expected 'monitoring' to be 'blocked'`.
- Passing after the fix: the same command. 21 of 21 tests pass.
- Affected suites: `vitest run` on `issue-display-status`, `issue-summaries-routes`, `blocked-inbox-review-wait-scope`, `company-search-service`, `issues-service`, `issue-list-assignee-filter-routes`, `status-cards` and `heartbeat-issue-liveness-escalation`. 8 files, 361 tests pass.
- List cost: a local probe (40 `issueService.list` calls) measured about the same time per call as `master`.
- Typecheck: `pnpm --dir server typecheck` (all layers) passes.
- Not run locally: the full `pnpm verify` selection (2,923 files). CI runs it.

## Risks

- Behavior change: more agent tasks can show as "Blocked". This occurs only when wake admission already refuses their wakes. The task lifecycle status, locks and recovery records do not change. This change does not write data.
- Query cost: the display status adds one small `EXISTS` subquery on `issue_recovery_actions` by company and source task. It runs only for agent tasks that are `in_progress` with no queued or running run and no active recovery record.
- Coverage: tasks held by a resolved legacy reconciliation hold, or by a legacy workspace-restore hold, still show their current status. Their release rules need run-level checks that are too costly for this projection.
- `in_review` and other lifecycle statuses keep their current display.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell, file edit, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:27:05 -06:00
DottaandClaude Opus 5.5 35563b7de5 fix(railway): report restricted environments instead of a consent error (#22214)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - The Railway connection gives agents governed deployment tools, such as `list-deployments`
> - Railway Environment RBAC lets non-admin members see a restricted environment, but Railway denies reads of its services, deployments, and logs
> - The client maps this denial to `railway_api_authorization_required` and tells the user to reconnect or to use IDs from a consented workspace
> - That advice is wrong for this case: workspace consent is correct, and reconnecting does not change the result
> - This pull request identifies the restricted environment and reports the correct remedy
> - The benefit is that agents and operators get a clear, actionable error instead of an incorrect reconnect loop

## Linked Issues or Issue Description

(B) No public issue exists. Bug description:

**What happened?**
An agent called `paperclip-railway-list-deployments` for a service. In the same session, `list-projects`, `list-services`, and `list-environments` succeeded for the same project. The deployment-history call failed every time with `railway_api_authorization_required` and the message "Use IDs from a workspace selected during consent, or reconnect". The project-level reads prove that workspace consent is correct. The denied environment is restricted for the connected member, so the message sends the user to the wrong fix.

**Expected behavior**
When Railway denies the environment-scoped target check because the environment is restricted, the tool must report a distinct error. The error must say that a Railway project admin must grant access. It must not tell the user to reconnect.

**Steps to reproduce**
1. Connect Railway with a member (non-admin) account in a workspace that has a restricted environment.
2. Call `paperclip-railway-list-environments` for the project. The restricted environment is in the list.
3. Call `paperclip-railway-list-deployments` (or `service-status`) for a service in that environment.
4. Railway returns `Not Authorized`, and Paperclip reports a workspace consent error.

**Paperclip version or commit**
`master` at 8282d386da.

## What Changed

- `server/src/services/railway.ts`: When the target query fails with an authorization denial, the client reads the project's environments with `canAccess`. If the requested environment has `canAccess: false`, the client throws `railway_environment_restricted` (HTTP 403) with the admin-access remedy. In all other cases, the client throws the original error. If the access check fails, the client also throws the original error. The access check is a read query. No mutation runs after a denied target check.
- `server/src/services/railway.ts`: `list-environments` now returns `canAccess` for each environment, so agents can choose an accessible environment before they call a target tool.
- `server/src/__tests__/railway.test.ts`: Adds a regression test for the restricted, accessible, and unknown cases.
- `doc/connections/RAILWAY.md`: Adds a recovery entry for the new error code.

## Verification

- Failing before the fix: `cd server && npx vitest run src/__tests__/railway.test.ts`. The new test "reports a restricted environment instead of a workspace consent error when Railway denies the target" failed. It expected `code: "railway_environment_restricted"`, but it received `code: "railway_api_authorization_required"`. The other 28 tests passed.
- Passing after the fix: the same file passes (29/29).
- Related suites pass: `npx vitest run src/__tests__/railway.test.ts src/__tests__/railway-connection.test.ts src/__tests__/railway-ssh.test.ts src/__tests__/connector-tool-policy.test.ts` (4 files, 63 tests).
- Scoped `tsc --noEmit` with the server `tsconfig.json` for the changed files passes.
- Not run: the full server typecheck and `pnpm verify`. In the test sandbox, they used more than the 8 GB memory limit. CI must confirm them.
- Public reference for the provider behavior: https://docs.railway.com/enterprise/environment-rbac

## Risks

- Low risk. The change applies only after Railway has already denied the target check. It adds one read query in that case.
- If Railway removes or renames `canAccess`, the access check fails, and the client throws the original error, as before this change.
- The access check reads the first 100 environments of the project. If a project has more environments, the client throws the original error.
- `list-environments` output gets one more field.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with tool use (shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:26:56 -06:00
DottaandClaude Opus 5.5 01d74a52a2 fix(connections): name an expired sign-in in connection search (#22211)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - Agents use connection search to find out if they can use an app, and what a person must do when they cannot
> - When an OAuth provider rejects a refresh token, Paperclip disables the shared connection and records `oauth_reauthorization_required`
> - Connection search then gave only the generic reason "Connection needs attention"
> - Agents could not tell a person the exact action. They had to read large app attention payloads to find the cause
> - This pull request makes connection search name the connection whose sign-in expired and say that a person must reconnect it
> - The benefit is a clear blocker that agents can report in one step, so the connection owner can renew the sign-in sooner

## Linked Issues or Issue Description

**What happened?**
A shared OAuth connection (for example, Sentry) came to the end of its provider grant. Paperclip tried the refresh token. The provider rejected it. Paperclip correctly disabled the connection and set its error to `oauth_reauthorization_required`. After that, `connections_search` returned `state: "needs_user_action"` with the reason "Connection needs attention". Scheduled agent work that needed the app stopped. The agents could not name the cause or the human action from the search result.

**Expected behavior**
Connection search tells the agent that the sign-in for the named connection expired, and that a person who manages the connection must reconnect it.

**Steps to reproduce**
1. Make a shared OAuth tool connection and give an agent access to its tools.
2. Make the provider reject the refresh token (for example, `invalid_grant`). Paperclip sets the connection to draft, disabled, health `error`, and last error `oauth_reauthorization_required`.
3. As the agent, search connections for the app.
4. See the reason "Connection needs attention". It does not say that the sign-in expired.

## What Changed

- `server/src/services/connection-intents.ts`: connection search reads the grants of each matching connection. When a grant in the caller's audience (the organization, the responsible user, or the agent) has status `needs_reauthorization`, and no grant in that audience is active, the reason names that connection. The reason says that the connection must be signed in again, and that a person who manages it must reconnect it on its app page.
- The search uses the same audience rule that protects indexed tool descriptions. A grant of another person does not show the name of that connection.
- The state, `connectionId`, and all access checks do not change.
- `server/src/__tests__/connection-intents-service.test.ts`: add two regression tests. The first test confirms that the healthy connection is `ready`. Then it writes the rows that a rejected refresh writes for an organization grant (grant `needs_reauthorization`, connection `draft` and disabled, no `lastError`), and examines the reason. The second test makes sure that the private connection of another person is not named.

## Verification

- Failing before the fix (in `server/`): `pnpm exec vitest run src/__tests__/connection-intents-service.test.ts -t "must be renewed"`. The shared test failed with `AssertionError: expected 'Review identity and access for this agent' to contain 'Saved sentry'`.
- The private-connection test failed against the first commit of this PR (`lastError` match without an audience check) with `expected 'The sign-in for "Private Sentry of another person" expired. ...' not to contain 'Private Sentry of another person'`.
- Passing after the fix: `connection-intents-service.test.ts` passes (37 of 37).
- Related suites pass: `agent-setup-connection-intents`, `connection-aggregator-fallback`, `satisfied-connection-intents` (168 of 168 with the file above).
- `pnpm run typecheck` in `server/` passes.
- The grant state comes from the refresh writer in `grant-refresh.ts`. Existing tests in `tool-access-service.test.ts` make sure that a rejected refresh sets the grant to `needs_reauthorization`.
- I did not run the full `pnpm verify`.

## Risks

- Low risk. The change only changes the text of one search reason. It does not change state, access decisions, or what agents can call.
- The reason does not use `lastError`, because health checks write display text into that column, and the default refresh path does not write it.
- The member-removal path also sets remaining grants to `needs_reauthorization`. The text "must be signed in again" is correct for that case too.
- I examined the refresh path. Paperclip already tries the refresh token before it marks the grant. This PR does not change that path.
- Out of scope: Paperclip does not send a notification to the connection owner, and the connection request card does not offer to reconnect the existing connection. These need separate changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)




Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:25:00 -06:00
DottaandClaude Opus 5.5 c36e791bd9 fix(native): back off and retry a provider rate limit that reports no reset (#22216)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - Native runs drive ACP agents, such as Claude, through the runner. The runner types each provider failure for the server.
> - When a provider limit stops a turn, the server holds the task. It retries the task automatically at the reset that the provider reports (#20692).
> - Claude can reject a request at the account rate limit with "This request would exceed your account's rate limit. Please try again later." This text has no reset time and no window.
> - The server then plans no automatic Retry. The task stays blocked until a person retries it, and the case work stops before a final reply.
> - This pull request gives a `rate_limit` failure without a reported reset the default provider-limit cooldown, and then requests the task's Retry.
> - The benefit is that a short account rate limit no longer strands a task. The existing Retry admission keeps the saved work and all gates.

## Linked Issues or Issue Description

Refs #20692, #19246, #21100

**What happened?**

A native task ran on a Claude agent. In the middle of the turn, Claude returned HTTP 429 with "This request would exceed your account's rate limit. Please try again later." The runner typed the failure as `rate_limit`, with no `retryAt`. The server put the task on hold with "Restore model provider usage capacity, then retry the task. The provider did not report when its usage limit resets." It did not plan an automatic Retry. The task stayed blocked until a person retried it.

**Expected behavior**

A rate limit passes by itself. Paperclip should wait a bounded time, and then retry the task. It should keep the saved work and should not start another request before the wait ends.

**Steps to reproduce**

1. Run a native task on a Claude agent whose account is at its rate limit.
2. Let Claude reject a request in the turn with the 429 account rate-limit text above.
3. See that the task hold has no reset time and that no automatic Retry is planned.

**Paperclip version or commit**

`master` at 8282d386da.

## What Changed

- `readNativeUsageLimitReset` (server, native runtime) now gives a `rate_limit` turn failure without a reported reset an estimated reset. The estimated reset is the failure time plus `PROVIDER_LIMIT_DEFAULT_COOLDOWN_MS` (one hour). This is the same wait that already holds the agent's new work after the same failure, so both holds end at about the same time.
- The reset carries an `estimated` flag. The hold text says that the provider did not report the reset, and it gives the time when Paperclip will retry.
- A `quota_exhausted` failure without a reported reset does not change. It still waits for a person, because an exhausted allowance does not pass by itself.
- The existing bounds still apply: a maximum of three automatic Retries in a row without a successful run, Retry admission gates, and the accountable-user check.
- This change is a fallback. When the runner sets `retryAt` (for example, #21100 for a named rate-limit window) or the text contains a reset, Paperclip uses that time.

## Verification

- New test in `server/src/services/native-runtime/native-provider-capacity.integration.test.ts`: "backs off and retries a Claude account rate limit that reports no reset".
- Before the fix, the test failed: `AssertionError: expected NaN to be greater than or equal to 1795028502736`. The hold had no `providerLimitResetAt`, because no reset was planned.
- After the fix: `cd server && pnpm vitest run src/services/native-runtime/native-provider-capacity.integration.test.ts` passes (11 tests).
- Related files pass: `native-provider-limit-cooldown.integration.test.ts`, `src/__tests__/heartbeat-provider-limit-wait.test.ts`, `task-interruption-presentation.test.ts`, and the capacity file (4 files, 90 tests).
- `cd server && pnpm typecheck` passes.
- `pnpm verify` selected 2,925 of 3,792 Vitest files, because the changed module is deep in the server import graph. Its first selected group passed (2 files, 12 tests). I stopped the broad local run and rely on the required CI test checks for that coverage.

## Risks

- Low risk. The change applies only to a native turn failure that is typed `rate_limit` and has no reported reset.
- If the account stays at its limit for more than one hour, the automatic Retry can fail again. After three automatic Retries in a row, the task waits for a person, as before.
- The task stays in the blocked state during the wait, as it does for a reported reset. A person can select Retry to start it sooner.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no doc change is necessary: no subsystem doc describes this hold, and the hold text and tests describe the behavior)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:24:52 -06:00
DottaandClaude Opus 5.5 f472991647 fix(native-runtime): resolve assigned tool names to the authorized alias (#22213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work.
> - Native runners give agents their assigned app tools through two wrapper tools: `paperclip_search_assigned_tools` and `paperclip_call_assigned_tool`.
> - Search returns each tool with a hashed alias (`app_<readable>_<digest>`). The description also shows the display name.
> - Agents sometimes pass the gateway tool name or the display name instead of the alias, in `schemaTool` or in the call `name`.
> - The wrapper rejected these values with a bare `assigned_mcp_tool_unknown`. The agent had to search the catalog again before it could call the tool.
> - This pull request resolves an unambiguous name to the authorized alias, and gives a corrective error when the name is not known.
> - The benefit is fewer failed tool calls and fewer repeated catalog searches, with no change to authorization.

## Linked Issues or Issue Description

**What happened?**
An agent searched its assigned tools and then called a tool with the gateway name or the display name (for example `sql_query`). The search with `schemaTool` and the call both failed with `assigned_mcp_tool_unknown`. The call error did not tell the agent what to do.

**Expected behavior**
When the name identifies exactly one tool in the session catalog, the wrapper uses that tool. When the name is not known or is ambiguous, the error tells the agent to use the exact name from search.

**Paperclip version or commit**
`master` at 8282d386da.

**Steps to reproduce**
1. Assign an app tool with the display name `sql_query` to an agent on an on-demand native runner.
2. Call `paperclip_search_assigned_tools` with `{"query":"","schemaTool":"sql_query"}`.
3. Call `paperclip_call_assigned_tool` with `{"name":"sql_query","arguments":{...}}`.
4. See `assigned_mcp_tool_unknown` for both calls.

## What Changed

- `server/src/services/native-runtime/assigned-mcp-tools.ts`: add `resolveAssignedName`. It accepts the exact alias. If the value is not an alias, it looks for tools whose gateway name or display name is equal to the value. It resolves only when exactly one tool in the session's pinned catalog matches in either field. It trims surrounding white space.
- Search with `schemaTool` uses the resolved alias. The fresh-grant and work-mode filters still apply to the result.
- `paperclip_call_assigned_tool` uses the resolved alias. An unknown name returns `assigned_mcp_tool_unknown: name must be an exact name returned by paperclip_search_assigned_tools. Search assigned tools again and use the returned name.`
- A name that matches more than one tool returns a different error: `assigned_mcp_tool_unknown: name matches more than one assigned tool. Use the exact name returned by paperclip_search_assigned_tools.` The `schemaTool` path gives the same error, with the text `schemaTool` and `inputSchemaRef`.
- `server/src/services/native-runtime/assigned-mcp-tools.test.ts`: add regression tests.

## Verification

- Failing before the first fix: `pnpm --filter @paperclipai/server exec vitest run src/services/native-runtime/assigned-mcp-tools.test.ts -t "resolves a gateway name"` failed with `promise rejected "Error: assigned_mcp_tool_unknown: schemaTool must match an authorized inputSchemaRef..." instead of resolving`.
- Failing before the review fix (second commit):
  - `rejects a name that is one tool's gateway name and another tool's display name` failed with `promise resolved "{ content: [...] }" instead of rejecting`. The old code ran the read tool `sql_query`, but the write tool has the display name `sql_query`.
  - `resolves a gateway name or unique display name ...` failed with `expected ... 'assigned_mcp_tool_unknown: name matches more than one assigned tool' but got 'assigned_mcp_tool_unknown: name must be an exact name ...'`.
- Passing after the fix: the full file `src/services/native-runtime/assigned-mcp-tools.test.ts` passes (59 of 59 tests).
- The tests also check these safety conditions:
  - An ambiguous display name (two tools named `search`) is rejected, on search and on call.
  - A name that is the gateway name of one tool and the display name of a different tool is rejected, and the gateway receives no call.
  - A write tool that the display name resolves to is still denied in planning mode.
  - A revoked tool is still rejected after the fresh grant list changes.
  - A tool that is not in the pinned catalog is still rejected.
  - The gateway receives only the resolved gateway tool name.
- No other code or test depends on the changed error text (`git grep assigned_mcp_tool_unknown`).
- Not run locally: the full server `tsc --noEmit`. The sandbox did not have enough memory and disk for it. CI runs the typecheck. It passed on the first commit.
- CI on the first commit: `Verify Paperclip Runner (vitest native server)` did not fail a test. All tests that ran passed. The job stopped because the CI runner received a shutdown signal. The cancelled jobs stopped for the same reason.

## Risks

- Low risk. Resolution only maps to tools that are already in the session catalog. The same permission, work-mode, fresh-grant, and gateway checks apply after resolution.
- A name that matches more than one tool, in either field, is not resolved. The agent must use the alias.
- Direct (non-wrapper) tool calls still require the exact alias.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)





Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:24:45 -06:00
DottaandClaude Opus 5.5 37de2b7259 fix(tools): keep audit summaries valid jsonb when the cut lands inside an emoji (#22207)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for work
> - Agents call connector tools through the tool gateway. The gateway writes an audit row in `tool_call_events` for each call.
> - The audit row keeps short request and result summaries in `jsonb` columns. `summarizeToolValue` cuts long summaries to 3997 UTF-16 code units.
> - When the cut falls between the two halves of an emoji, the summary ends in a lone high surrogate. PostgreSQL `jsonb` rejects this value.
> - The audit insert then fails. The agent gets the database error and not the connector result.
> - This pull request makes the cut keep whole characters.
> - The benefit is that connector calls with long, emoji-rich results return their result and keep their audit record.

## Linked Issues or Issue Description

**What happened?**
Some connector tool calls failed with a database error. The error text starts with `Failed query: insert into "tool_call_events"`. The connector call itself worked, but the agent did not get the result. This occurred in a small part of calls (less than 1%).

**Expected behavior**
The connector call returns its result. Paperclip writes the audit row for the call.

**Steps to reproduce**
1. Call a connector tool that returns a long result (more than 4000 characters) with emoji in it.
2. Make sure an emoji is at the summary cut point (UTF-16 code unit 3997 of the serialized value).
3. The `tool_call_events` insert fails with `22P02 invalid input syntax for type json` ("Unicode low surrogate must follow a high surrogate"). The tool call returns that error.

**Relevant logs or output**
`Failed query: insert into "tool_call_events" (...)` from the gateway `call_completed` audit write. The row has request and result summaries.

## What Changed

- `server/src/services/tool-content-guards.ts`: `summarizeToolValue` now cuts long summaries with a small `cutWholeCharacters` helper. If the last kept code unit is a high surrogate, the helper drops it. The summary is still at most 4000 code units. The `sha256` and `sizeBytes` values do not change, because they use the full serialized value.
- `server/src/__tests__/tool-content-guards.test.ts`: add a regression test. It puts an emoji across the cut point and checks that the summary has no lone surrogate.

## Verification

- Failing before the fix: `pnpm --dir server exec vitest run src/__tests__/tool-content-guards.test.ts -t "without splitting an emoji"` failed. The received summary ended with a lone high surrogate before `...`, and the expected summary did not.
- Passing after the fix: the same command passes. The full file passes (47 tests).
- Database check: on embedded PostgreSQL, the old summary shape as `jsonb` fails with `22P02` ("Unicode low surrogate must follow a high surrogate"). The new summary shape is accepted. I used a temporary probe for this check. I did not commit it.
- `pnpm --dir server typecheck`: no errors in `server`. The local run showed errors only in `cli` and `kv-demo-mcp-server`, because I installed only the server dependencies (low disk space).
- Not run: the full `pnpm verify`. `prompt-injection-prevention.test.ts` does not load in my environment (`baseURL must be a non-empty string`). The same error occurs on unchanged `master`.

## Risks

- Low risk. The change affects only the readable summary text. A summary can be one code unit shorter when the cut point is inside an emoji. Hashes, sizes, redaction, screening, and authorization do not change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution, in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context to this change
- [x] I have specified the model used (with version and capability details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template
- [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)



Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:24:18 -06:00
84a529df4f fix(monitors): let low-trust reviewers schedule a self-wake on their own task (#22212)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use the `set_task_monitor` tool to save a timed follow-up
check on their task
> - Agents that run with the `low_trust_review` preset (for example,
pull request reviewers) cannot save this check
> - The tool requires company-wide `runtime:manage` access, and the
preset denies that access
> - Thus a reviewer must wait for a different event before it can
continue its review
> - This pull request lets the assignee save or clear a self-wake
monitor on its own task, and keeps the company-wide restriction
> - The benefit is that reviewers can continue their own review on time,
with no new company-wide access

## Linked Issues or Issue Description

**What happened?**

A `low_trust_review` agent calls `set_task_monitor` on its own
in-progress task. The call fails with this error: "low_trust_review
agents cannot use company-wide or privileged runtime:manage APIs by
default." The tool is advertised to the agent, but the agent cannot use
it. The agent cannot schedule its own follow-up check.

**Expected behavior**

The assignee can schedule or clear a monitor that only wakes itself on
its own task inside its trust boundary. Monitors that create recovery
tasks or escalate to the board stay denied. Company-wide
`runtime:manage` stays denied.

**Steps to reproduce**

1. Assign an in-progress task to an agent.
2. Start a native run for that task with a `low_trust_review` execution
policy whose boundary root is the task.
3. Call `set_task_monitor` with a future `nextCheckAt` and notes.
4. See the 403 error with reason `deny_low_trust_boundary`.

**Paperclip version or commit**

`master` at 8282d386da.

## What Changed

- `assertCanManageIssueMonitor` accepts an optional own-task scope. With
this scope, a `deny_low_trust_boundary` result for `runtime:manage` does
not block the assignee when both the requested monitor and the stored
monitor are the assignee's own self-wake (`wake_owner` recovery,
scheduled by the assignee, not the provider-quota service) or are empty.
- The native `set_task_monitor` tool passes this scope only when the
agent, project and task policies alone resolve the same low-trust
boundary as the current run. A monitor wake starts a run from those
sources, not from the run snapshot. Thus the woken run keeps the same
low-trust limits. A low-trust boundary that only the run snapshot
carries is still denied.
- The tool still checks `issue:read` and `issue:mutate` on the target
first, and still requires an in-progress or in-review task assigned to
the calling agent.
- The `create_recovery_issue` and `escalate_to_board` recovery policies
still require `runtime:manage`. The assignee also cannot clear or
replace a board-scheduled, recovery, escalation or provider-quota
monitor.
- The HTTP issue routes do not change.
- `doc/LOW-TRUST-PRESETS.md` describes the new rule.

## Verification

- Test 1 failing before the fix: `pnpm vitest run
src/services/native-runtime/task-monitor.integration.test.ts -t
"low-trust reviewer"` (in `server/`). The new test failed: `promise
rejected "Error: low_trust_review agents cannot use company-wide or
privileged runtime:manage APIs by default."` from
`assertCanManageIssueMonitor`, with `{ status: 403, details: { reason:
'deny_low_trust_boundary' } }`.
- Test 2 failing before the trust fix: `pnpm vitest run
src/__tests__/issue-monitor-scheduler.test.ts -t "low-trust limits"`. A
run with a low-trust policy only in its snapshot could save a monitor
(`promise resolved ... instead of rejecting`). After the fix, that call
is denied. When the task carries the policy, the monitor is saved, the
real scheduler fires it, and the woken run resolves to
`low_trust_review` with the same boundary.
- Test 3 failing before the stored-monitor fix: the extended "low-trust
reviewer" test cleared a board `escalate_to_board` monitor (`promise
resolved ... instead of rejecting`). After the fix, clear and replace
are denied for board-scheduled, `escalate_to_board`,
`create_recovery_issue` and provider-quota monitors.
- Passing after: `task-monitor.integration.test.ts`,
`issue-monitor-scheduler.test.ts` and
`native-preserved-monitor-wait.test.ts` pass (100/100).
- `pnpm run typecheck` in `server/` passes.
- I did not run the full `pnpm verify`.

## Risks

- Low risk. The change applies only to the native tool, only to the
assignee, and only to the task-scoped `wake_owner` monitor. A monitor
wake goes only to the same agent on the same task, and that run resolves
the same low-trust boundary.
- The bypass depends on the `deny_low_trust_boundary` decision reason.
Other denials (for example, a viewer responsible user, a task bridge
key, or a skill-test token) still block the call.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in an agent harness.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:19:11 -06:00
DottaandClaude Opus 5.5 8a295f944d fix(runner): fail fast when an artifact storage read stalls (#22210)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use the `download_artifact` tool to copy a saved API artifact
into their workspace
> - The server reads the artifact bytes from storage before it installs
the file
> - This storage read has no deadline. A stalled storage response or
stream keeps the tool call open
> - The provider MCP client stops the call after about five minutes with
the error "The operation timed out."
> - This pull request puts a 60-second deadline on the storage read and
returns a clear, retryable error
> - The benefit is that the agent gets a fast, clear result and can
retry, and does not wait for five minutes

## Linked Issues or Issue Description

**What happened?**
Some `download_artifact` calls stopped for four to five minutes. Then
the agent got only "The operation timed out." In one case, a retry of
the same 1 MB artifact completed in about 20 seconds.

**Expected behavior**
A stalled storage read must fail quickly. The error must tell the agent
that no file was written and that it can retry the same download.

**Steps to reproduce**
1. Give `PaperclipRunnerToolAuthority` a storage service whose
`getObject` never resolves, or whose stream sends one chunk and then
stops.
2. Call `download_artifact` for an artifact in that storage.
3. The call does not settle. Nothing aborts the storage request.

**Root cause**
`#readApiFile` called `storage.getObject(...)` without a `signal` and
read the stream with no deadline. Other storage reads in the runner, for
example chat attachment reads, already have a deadline.

## What Changed

-
`server/src/services/native-runtime/paperclip-runner-tool-authority.ts`:
the artifact branch of `#readApiFile` now uses a 60-second
`AbortSignal.timeout`. It sends the signal to `storage.getObject`. It
destroys the stream when the deadline expires. Every storage stream gets
an error listener before it can be destroyed, so a late reply cannot
raise an uncaught `AbortError`. It returns HTTP 504 with code
`artifact_storage_timeout` and the message "Artifact storage read did
not finish within 60 seconds. No file was written; retry the same
download."
- A new optional binding field, `storageReadTimeoutMs`, sets the
deadline for tests. The default is 60 seconds. The production callers do
not change.
-
`server/src/services/native-runtime/artifact-transfer.integration.test.ts`:
a new regression test covers a storage response that never arrives and a
stream that stops after one chunk.

## Verification

- Failing before the first fix: `cd server && npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts -t
"stalls"` failed with `Error: Test timed out in 15000ms.` The
`download_artifact` call never settled.
- Failing before the late-reply fix: the same test now also covers a
storage reply that arrives after the deadline, with a real
`createReadStream` that gets the aborted signal. It failed with an
uncaught `AbortError` (`ABORT_ERR`).
- Passing after both fixes: the same command passes. Each stalled call
fails in less than 5 seconds with the retryable error. All storage
signals are aborted. No uncaught error occurs. The workspace has no
files.
- `npx vitest run
src/services/native-runtime/artifact-transfer.integration.test.ts
src/services/native-runtime/paperclip-runner-tool-authority.test.ts`: 49
passed.
- `npx vitest run
src/services/native-runtime/runner-api-artifacts.integration.test.ts
src/services/native-runtime/runner-api-artifacts.test.ts
src/services/native-runtime/issue-document-batch.integration.test.ts`:
38 passed (first revision).
- `pnpm run typecheck` (server): passed.
- `pnpm verify` selects a very large set of files for this module. I did
not run the full set locally. CI covers it.

## Risks

- Low risk. The deadline applies only to the artifact read from storage,
and artifacts are at most 10 MiB. A healthy read takes seconds.
- A very slow but working storage provider that needs more than 60
seconds for 10 MiB now gets a retryable error.
- The error occurs before any file write, so no partial file stays in
the workspace.
- This change does not fix a slow storage provider. It makes the failure
fast and clear.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in a sandboxed workspace.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:14:46 -06:00
DottaandPaperclip ee68098584 fix(runner): keep current task requests readable in OpenCode traces (#21584)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Company traces let operators compare each request with its reply.
> - OpenCode limits each text field to 8 KiB.
> - A fresh session can put old history before the current request.
> - The limit can remove the current request from the trace.
> - This change selects the current task text first and links the
retained submission.

## Linked Issues or Issue Description

Refs #18098 and #18699. Neither merged change selects the current task
request.

**What existing behavior does this improve?**

OpenCode company traces can contain only part of the old session
history.

**Current behavior**

The exporter clips the full model envelope from its start. A sampled
trace has a useful HOLD reply but a request that ends inside old
history. A truncation flag is not proof that the task failed.

**Proposed behavior**

Select the task prompt before the text limit. Remove a valid leading
generated handoff packet from the excerpt. Link the full retained run
event through the existing API.

[Source
pattern](https://www.braintrust.dev/app/Paperclip/p/Paperclip%20Staging/patterns/e6d8da04-ca3e-4bd9-bd90-737f7717a9e1).

### Hypothesis and thesis for a fix

The baseline exporter spends its text budget on the envelope and old
history. The new regression confirms this cause: the received OTLP input
loses a short current request after a long handoff. Select the current
request in the OpenCode projection before redaction and clipping. Keep
storage and access checks with their existing owners.

## What Changed

- Select task identity and prompt for known native envelopes. Put
pending user messages first in continuation excerpts.
- Omit only a valid leading generated handoff packet. Keep plain text
and unknown formats on the existing path.
- Add a relative run-events reference under the prompt capture switch.
Readers must sign in to the source instance and page to the matching
turn.
- Keep raw tool argument values excluded. Keep redaction, opt-in,
capture switches, company access and size limits.
- Add exporter, proxy, installed-runtime and private-run retrieval
tests. Fix the proxy test's session-directory lookup so it skips the
output-custody directory.
- Document the excerpt and full-event retrieval path.

## Verification

- Baseline `6b6c94a4ea8f3d1b58767d67a7a2fc4ceaecda9c`: the new
long-handoff regression failed because OTLP input did not contain the
current request.
- Same regression passes after the fix. Exporter, redaction and bundled
proxy suites: 39 tests passed.
- `PAPERCLIP_OPENCODE_QUALIFY=1 pnpm --dir packages/paperclip-runner
exec vitest run
src/drivers/opencode/company-observability-runtime.test.ts`: 2 passed.
Real installed OpenCode 1.18.34 and real HTTP OTLP receiver; local SSE
model fixture. Both capture settings pass. The model receives the full
original submission.
- Private-run API suite: 31 tests passed on base
`340157d1d6ec01ad0fa55970e148c6cd332bbcb1` with these changes.
Authorized readers retrieve a complete request over 8 KiB through
pagination. Other readers receive 404.
- Runner TypeScript source and test checks and TypeScript build passed.
`git diff --check` passed.
- `pnpm verify --explain` returned exit 2 and required broader CI. No
full local-suite pass is claimed.
- Initial harness attempts used the wrong Vitest working directory,
lacked built plugin-sdk, selected the wrong proxy fixture directory, or
expected 403 instead of the API's 404. These are not red regression
results. The corrected checks above pass.
- Review follow-up `dfe995a77f3eb3280a6b081b4df9428fac85f540` removes
added code comments required by CONTRIBUTING.md. Exporter + installed
OpenCode runtime suites reran: 30 tests passed, with the local SSE model
and real HTTP OTLP receiver. `git diff --check` passed. The review
thread is resolved. Greptile passed on this exact revision with
confidence 5/5 and no actionable findings. Final-head typecheck, build,
Runner and E2E checks passed; server test shards and the amd64 Runner
image build remain in progress. No hosted model call or deployed
recovery is claimed.

## Risks

Known envelope excerpts change shape from serialized envelope text to
readable task text. The handoff selector depends on the current
generated packet format; unknown framing falls back to the full prompt
excerpt. Long current requests still clip. Full-event access depends on
source-instance permission and retention. Export remains best effort.
The new reference grants no access and includes no credential. This PR
is unmerged and not deployed.

## Model Used

OpenAI GPT-6 through Codex, with reasoning, tool use and code execution.
The exact serving variant and context size are not exposed to this
session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Final verification (2026-11-14): exact head
`dfe995a77f3eb3280a6b081b4df9428fac85f540` has 60 successful checks, one
neutral security scan, and five conditional skips (cache write,
publication/attestation, incremental selection, Storybook paths).
Applicable typecheck, build, general and serialized tests, Runner tests,
E2E and both Runner image architectures passed.
[CI](https://github.com/paperclipai/paperclip/actions/runs/43572758306),
[Runner
images](https://github.com/paperclipai/paperclip/actions/runs/43572758233),
[Greptile](https://github.com/paperclipai/paperclip/runs/132380487679).
Greptile 5/5, no unresolved actionable findings. Unmerged and not
deployed.

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 15:08:04 -06:00
DottaandClaude Opus 5.5 7f9c115cc2 fix(github): let a queued push start the next review turn instead of a blocker continuation (#22223)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A GitHub review bot reviews each PR in a task. Each push sends a new
review request to that task as a chat message.
> - A push that arrives during a review waits as a deferred chat wake.
That wake starts a new turn. The turn binds its tools to the newest
pushed commit.
> - #21959 and #22140 continue a run that reports a blocker in a fresh
session. That continuation keeps the source binding of the blocked run.
It also holds the task, so the waiting push request does not run.
> - On staging, a push during a review made the publication fail with
"requested for an older pull request head". The agent reported a
blocker. Paperclip then continued twice with the old binding. Each
continuation finished the review and could not publish it.
> - A second failure mode: a complete re-review that listed only the
files changed since the prior review got a short rejection. The
rejection named no files, and a small model stopped instead of
correcting the submission.
> - This pull request lets a queued chat request take the next turn
instead of a blocker continuation. It also names the missing files in
the coverage rejection.
> - The benefit is that reviews publish for the newest push without a
person, and agents can correct incomplete coverage on the next call.

## Linked Issues or Issue Description

**What happened?**
On staging, automatic GitHub reviews stopped publishing. The bots posted
"Publication is currently blocked" and "Paperclip must authorize the
review for <newest commit>". The run history showed this sequence: a
review run started for the first commit. Two pushes arrived during the
run. Steering into the active run was rejected for both pushes, so they
waited as deferred chat wakes. The run reported a blocker. Paperclip
then started two `native_environment_blocker` continuations. Each
continuation had no message of its own, so its GitHub tools resolved to
the first commit's request and refused the current head. The waiting
push requests did not run until a later push.

**What did you expect?**
A push request that waits behind a run starts the next turn, bound to
that push. The agent then reviews and publishes for the current head.

**Steps to reproduce**
1. Start an automatic review on a PR.
2. Push a new commit while the review runs, so the review request waits
as a deferred chat wake.
3. Let the run report a blocker (for example after the stale-head
publication refusal).
4. See that a fresh-session retry starts with the old binding and the
push request stays deferred.

Refs #21959, #22140, #22066

## What Changed

- `status-arbiter.ts`: new rule before the blocker continuation lanes.
When a blocked run has a queued chat request for the same task and
agent, the decision is `preserve` with only `release_checkout` (reason
`native_blocker_superseded_by_queued_request`). The normal release drain
then starts the queued request. A pause hold or a board-owned blocker
keeps the existing behavior.
- `native-runtime/queued-chat-request.ts`: new `hasQueuedChatRequest`
query. It uses the same chat-wake test that keeps chat wakes out of
first turns (`chatConversationId` or `externalChatProvider`).
- `native-run-finalizer.ts`: supplies `hasQueuedChatRequest` to the
arbiter, only for a blocked report.
- `chat-github-review-policy.ts` / `chat-github-reviews.ts`:
`githubCompleteCoverageGap` replaces the inline check. The rejection now
names the missing files (up to 20), says that a complete review covers
all changed files, and gives the two corrections (add them to
`reviewedPaths`, or submit `complete: false` with `omittedPaths`).

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-environment-blocker.integration.test.ts`.
The new case failed before the fix with reason
`native_environment_blocker` and one scheduled retry, as on staging.
After the fix it passes: no retry, the task stays `in_progress`, and the
chat wake stays deferred for the release drain.
- Red/green: `npx vitest run
src/services/native-runtime/status-arbiter.test.ts -t "queued chat
request"`.
- Red/green: `npx vitest run
src/services/chat-github-review-policy.test.ts -t
githubCompleteCoverageGap`.
- `native-agent-blocker.integration.test.ts` and the full arbiter and
review-policy suites pass.
- `cd server && npx tsc --noEmit -p .` passes.
- `pnpm verify` on the changed files (results in the CI checks below).

## Risks

- Low to medium. The new rule changes only blocked reports that would
otherwise auto-continue while a chat request waits behind the run.
Without a waiting chat request, the behavior does not change.
- The waiting request now starts the next turn instead of the
continuation. If the blocker is real, that turn can report it again, and
the existing lanes apply.
- The finalizer does one more indexed read, only for blocked reports.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code with tool use: code
search, editing, shell, test runs, and staging run inspection through
the browser.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 15:00:57 -06:00
DottaandClaude Fable 5.2 33fac97cad feat(routines): ship the redesigned routine page and let live webhooks be tested again (#22205)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Routines start agent work on a schedule, from a webhook, or on
demand
> - The routine detail page split one routine across eight sections
behind a section sidebar (overview, triggers, variables, secrets,
delivery, runs, activity, history)
> - People had to hunt through sections to see what a routine does and
to change it, and a finished webhook could not be tested again without
starting the routine
> - This pull request makes the approved two-column routine page, one
Edit dialog and one Webhook dialog the real routed page, and lets a live
webhook go back into test mode
> - The benefit is one page that answers what the routine does, when it
runs, who runs it and what it made, with every change in one dialog

## Linked Issues or Issue Description

**Problem**
The routine detail page spreads one routine over eight sections and a
section sidebar. Changing a routine means visiting several sections,
each with its own save bar. A webhook that is already live cannot be
tested again: the API only accepts `setupPending: false`, so testing a
live webhook starts the routine.

**Proposed solution**
Ship the reviewed design (stories `routine-page-design`,
`routine-edit-dialog-design`, `routine-webhook-dialog-design`) as the
routed page in both UI shells, wired to the real API. Let `PATCH
/api/routine-triggers/:id` accept `setupPending: true` for a live
webhook so the next events are checked but never start the routine.

**Alternatives considered**
Keep the sections and restyle them. The owner rejected this: the section
sidebar should go.

## What Changed

- `/routines/:id` renders `RoutinePage` in both shells
(`RoutineDetail.production.tsx` is removed). Status switch, Repeats
rows, default agent, responsible user, project, Asks for, Secrets, Run
rules, Managed by plugin, Instructions with Show all, copy and the
comment count, and History.
- History is paged: 20 tasks per page from `GET
/companies/:id/issues?originKind=routine_execution&originId=…&sortField=created&sortDir=desc&limit=21&offset=…`,
with a real Load more. Coalesced runs show as "folded into a task".
- A failed refetch keeps the loaded page on screen (`useQueryView`); a
History failure shows `QueryErrorState` in place.
- Header: Edit, "…" (Version history, Audit log, Archive / Restore) and
Run now. In the streamlined desktop shell they sit in the breadcrumb
toolbar; elsewhere above the page.
- Version history (`/routines/:id/history`, restore and restored-webhook
keys kept) and Audit log (`/routines/:id/activity`,
`RoutineAuditActivity`) open as sub-views with a way back. Old section
addresses and `?tab=` redirect to `/routines/:id`.
- Edit dialog saves title, instructions, variables, secrets, agent,
project and responsible user in one request against the revision the
dialog opened with (save conflict shows "changed while you were
editing"). The first schedule is created, changed or removed after that.
Priority and delivery settings are never sent, so saved values stay.
- Webhooks are created, changed, tested and removed (with Undo) right
away from the Edit dialog, with the Webhook dialog nested inside it.
Closing the Webhook dialog ends setup or test mode. A webhook left in
test mode shows "Testing · events don’t run" on the page and offers
Finish testing.
- Responsibility (only with `enableResponsibilities`): the page shows
direct and through-project responsibilities from the link lookup; the
dialog links or unlinks the direct one. With the flag off nothing is
fetched or shown.
- The routines list creates routines with the same dialog in create
mode, filed in the open folder, and opens the new routine page.
- Server: `setupPending: true` on a live webhook enters test mode
(planned by a pure `planTriggerUpdate`, which also gives the activity
action) (422 for other trigger kinds; 409 for a disabled or removed
webhook or an archived routine). Test deliveries keep the last real
delivery as `lastWebhookDelivery.previous`, and finishing testing
restores it. Activity `routine.trigger_test_started`.
- Additions to the reviewed design, from review, so no capability is
lost: the schedule popover has Pause/Resume; standing-check and API rows
open a small popover with Pause/Resume and Remove; removing a trigger
from the page leaves a "… removed. Undo" line under Repeats. Stories:
`StandingCheckOpened`, `ScheduleTriggerPaused(Opened)`,
`RemovedWithUndo`.
- Save is several requests (routine, schedule, responsibility). It
changes only the schedule the form opened with, records each finished
step, and a partial failure says what was saved and that saving again
finishes it. The dialog cannot be dismissed while saving. Failed reads
show a Retry state, never empty choices.
- Removed: the routine section sidebar and contextual sidebar, the
trigger wizard, trigger cards, save bar, section components and their
tests.
- Storybook: the three routine stories moved to `stories/routines` as
current production stories, a new story mounts the routed page against a
fixture API, and stories that used the old page were updated or removed.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/routines-e2e.test.ts` (17/17) and
`src/__tests__/routines-service.test.ts` (105/105).
- `pnpm exec vitest run` on the routine page, routines list, routine
form, webhook instructions, Layout, shell navigation, audit and editor
tests: 275 tests pass.
- `cd ui && npx tsc -p tsconfig.json --noEmit`, `node
scripts/check-token-gates.mjs`, `node scripts/check-maps.mjs`, `node
scripts/storybook-catalog.mjs --check`: pass.
- `pnpm verify` exits 2 (global CI input changed); required CI supplies
the broad coverage.
- Manual: an isolated worktree instance (`pnpm paperclipai worktree init
--empty`, `pnpm dev`) with a seeded routine. Headless Playwright: an old
`/triggers` address redirects, no section sidebar, History shows 20 then
23 after Load more, the schedule popover opens, pause and resume, Edit →
Add webhook → created key → Finish, title save, Test webhook → Finish
testing, Version history, Audit log shows "Started testing a webhook",
phone width has no horizontal overflow, and Create routine from the list
opens the new page. No console errors or failed API calls.

## Risks

- Behaviour change: the section pages are gone. Old links redirect to
the single page.
- The delivery-setting editors (concurrency, catch-up, activity gate)
and priority are removed from the UI. Saved values stay, and the API
still accepts them.
- The create dialog no longer offers a folder picker or delivery
settings. A new routine goes into the folder the list is showing and
takes the default settings.
- Replace key in the Webhook dialog acts at once, as in the reviewed
design. The old confirmation is gone.
- No migration. `lastWebhookDelivery.previous` is an optional field
inside the existing jsonb column.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, in Claude Code with
tool use (shell, file edits, headless browser) and subagents for the
server change and the Storybook move.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-18 13:13:15 -06:00
651749bdb4 feat(connections): AWS connector with credentials minted per connection (#22195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use outside services through connections, and the control
plane holds each connection's credentials
> - Companies want agents to "do things on AWS" with exactly the
permissions they choose, without Paperclip storing long-lived AWS keys
or rebuilding the AWS API
> - AWS hosts an MCP server that runs any AWS API call in its own
sandbox, and it accepts a one-hour bearer token that can be minted from
IAM credentials
> - This pull request adds an AWS connection that mints that token for
each connection at call time. The role method proves the connection with
the company's workload identity issuer (#22083), so no AWS key is stored
> - The benefit is that each AWS connection carries its own role and
permissions, AWS enforces them, and agents never see any AWS credential

## Linked Issues or Issue Description

Refs #22083 (workload identity issuer) and #22191 (spike findings).

**Subsystem affected**
Connections (tool access), tool gateway, shared app definitions,
instance settings.

**Problem or motivation**
Paperclip has no way to connect AWS. AWS's hosted MCP server needs a
bearer token from AWS Sign-in, minted from IAM credentials. No existing
credential path mints a header at call time from a role or key.

**Proposed solution**
Methods can declare `credentialMinter: "aws_mcp"`. The gateway and
connection health checks mint the `Authorization` header at call time,
and they share one in-memory cache per connection and grant (about 55
minutes). There are two methods:
- Role (`AssumeRoleWithWebIdentity` with a Paperclip-signed connection
token).
- Access key (the key directly, `GetFederationToken`, or `AssumeRole`).

Read-only connections add `ReadOnlyAccess` and
`AWSMCPSignInOAuthAccessPolicy` as session policies, so AWS refuses
changes. The feature is behind the `enableAwsConnector` experimental
setting.

**Alternatives considered**
- Run AWS's `mcp-proxy-for-aws` process on the control plane. Rejected:
it adds a Python runtime and processes, and gives nothing the token
exchange does not.
- Put credentials in the agent sandbox and use the AWS CLI. Deferred:
there is no per-call governance, and every sandbox would need the CLI.
- Add a new `credentialSource`. Rejected: it needs a migration, and the
`vercel_connect` database rules do not fit stored access keys.

## What Changed

- `packages/shared`:
- `credentialMinter` on connection methods. The validator requires
`mcp_remote` and `api_key`, and does not allow `keyPlacement` or another
credential source.
- The `aws` app definition (two methods, reviewed tools,
`mcpSessionRequired`, experimental key).
- The `enableAwsConnector` setting in the catalog, types and validators.
  - `aws` added to the connectable apps.
- `server/src/services/connectors/aws/mcp-credentials.ts`:
- Settings parsing and checks. The URL must be AWS's endpoint for the
saved region, and the role ARN must be valid.
- The minter: role or key session, read-only session policies, Sign-in
exchange, cache keyed on settings, key references and status, in-flight
de-duplication, eviction.
- A plain-language error mapping that never shows AWS's own text,
because that text can contain account IDs and ARNs.
- Gateway (`resolveCredentialHeadersUnrecorded`), credential access
(`resolveCredentialHeaders`), catalog discovery and health-failure
mapping:
  - The AWS branch.
- After a 401, the gateway evicts the token and mints again, then
requires an explicit retry (no replay). Discovery lists again.
- Setup:
- Minted methods may save a draft without contacting the provider. The
role must trust a connection that already exists.
- Saving an AWS role draft publishes the company's issuer, so AWS can
validate it when the customer registers it.
- `workload-identity.ts`: a draft connection can get a token, so setup
can test it. Disabled, archived or switched-off connections still get
none.
- `server/package.json`: `@aws-sdk/client-sts` and
`@aws-sdk/client-signin`. The lockfile is left to the refresh workflow.
- Artwork: AWS light and dark marks from LobeHub's MIT icon set (already
licensed in the repo). The manifest entry passes the brand checks.
- `scripts/ingest-app-definitions.mjs`: minted methods are exempt from
the `keyPlacement` requirement. The override source is
`scripts/app-definition-overrides/aws.json`. Permission reviews are
added.
- UI: the experimental settings toggle, in alphabetical position.
- Docs: `doc/connections/AWS.md`, the connections index, the issuer API
doc (draft rule), and the plan's phase table.

## Verification

- `server/src/services/connectors/aws/mcp-credentials.test.ts` (7 tests)
covers these items:
  - Both methods and the read-only policies.
  - Caching, revision, forced refresh, expiry and eviction.
  - Concurrent de-duplication.
  - The error mapping, with no AWS text leaking.
- Settings validation, including the URL and region check and the rule
that an unknown access level means read only.
- `server/src/__tests__/aws-mcp-connector.test.ts` (3 tests, real setup,
catalog and gateway code with a fake AWS and a session-requiring fake
MCP server) covers these items:
  - The setting gates setup.
  - The access key is never sent as a header or stored in config.
  - A cached token is reused.
  - After a 401, a fresh token is minted and the call is not replayed.
  - Role draft-then-resume publishes the issuer.
  - The signed token has the right `iss`, `sub` and `aud`.
  - An AWS refusal shows a plain health message without the account ID.
- Red/green:
  - Removing the gateway hook fails the access-key test.
  - Turning off `mcpSessionRequired` fails both integration tests.
- Removing draft support for minted methods fails both integration
tests.
  - Restoring each change passes again.
- Nearby suites: `tool-gateway-service`, `tool-access-service`,
`remote-mcp-connectors`, `instance-settings-service`,
`workload-identity`, plus the new files. 775 of 775 pass.
- Shared `app-definitions.test.ts` passes (42), including new
`credentialMinter` schema tests. UI `InstanceExperimentalSettings` (63)
and the brand asset tests pass. `check-app-brand-assets` validates 112
identities.
- Typecheck is clean for `packages/shared`, `server` and `ui`. The only
UI errors are in Storybook's generated catalog files, which this
worktree has not built.
- **Real AWS** (Paperclip dev account, integrated code path, issuer
behind a temporary public tunnel):
- Access key, read only: 8 tools listed. `run_script` `ListBuckets`
succeeded in 8.9 s on a cold sandbox. AWS refused `CreateBucket` ("no
session policy allows").
- Role: the draft was saved without contacting AWS, and the issuer was
published. After an IAM provider and a role trusting only that
connection were created, resuming setup listed 8 tools. `ListBuckets`
succeeded in 12.8 s. The role's own permissions refused `CreateBucket`.
- This run found two bugs the fakes did not: AWS needs MCP sessions for
calls, and the role method needs draft-first setup with the issuer
published. Both are fixed and both now have tests.
- All IAM users, keys, roles, providers and buckets were deleted, and a
listing confirmed it.
- Not run: Paperclip Cloud staging (the setting is off by default), and
`AssumeRole` from an access key with a role ARN against real AWS (unit
tests only).

## Risks

- The feature is off by default, behind `enableAwsConnector`. No
migration.
- New dependencies `@aws-sdk/client-sts` and `@aws-sdk/client-signin`
are in the existing `@aws-sdk` family (the server already uses
`client-s3`).
- Minted tokens are held only in process memory. A restart mints again.
A cached token can outlive a deactivated key by up to an hour, unless
AWS returns 401, which evicts it.
- `aws___run_script` is classified as a write, so the normal approval
and ask-first policy applies. Read-only connections rely on AWS session
policies.
- The guided setup screen and the CloudFormation template are not in
this PR. Setup uses the generic form and the documented draft and resume
steps.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, AWS SDK for JavaScript against the Paperclip dev AWS
account, a Cloudflare quick tunnel, an exploration subagent).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 12:48:49 -06:00
Devin FoleyandPaperclip 8285a81280 fix(native): retry a fresh launch whose provider pack could not be verified (#22106)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote native run starts its provider, the controller
selects a provider pack and verifies it in the sandbox
> - The verification command retries once. If both reads fail without a
result (a timeout, a signal, or a transport error such as HTTP 502), the
run fails with `runner_remote_provider_verification_unavailable`
> - That code holds the run for a person in every case. The hold exists
for reattachment, which does not spend a new provider attempt and could
loop
> - On a fresh launch the code is also not on the list of failures that
prove no provider session started. So the run is classified as
"ambiguous state", and the task waits behind a Board-owned Retry
> - This pull request retries a fresh launch's unverified pack on the
same run, within the provider attempt budget, and keeps the hold for
every other attempt
> - The benefit is that a short sandbox outage during startup no longer
stops a chat or task until a person clicks Retry

## Linked Issues or Issue Description

**What happened?**
A chat message started a fresh native run on a new sandbox. The
preinstalled provider pack did not match, so the controller uploaded the
pack. Both reads that verified the uploaded pack failed with "Request
failed with status code 502". No provider process started. The run
failed with `runner_remote_provider_verification_unavailable`, the
recovery mode was `ambiguous_state`, and the chat showed a Retry button.
A manual Retry then succeeded.

**Steps to reproduce**
1. Start a fresh remote native run.
2. Make both provider-pack verification reads fail without a result, for
example with a 502 from the sandbox command API.

**Expected behavior**
The run retries its startup on the same run, with backoff, because no
provider session started.

**Actual behavior**
The run is held for a person with "Inspect the original provider failure
and explicitly resolve the ambiguous session state".

## What Changed

- `native-session-executor.ts`:
- `nativeProviderRecoveryEvidence` treats
`runner_remote_provider_verification_unavailable` as definitely
pre-session when the attempt is a fresh bootstrap (the caller passes
`bootstrap`). As for every pre-session code, this applies only when the
run has no checkpoint and no provider event.
- `nativeSessionFailureDisposition` takes an optional `freshBootstrap`
flag. With it, `runner_remote_provider_verification_unavailable` is
retryable instead of permanent, so the existing same-run startup retry
and its three-attempt budget apply. The executor passes the same
fresh-bootstrap condition it already uses for `bootstrap`.
- Reattachment, warm session recovery and retained transitions do not
pass `bootstrap`, so they keep the hold.
- `doc/native-workspace-finalization-recovery.md`: describe the
fresh-launch retry.
- Tests: the disposition retries attempts 1 and 2 of a fresh launch and
stops at attempt 3, and the existing reattachment hold test is
unchanged. An integration test shows `bootstrap_retry` for a fresh
launch with no checkpoint, and `ambiguous_state` without `bootstrap`.

## Verification

- `npx vitest run` on the new cases: pass. Both fail without the source
change.
- `npx vitest run` on the native session executor, controller recovery,
remote provider pack, online source, status arbiter corpus, continuation
and startup/retry suites: 2203 pass, 8 skipped.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low. Only a fresh launch with no checkpoint and no provider event
changes. A deterministic verification outage now costs up to two more
startup attempts, with backoff, before the run is held.
- A completed verification that fails
(`runner_remote_provider_artifact_incompatible`) is unchanged.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 10:40:31 -08:00
DottaandClaude Opus 5.5 f0bd99ba8a fix(recovery): wait out a sandbox the provider could not start, with no Retry (#22194)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - Sometimes the provider creates the sandbox but cannot start it.
Daytona then puts the sandbox in its `error` state
> - The Daytona plugin deletes that failed sandbox and rethrows the
provider error. The error has no HTTP status and no errno, so the host
cannot tell that it is a provider-side failure
> - The run fails with `setup_failed`, and the task rests on "Retry when
ready". No agent work had started, and a fresh sandbox works, so a
person must click Retry for something the system can do itself
> - Recovery must just work, and the target for runs that need Retry is
0%
> - This pull request makes the plugin report a removed sandbox that the
provider could not start as structured evidence (the step and the
provider state). The host classifies it as a transient provider failure.
The task then uses the existing bounded retries and the provider-outage
wait from #22139
> - The benefit is that a provider start failure delays the task by
seconds instead of stranding it on Retry

## Linked Issues or Issue Description

Refs #22139 (the provider-outage wait lane that this failure now
reaches). Refs #22128 (setup failure after work; this failure happens
before any work, so that path did not apply).

**What happened?**
In a restart-chaos wave with graceful release swaps on 2026-11-18, two
tasks started a run at 14:51:15Z and 14:51:24Z. Both runs failed at
14:51:53Z with `setup_failed` and `failureCause: lease_acquire_failed`.
The cause was `Sandbox <id> failed to start with status: error, error
reason: null` from the Daytona SDK, for two different sandboxes at the
same moment. Both tasks rested on Retry. A Retry click started each task
again, and both tasks completed.

**Expected behavior**
A sandbox that the provider cannot start is a provider failure before
any agent work. Paperclip deletes it and tries again with a new sandbox,
with no Retry. If the provider keeps failing, the task waits for the
provider without a Retry, and a person gets a working Retry only after
the 30-minute ceiling.

**Steps to reproduce**
1. Make the Daytona create call fail while the new sandbox is in the
provider's `error` state.
2. Assign a task to an agent that uses that environment.
3. The run fails with `setup_failed`, and the task rests on Retry.

## What Changed

- Plugin SDK (`environment-creation-cleanup.ts`,
`environment-acquire-error.ts`, `index.ts`): new
`PluginEnvironmentAcquisitionError`. It carries the acquisition
diagnostic (step, elapsed time, budget, provider state) for a failed
allocation that the provider already removed, with no cleanup envelope.
The acquire RPC error data (`paperclip/environment-acquire-error/v1`)
now carries this diagnostic. `readEnvironmentAcquisitionDiagnostic`
reads it from that envelope. The existing cleanup envelope path is
unchanged. No provider text crosses the RPC.
- Daytona plugin (`plugin.ts`): `destroyFailedCreation` returns the
state of the sandbox that it deleted. When a create fails without an
HTTP status and the deleted sandbox was in the `error` state, the plugin
throws `PluginEnvironmentAcquisitionError` with `providerState:
"error"`. A `build_failed` sandbox (a deterministic snapshot build
failure) and any other state keep the original error.
- Server (`environment-runtime.ts`): the acquire path records the
diagnostic for this run also when the error has no cleanup envelope. A
cleanup envelope for another run, company or environment still records
nothing.
- Server (`environment-provider-unavailable-error.ts`): an acquisition
diagnostic with `providerState: "error"` is the transient reason
`provider_start_failed` ("the provider could not start the sandbox"). It
becomes `EnvironmentProviderUnavailableError`, so the run uses the two
bounded retries and then the provider-outage wait lane (one probe per
environment, 30-minute ceiling).
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed pins for
the changed SDK sources (generated by `node
tests/runner-e2e/source-sdk-public-inputs.mjs <sha> --write`).

Module boundaries: the change to the shared SDK contract is additive.
The Daytona plugin owns the provider state. The host reads only the
closed diagnostic fields. A plugin that does not use the new class
behaves as before.

## Verification

- `packages/plugins/sandbox-providers/daytona`: `npx vitest run
src/plugin.test.ts -t "failed sandbox creation cleanup"` (37 passed).
New: "reports a sandbox that failed to start as a removed allocation
with its observed step" (red before the plugin change, green after) and
"keeps the create error when the removed sandbox was build_failed /
started".
- `packages/plugins/sdk`: `npx vitest run
tests/environment-acquire-error.test.ts
tests/environment-creation-cleanup.test.ts` (50 passed). New: the
diagnostic crosses the real worker RPC without provider text, and forged
or unstructured envelopes give no diagnostic.
- `server`: `npx vitest run
src/services/environment-provider-unavailable-error.test.ts
src/__tests__/environment-acquisition-diagnostics.test.ts` (74 passed).
New: the reason is `provider_start_failed` with a transient provider
failure, and a diagnostic that was not recorded for this run is not
retried.
- `server`: `npx vitest run src/__tests__/environment-runtime.test.ts -t
"provider could not start|retains uncertain plugin creation"` (12
passed). New: "records the step of a sandbox the provider could not
start and removed" for a worker error (red before the runtime change,
green after) and for a forged error (not recorded).
- `server`: `npx vitest run
src/services/__tests__/run-failure-diagnostics.test.ts
src/__tests__/heartbeat-process-recovery.test.ts -t
"provider|acquisition|diagnostic"` (167 passed).
- Typecheck: `server` `tsc --noEmit`, `pnpm run typecheck` in
`packages/plugins/sdk` and in the Daytona plugin all pass.
- `pnpm verify` exits 2 (the SDK input registry is a global CI input).
Required CI supplies the broad coverage.

## Risks

- **Automatic retries for a deterministic start failure.** If a provider
image or configuration made every sandbox go to `error`, the task now
waits in the provider-outage lane instead of resting on Retry at once.
The lane uses one probe per environment on a 1, 2, then 5 minute
backoff, and it hands the task to a person with a working Retry after 30
minutes. The aggregate automatic-run brake is unchanged. `build_failed`
is not retried by this change.
- **Shared SDK contract.** The new field is additive. An older host
ignores it, and the run fails as before.
- No database migration. No UI change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used in
Claude Code with tool use (shell, file edits) and extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:47:33 -06:00
DottaandClaude Opus 5.5 adb2bcf896 fix(connectors): activate plugin connectors with agent grants and no sign-in (#22201)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins can now add connector types (#22004) and serve their tools
with the `plugin` transport (#22019)
> - A plugin connector method can declare `grantKinds: ["agent"]` with
`auth: "none"` or `"api_key"`
> - Such a connector cannot reach its agent: the connect form cannot
name the agent, connect creates no agent grant, and Paperclip Runner
runs never list plugin-transport tools
> - The product rule is that a plugin connector behaves exactly like any
existing connector
> - This pull request fixes the connect form, creates the agent grant at
connect, and lets runner runs list plugin-transport tools
> - The benefit is that an operator can activate a per-agent plugin
connector, and only the chosen agent gets its tools

## Linked Issues or Issue Description

Refs #22004
Refs #22019

**Subsystem affected**
Connectors (connect wizard, tool access service), Paperclip Runner
assigned-tool gateway.

**Problem or motivation**
A plugin connector with agent grants and no sign-in cannot be activated:
1. Connect form: it says "No sign-in needed, available to all agents".
"Change" shows "No identity required" and "Which agent owns this GitHub
account?", with no agent picker. Connect sends `grantKind: "agent"`
without `subjectAgentId`, and the server rejects it with a validation
error. The "Finish setup" resume form has the same problem.
2. Grant: connect with `subjectAgentId` saves a `per_agent` connection
with zero grants. Finish adds the agent install, but there is still no
grant. Only the OAuth callback writes agent grants, and these methods
have no OAuth step. For `api_key` methods connect fails, because the
health check finds no agent grant, and the key has nowhere to go.
3. Runner: a Paperclip Runner run builds its assigned-tool gateway from
an allowlist of transports (`mcp_remote`, `local_stdio`, REST, Browser
Use, GitHub bot, voice). `plugin` is not in it, so the run sees no
plugin-transport tools, even when the install, profile and grant allow
them.

**Proposed solution**
Create the agent grant at connect for curated methods without sign-in.
Add `plugin` to the runner assignment and its digest. Show a one-agent
picker with connector-neutral copy.

**Expected behavior**
Connecting alone grants no tools. Finish with agent A gives only agent A
the tools, in gateway discovery and in a runner run. The connection has
exactly one active agent grant, for A.

## What Changed

- `server/src/services/tool-access.ts`: `connectGalleryApp` writes an
active agent grant for `subjectAgentId` when a curated method with
`auth` `none` or `api_key` uses `grantKind: "agent"`. The API key goes
on that grant, not on the connection. A resume without `subjectAgentId`
keeps the agent of the existing grant and keeps its saved key. A second
agent on the same connection is a 409 conflict. The grant write is
audited and joins the existing revive rollback.
- Finish does not create agent grants. Finish carries no credential, and
it can list many agents while a per-agent connection has one identity. A
credential-less grant would also claim a connected OAuth identity that
does not exist. The grant comes from connect (or the OAuth callback).
Finish creates only the install.
- `server/src/services/heartbeat/run-preparation.ts` and
`native-runtime/runtime-context.ts`: include the `plugin` transport in
the runner assignment and in its digest (the two lists must match).
- `ui/src/features/connections/ConnectionSetupFlow.tsx`: for an agent
grant, show the agent picker (exactly one agent), keep the install
choice on "specific", open the "Change" panel and disable Connect until
one agent is chosen, and send `subjectAgentId`. Copy is neutral ("Which
agent uses this connection?"); the GitHub copy stays for GitHub. The
summary no longer says "available to all agents" for an agent grant.
Finish sends the chosen agent, never `all_agents`, for an agent grant.
- `doc/connections/CONNECTOR-PLAYBOOK.md`: document agent grants for
methods without sign-in.

## Verification

Each new test failed before its fix and passes after it:
- `server/src/__tests__/plugin-connector-agent-grant.test.ts` (embedded
Postgres, real routes and gateway):
- `auth: "none"` + agent grants: after connect, no agent has tools;
after finish with agent A, A's discovery lists the tools and B's lists
none; exactly one active agent grant, for A. (Red: zero grants.)
- `auth: "api_key"` + agent grants: same, the key is on A's grant, and a
tool call receives it. (Red: connect returned 422.)
  - A second agent on the same connection is a 409. (Red: 201.)
- Resume of an API-key draft without the key keeps the key on the grant.
- Paperclip Runner session: `buildPaperclipRuntimeMcpServers` +
`createAssignedMcpTools` list the installed plugin tools for A, and
search by the manifest tool name (`update_record`) finds the gateway
tool. B gets none. (Red: no tools.)
- Regression: an organization-grant method keeps one default
organization grant, and both agents get the tools with `all_agents`.
- `ui/src/pages/apps/AppsConnect.test.tsx`: for `none` and `api_key`
agent-grant plugin methods, the form shows the picker, has no GitHub or
"available to all agents" copy, blocks Connect until an agent is chosen,
and sends `subjectAgentId`. (Red: "available to all agents" shown.)
- Regression suites: `plugin-connectors-plugin-transport`,
`plugin-connectors-circleback-parity`, `heartbeat-runtime-mcp-servers`,
`run-preparation`, `tool-access-service`,
`tool-access-oauth-protocol.integration`, `generic-mcp-connection`
(OAuth, GitHub per-agent and organization flows) pass. UI:
`src/features/connections`, `src/pages/apps`, `App.deep-link-routing`
pass (66 files, 1103 tests).
- `tsc --noEmit` passes for `server` and `ui` source. `pnpm
check:module-boundaries` passes.

## Risks

- Low. The grant change applies only to curated methods with agent
grants and no OAuth. Organization, personal and OAuth flows keep their
code paths.
- The runner change makes plugin-transport tools visible to runner runs.
Access still needs an install, a profile and a grant, and every call is
re-authorized by the gateway.
- Connections created before this fix have no agent grant. Connecting
again (or "Finish setup") with the agent creates it.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:25:25 -06:00
DottaandClaude Opus 5.5 a7ea2a799a fix(ephemeral): save retained runtime skill bundles at startup and never strand a volume-free server on lost ones (#22190)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A volume-free server keeps no local volume. It must find every file
it needs in durable storage.
> - Before a volume-free server starts, a data gate refuses to start if
records still point at files on a local disk.
> - Old native runs can keep runtime skill bundles that a release before
#19729 wrote only to its own disk. The gate reports these as
`runtime_skill_assets_require_migration`.
> - On a volume-free server those files are not on the disk, and no
volume can be attached again. So the refusal repeats on every restart,
and nothing can clear it. A staging stack crash-looped like this and
stayed down.
> - This pull request makes the server copy every retained bundle that
is still on its disk into durable storage, and start over the bundles
that are gone, the same way #20990 handles harness backups.
> - The benefit is that a volume-free stack can always start again. No
one has to repair it by hand.

## Linked Issues or Issue Description

Refs #19675, #19729, #20990

**What happened?**

A volume-free staging stack ran for twelve days on a release that kept
runtime skill bundles only on its container disk. When that container
stopped, every newer release crash-looped at boot with `Volume-free
server data is not eligible: runtime_skill_assets_require_migration`.
The old container was gone, so the bundles were gone. The deploy could
not roll back, and a retry could not help.

**Expected behavior**

A volume-free server starts. It saves each retained bundle that is still
on its disk to durable storage. It starts over a bundle that is not on
its disk when the volume-free owner marker proves that no volume is
mounted, and restart recovery owns those runs.

**Steps to reproduce**

1. Retain a native run with an admitted runtime skill bundle that has no
version ID and no durable preservation receipt.
2. Remove the bundle from the instance root.
3. Start the server with `PAPERCLIP_EPHEMERAL_SERVER=true` and the
volume-free owner marker present.
4. Startup fails with `runtime_skill_assets_require_migration`, on every
restart.

## What Changed

- `runtime-skill-preservation.ts`:
- The obligation scan is one generator.
`hasLocalRuntimeSkillObligations` uses it.
- New `preserveLocalRuntimeSkillObligations` copies each obligated run's
bundles with `preserveRuntimeSkillAssets` and returns `{ bundles,
unsaved }`.
- A missing source is decided at its reader:
`readLocalManifest`/`readSource` throw `RuntimeSkillSourceAbsent` (other
instance root, or a missing directory, manifest or listed file). Only
that error marks a run unsaved. Any other failure, including an `ENOENT`
from storage or cache work, propagates.
- `preserveRuntimeSkillAssets` no longer aborts on an invalid or
non-durable existing receipt. It copies the local bytes again and
replaces the receipt.
- One durability helper (`preservedAssetRowDurable` for metadata,
`preservedAssetDurable` with a metadata or object read) replaces the
copies in the scan, staging, locking, preservation and rehydration.
- `runtime-skill-preservation-reference.ts`: the scan matches only null
pins, which is exactly what preservation copies. A non-null pin is a
version pin and cannot become an obligation that nothing can settle.
- `file-manifest-cache.ts`: new `inspectFileManifest` returns
`verified`, `missing` or `changed` from the same walk.
`verifyFileManifest` wraps it.
- `ephemeral-server-contract.ts`: the gate saves the bundles still on
the disk. If none stay unsaved, the blocker clears. Unsaved runs are
waived only when the volume-free owner marker is present, with a
warning. A save failure still refuses. The decision uses the copy pass
result, not another scan. One exported blocker constant.
- `ephemeral-migration-data.ts`: offline enrollment accepts the blocker
and copies the bundles from the attached volume, in a phase after the
package phases. Any unsaved run still fails the enrollment.
- `index.ts`: comment update only.
- Tests: `ephemeral-server-runtime-skill-startup.test.ts` (real scan,
real copy, embedded Postgres) and an S3-gated case in
`ephemeral-migration.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/ephemeral-server-runtime-skill-startup.test.ts`
- Red on `origin/master`: 2 of 3 fail with the production error
`Volume-free server data is not eligible:
runtime_skill_assets_require_migration`.
- Green with this change: all cases pass (lost bundle, saved bundle, no
marker, mixed present and lost, one missing listed file, non-version
pin, mixed pins, non-durable receipt recapture, storage failure, storage
ENOENT, unsaved set).
- `cd server && npx vitest run
src/__tests__/ephemeral-server-startup-run-state.test.ts
src/__tests__/runtime-skill-preservation.test.ts` pass (existing gate
and preservation behavior unchanged).
- `cd server && npx tsc --noEmit -p .` passes.
- `ephemeral-migration.test.ts` needs `PAPERCLIP_TEST_S3_ENDPOINT`. It
typechecks, but it was not run here.

## Risks

- A run whose bundle is lost now starts under restart recovery instead
of blocking the whole server. Recovery of that one run can fail. Before,
the whole server stayed down, and the bundle was lost anyway.
- Startup does more work only when unsaved legacy bundles exist. A
current release saves bundles before admission (#19729), so new runs
never add this work.
- No schema change. No API change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 11:00:20 -06:00
DottaandClaude Opus 5.5 3c460f20cc feat(analytics): ask EU, UK and Swiss visitors before Cloud browser analytics (#22186)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud can send browser product analytics and session
replays to PostHog. Self-hosted installs never do this.
> - Today, browser analytics starts for every Cloud visitor after the
page settles.
> - Visitors in the EU, the EEA, the UK and Switzerland must agree
before analytics cookies and session recordings start.
> - This pull request adds a consent step for these visitors and a
switch in the user's profile settings.
> - The benefit is that Cloud analytics follows the owner decision: ask
in Europe, keep the current behavior elsewhere, and let every user
change the choice.

## Linked Issues or Issue Description

**Subsystem affected**
Paperclip Cloud browser product analytics (`ui/src/lib/analytics`,
`ProductAnalyticsGate`, the session route, and Profile settings).

**Problem or motivation**
Browser analytics and session replay start for every Cloud visitor
without a choice. EU, EEA, UK and Swiss visitors must consent first. The
privacy policy draft also says that users can change the choice in
settings. No such setting exists.

**Proposed solution**
Ask visitors in the EU, the EEA, the UK and Switzerland with a small
bottom banner (Accept or Decline). Do not load posthog-js before they
accept. Treat Do Not Track and Global Privacy Control as a decline for
these visitors. Keep the choice in `localStorage`. Add a "Product
analytics" switch to Profile settings for every user when browser
analytics is on.

**Alternatives considered**
A server-side country from a trusted edge header. Paperclip Cloud has no
such header today: Cloudflare records are DNS-only, so no `CF-IPCountry`
arrives, and the Railway edge adds no country header. This PR supports
an optional header (`PAPERCLIP_POSTHOG_COUNTRY_HEADER`) for a future
edge, and uses the browser time zone until then.

## What Changed

- `packages/shared`: `browserAnalytics` has a new optional
`consentRequired: boolean | null`.
- `packages/shared/src/analytics-consent.ts` (new): the one region
policy. `countryRequiresAnalyticsConsent` (EU and EEA with outermost
regions and Svalbard, GB with the Crown Dependencies and Gibraltar, CH)
and `timeZoneRequiresAnalyticsConsent` (`Europe/*`, EU, EEA and UK zones
outside Europe, and unknown zones such as `UTC` or `Etc/*`). The server
and the UI both use it.
- `server`: `resolveBrowserAnalyticsConfig` sets `consentRequired` from
the header that `PAPERCLIP_POSTHOG_COUNTRY_HEADER` names. No header or
an unknown code (`XX`) gives `null`.
- `ui/src/lib/analytics/consent.ts` (new): decides `granted`, `denied`
or `ask`. Order: for a visitor who must consent, Do Not Track or Global
Privacy Control gives `denied` (also over a stored Accept); then the
choice made on this page; then the stored choice; then the region
(server value, else browser time zone). The choice is in `localStorage`
key `paperclip.analyticsConsent`. All storage access is in `try/catch`.
A cross-tab `storage` event updates the page.
- `ProductAnalyticsGate`: one effect calls `syncProductAnalytics(config,
consent)`. Shows `AnalyticsConsentBanner` when consent is `ask`. Accept
loads posthog-js at once, without a reload.
- `product-analytics.ts`: new `syncProductAnalytics`. It starts, opts
in, opts out (`opt_out_capturing()` and `stopSessionRecording()`) or
cancels a pending load. Any consent other than `granted` stops
analytics. If consent is withdrawn while the script downloads, `init()`
does not run until a later opt-in.
- `AnalyticsConsentBanner` (new): a labelled bottom region with plain
copy, a privacy policy link, and Decline and Accept buttons (shared
`Button`). The copy says that recordings hide what you type and your
messages, documents and connected accounts, but may show page names like
task titles. It does not block the page. On mobile it sits above the
bottom navigation.
- `ProductAnalyticsSetting` (new) in Profile settings: a "Product
analytics" switch (shared `ToggleSwitch`). It shows only when browser
analytics is on for the instance. When GPC or DNT blocks analytics, it
is off and disabled and says why.
- `doc/cloud-browser-analytics.md`: new Consent section (signal,
behavior table, storage key, how to reset) and the new variable.
- `tests/runner-e2e/source-sdk-public-inputs.json`: refreshed for the
changed shared validator.

## Verification

- Red/green: the 7 new `ProductAnalyticsGate` consent tests fail against
the master gate (7 failed, 3 passed). The 5 review-fix tests (failed
write with working read, cross-tab change, decline during script
download, granted -> ask from another tab, granted -> ask from a session
refresh) fail on the first commit (5 failed, 16 passed). All pass now.
- `cd ui && npx vitest run src/lib/analytics
src/components/ProductAnalyticsGate.test.tsx
src/components/AnalyticsConsentBanner.test.tsx
src/components/ProductAnalyticsSetting.test.tsx
src/pages/ProfileSettings.test.tsx`: 267 tests pass.
- `cd server && npx vitest run
src/__tests__/cloud-posthog-browser.test.ts
src/__tests__/auth-routes.test.ts
src/__tests__/auth-session-route.test.ts`: 87 tests pass.
`packages/shared` `analytics-consent.test.ts` passes. `tsc --noEmit` for
`ui`, `server` and `packages/shared` shows no new errors.
- `pnpm verify` exits 2 (a doc is a global CI input). Required CI gives
the broad coverage.
- Real check in headless Chromium. I ran a local Cloud-managed instance
(external local PostgreSQL, `PAPERCLIP_CLOUD_TENANT_SERVER_TOKEN`,
`PAPERCLIP_POSTHOG_BROWSER_ENABLED=1`). `PAPERCLIP_POSTHOG_HOST` pointed
at a local capture server, not PostHog. The capture server served a
small stand-in for `array.js` that sets a `ph_*` cookie and
`localStorage` key and posts `/e/` events. I set the visitor region with
the browser time zone:

| Visitor | Result |
| --- | --- |
| `Europe/Berlin`, no choice, 18 s wait | Banner shows. 0 `/_ph`
requests. No `ph_*` cookie or storage key. |
| Then Accept (keyboard Enter) | Banner closes. `GET
/_ph/static/array.js` and `POST /_ph/e/` in 3 s. Key is `granted`. After
reload: no banner, events continue. |
| `Europe/London`, Decline, wait, reload, wait | No banner. 0 `/_ph`
requests. No `ph_*` cookie or storage key. Key is `denied`. |
| `Europe/Zurich` with Global Privacy Control | No banner. 0 `/_ph`
requests. No `ph_*` cookie or storage key. |
| `America/New_York` | No banner. posthog-js loads after the page
settles, as before. |
| Profile settings switch (US) | Shows on. Off: client is opted out at
once. On: opt-in, events continue. |

- I ran the browser check again after the review fixes, with the same
results.
- Screenshots (banner light, banner dark, mobile 375 px light and dark,
settings switch): see the PR comment.
- Cold load (local dev server, fresh browser context per run, median of
7, ms):

| Build | Time zone | FCP | DOMContentLoaded | load |
| --- | --- | --- | --- | --- |
| master | New York | 728 / 724 | 687 / 684 | 689 / 686 |
| this PR | New York | 748 / 724 | 708 / 684 | 710 / 686 |
| master | Berlin | 740 | 694 | 697 |
| this PR | Berlin | 756 / 724 | 716 / 684 | 718 / 686 |

The second round gives the same medians for master and this PR. The
first-round difference is noise. The consent code adds no request and no
load-time wait.

## Risks

- Region by time zone is approximate. It asks some visitors outside the
EU (for example `Europe/Istanbul`, or `UTC` browsers). This is the safe
direction.
- A forged country header (only when `PAPERCLIP_POSTHOG_COUNTRY_HEADER`
is set) changes only the default for that visitor's own browser.
- The choice is per browser. A user on a new browser sees the banner
again (in consent regions).
- Visitors in consent regions now send no browser analytics until they
accept. Expect fewer EU/UK sessions in PostHog.
- Self-hosted installs: no change. They get no `browserAnalytics`
config, so no banner, no switch and no load.

## Model Used

- Claude Opus 5.5 (claude-opus-5-5), Anthropic, used through Claude Code
with tool use (shell, file edits, headless browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:52:20 -06:00
DottaandPaperclip 875e4d375e fix(auth): preserve identity across connection request handoffs
Continue accepted connection requests as the verified recipient and retain their original audit parent. Keep declined and automatically closed requests on the source identity. Preserve the latest accepted retry context, ordinary preparation retries, and saved approval origins.

Verification: 209 focused local tests passed. Final-head CI passed, Greptile and both Paperclip reviewers rated the change 5/5, and all review threads are resolved.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-18 10:45:55 -06:00
DottaandClaude Opus 5.5 2a43fe3e34 feat(telemetry): add company_id to Cloud direct events from background work (#22183)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Cloud sends server telemetry straight to PostHog
(`server/src/services/posthog-direct/`)
> - Request events get `company_id` from the request context. Background
events have no request.
> - So `agent.task_run`, `routine.run`, `connection.invoked`, and other
background events arrive with `stack_id` but without `company_id`. On
staging, 66 of 66 `agent.task_run` events had a null `company_id`.
> - Without `company_id`, product analytics cannot split agent and
routine activity by company.
> - This pull request lets each background emit site name the company of
the entity that it reports on. Only the direct PostHog transport reads
that company.
> - The benefit is that Cloud background events carry `company_id`, and
the self-hosted collector output does not change.

## Linked Issues or Issue Description

Refs #22006

**Subsystem affected**
Server telemetry: Cloud direct delivery to PostHog
(`server/src/services/posthog-direct/`) and the server emit sites for
background events.

**Problem or motivation**
Cloud direct events from background work have no `company_id`. Heartbeat
runs, scheduled routines, connection invocations, and interaction
updates do not run inside an HTTP request, so the request context gives
no company. On paperclip.staging, all 66 sampled `agent.task_run` events
had `company_id` null.

**Proposed solution**
Add a small AsyncLocalStorage scope, `withTelemetryCompany(companyId,
fn)`, in `server/src/telemetry-company-scope.ts`. Each background emit
site wraps its typed `track*` call with the company id of the row that
it already has. The direct transport reads the scope at track time,
through the existing `captureEventContext()` seam. A live request
company still wins. The collector path never reads the scope.

**Alternatives considered**
1. An optional context argument on the shared `track*` helpers. This
changes the public helpers in `@paperclipai/shared` for a Cloud-only
value, so I did not use it.
2. A scope around each whole work unit (heartbeat run, routine run,
invocation). The emit calls are fire-and-forget and are spread across
many modules, so it is easy to miss one or to leak a company into other
work. A scope at the emit site is narrow and explicit.
3. Let the transport look up the company in the database. The transport
must not query the database.

**Roadmap alignment**
This completes the Cloud direct PostHog work from #22006. It does not
change the telemetry contract.

## What Changed

- Add `server/src/telemetry-company-scope.ts`. It has
`withTelemetryCompany(companyId, fn)` and `scopedTelemetryCompanyId()`.
With no company id, it calls `fn` directly.
- `capturePostHogEventContext()` uses the scoped company when no live
request gives a company. The request company, user, and session behave
as before.
- Wrap these emit sites with the company of their entity:
- `agent.task_run` (`emitAgentTaskRun`, `emitAgentTaskRunById`;
`companyId` is now part of the run that the function reads). This covers
heartbeat, native runtime, recovery, run-dispatch, and watchdog callers.
- `routine.run` (`routineService` dispatch, which covers scheduler,
webhook, and manual runs)
  - `agent.first_heartbeat` (heartbeat agent status)
  - `connection.created`, `connection.updated`, `connection.invoked`
  - `interaction.created`, `interaction.resolved`
- `agent.task_completed` (issue route; the issue company is the fallback
when the request gives no company)
- Request-only events (`project.created`, `goal.created`,
`agent.created`, `routine.created`, `skill.imported`) keep the request
context. `server.started` and `error_handler.crash` are not
company-scoped. Dynamic plugin and `question_response.delivery` events
are not in the contract, so the direct path drops them.
- Update the "Cloud Direct Delivery" section of
`packages/shared/src/telemetry/README.md`.

Design note on module boundaries: the scope lives beside
`server/src/telemetry.ts`, not inside `posthog-direct/`. Emit sites do
not import the transport, and the transport does not know which services
exist. The shared telemetry package does not change.

## Verification

- Red/green: with the source changes removed, these 4 new tests fail.
With the changes, they pass.
- `agent-task-run-telemetry.test.ts`: `emitAgentTaskRun` and
`emitAgentTaskRunById` track in the run company scope.
- `routine-run-telemetry.test.ts`: a routine run tracks in the routine
company scope.
- `posthog-direct-telemetry.test.ts`: on Cloud, background
`agent.task_run` and `routine.run` (also after an `await`) carry
`company_id`, and an unscoped event does not.
- `posthog-direct-telemetry.test.ts`: a live request company wins over
the scope, and the scope fills the gap when the request has no company.
- Red/green per emit family: the tests for `connection.created`,
`connection.updated`, `connection.invoked`, `interaction.created`,
`interaction.resolved`, `agent.first_heartbeat`, and
`agent.task_completed` assert the company scope at track time. With the
emit-site wrappers removed, all 7 assertions fail.
- New guard test: on self-hosted, a scoped `agent.task_run` and
`routine.run` produce a byte-identical collector envelope with no
`company` text.
- `cd server && npx vitest run` on 16 telemetry, PostHog, routine,
heartbeat status, interaction, and connector test files: 408 passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- Local check with a real `posthog-node` client sent to a local capture
server (never real PostHog), embedded Postgres, and a real
`routineService.tickScheduledTriggers()` tick plus `emitAgentTaskRun`:
- Cloud mode: the captured `routine.run` (`source: schedule`) and
`agent.task_run` both had the seeded `company_id`, plus `stack_id` and
`environment`. `server.started` had no `company_id`.
- Self-hosted mode: the collector body from this branch and from
`origin/master` are identical after I normalize random ids and
timestamps.
- I did not boot a full Cloud-managed server, because it needs S3 and an
external Postgres. The check above uses the same server modules and the
real PostHog client.

## Risks

- Privacy review: this is a Telemetry change under
`rules/data-paths.mdc`, so I request a privacy review. Background Cloud
events now send the real Paperclip `company_id` to PostHog. #22006
already approved `company_id` as a Cloud direct property for request
events. This PR only fills it for background events. It adds no new
property, dimension, or event, so the generated contract does not
change. `packages/shared/src/telemetry/README.md` is updated in this PR.
- Low runtime risk. The self-hosted collector path does not read the
scope, and a test proves its output does not change.
- On Cloud, more events now have `company_id`. No event loses a field.
- If a request with a company tracks an event for an entity in a
different company, the request company wins. This is the same as before
this change.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, and test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 10:34:39 -06:00
DottaandPaperclip da306ac2da fix(routines): preserve the manual caller during execution dispatch (#22177)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Routines let people repeat work and select an agent for each manual
run.
> - The person who starts a manual run must be responsible for that run.
> - Routine dispatch queues execution before its transaction commits.
> - The execution queue cannot yet read the new routine-run record. It
can select the routine owner's account instead.
> - This pull request carries the selected responsible user into the
assignment wake context.
> - The agent can then use the manual caller's account through the
existing access checks.

## Linked Issues or Issue Description

**What happened?**

A person used Run now on another person's routine and selected their own
agent. The task belonged to the caller, but the execution used the
routine owner's identity. Account access then failed.

**Expected behavior**

The task, routine run, and agent execution must use the person who
starts the manual run. The saved routine must keep its owner.

**Steps to reproduce**

1. Create a routine as one person.
2. Use Run now as a second person and select that person's agent.
3. Compare the responsible users on the task and the queued execution.

**Paperclip version or commit**

Reproduced on master at `2d67b060260f` with a real PostgreSQL database
and the heartbeat service.

**Deployment mode**

Reported on a hosted instance. Reproduced in a source checkout.

Related ownership changes: #19128 and #19222. The issue search found no
duplicate for this transaction visibility failure.

## What Changed

- Add an optional responsible user to the assignment wake helper. Other
callers keep their current behavior.
- Pass the user already selected by routine dispatch into the wake
context. Keep the dispatch lock through queue admission.
- Add a regression that runs another person's routine with the caller's
selected agent through the real heartbeat service. Check both queued and
completed execution identity.
- Keep the saved routine owner and the existing account access checks.

## Verification

- Before the fix, the new regression failed: the execution used the
routine owner instead of the caller.
- The full heartbeat identity suite passed: 19 tests, including the new
regression. The routine service and route suites passed: 130 tests.
- The first post-fix run found that the new test caller had no company
membership. The fixture now adds an active member, and the full identity
suite passes.
- Local scope: Node.js 25.9.0 on Linux, PostgreSQL-backed tests, head
`93fa698ebae875d7f7357af1d7854b48aef9f824`.
- `pnpm verify` selected 2,923 of 3,786 files. Stopped the broad local
run under the repository guidance to use focused checks and required CI.
Its first 12 tests passed, but the interrupted run is not a pass.
Earlier selections were invalidated by the fixture edit.
- Greptile: 5/5 on head `93fa698ebae875d7f7357af1d7854b48aef9f824`, with
no review threads. Required CI passed at this head: [CI run 44325396087,
attempt
2](https://github.com/paperclipai/paperclip/actions/runs/44325396087/attempts/2).
All 61 checks are complete: 55 success, 4 skipped, and 2 neutral. Build,
typecheck, server tests, Runner tests, and browser tests passed. Two
server shards were retried after their runners shut down; both passed.
- Worktree preflight installed dependencies and built the shared package
and plugin SDK. Its Runner build could not run because Cargo is absent.
The focused tests use the source Runner alias. CI supplied build and
typecheck coverage; both passed.

## Risks

- Low risk. The change carries a server-selected identity through an
existing wake context. It adds no database or public API fields.
- The optional helper field affects routine dispatch only. Scheduled
runs keep the user selected from the saved routine revision.

## Model Used

OpenAI Codex, GPT-6, with reasoning, repository tools, and code
execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-18 09:03:58 -06:00
DottaandClaude Opus 5.5 1c556d5340 fix(native): never rest a task blocked on the agent's own blocker (#22140)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends with one semantic report. A `paperclip_block`
report names a blocker, its owner and the unblock action
> - The status arbiter binds a task-wide blocker. When the owner is not
a person, it binds the blocker to the same agent and wakes that agent
> - The dispatcher refuses those self-wakes
(`chat_control_recovery_proof_unresolved`), because only continuation
and retry effects are proof for a native wake. The task then rests
blocked with no owner, no wake and no Retry
> - #21959 continues a system blocker in a fresh session, but only for
the reason codes `environment_unavailable` and `tool_unavailable`. A
restart-chaos rerun showed the same dead end with the reason code
`tool_output_failure`
> - This pull request makes sure that a blocker that the agent reports
for itself always has a next owner: the automatic environment lane, one
bounded self-continuation, or a person with a Retry that works
> - The benefit is that no task rests blocked with nobody to act, and no
path wakes the agent in a loop

## Linked Issues or Issue Description

Refs #21959

**What happened?**

A native OpenCode run (DeepSeek) said that every tool call returned the
text "cryptic". It reported `paperclip_block` with owner `{kind:
system}`, scope `task_wide`, reason code `tool_output_failure`, and an
unblock action about the tool-output failure. The run succeeded with
disposition `blocked`. The arbiter bound the blocker to the same agent
(`task_wide_blocker_bound`). The `notify-owner` wake coalesced, and the
`blocker-owner` wake was skipped as
`chat_control_recovery_proof_unresolved`. The task rested blocked
(`needs_attention`, 0 blockers), with no Retry and no wake. An earlier
rerun showed the same shape with reason code `environment_unavailable`,
which #21959 fixed for those two codes only.

**Expected behavior**

A blocker that the agent reports for its own tools or environment is a
harness problem. Paperclip continues it automatically, with a bound. A
blocker that the agent says it owns gets one fresh session. A blocker
that needs a person goes to a person, with a clear prompt and a Retry
that starts the task again. Nothing is left with nobody.

**Steps to reproduce**

1. Run a native task. Make the agent call `paperclip_block` with owner
kind `system`, scope `task_wide` and a reason code other than
`environment_unavailable` or `tool_unavailable` (for example
`tool_output_failure`).
2. See the decision `task_wide_blocker_bound` with the agent as the
unblock owner.
3. See both owner wakes refused, and the task blocked with no Retry.

**Paperclip version or commit**

`master` at 9e1a85fb46.

## What Changed

- `native-environment-blocker.ts`: a system-owned blocker now uses the
bounded fresh-session environment lane for any reason code, except the
person-only codes `permission_denied`, `credential_missing`,
`policy_restricted` and `budget_exhausted`. An agent-owned blocker with
`environment_unavailable` or `tool_unavailable` also uses it. The lane
is unchanged: at most 2 retries on the shared failure counter, then a
board blocker.
- New `native-agent-blocker.ts` and a new arbiter branch for a task-wide
blocker that is not person-owned, from the current owner's own succeeded
run (not a chat, not a review):
- Owner kind `agent`: one fresh-session continuation
(`native_agent_blocker_continuation`). It uses the shared failure
counter with a maximum of 1, so it never runs after another automatic
retry in the same chain. A paused task does not get it.
- All other cases, and a repeated blocker: the blocker goes to the board
with the agent's unblock action (`native_agent_blocker_needs_person`).
There is no self-wake.
- `native-provider-capacity-retry.ts`, `status-decision-committer.ts`,
`native-run-finalizer.ts`: materialize the continuation through the
existing fresh-session retry lane (`forceFreshSession`, `retryOfRunId`,
30 s delay, refuses a stopped run). Both new reason codes need the
provider-failure owner recheck under the issue lock. The finalizer reads
the pause hold for every blocked result.
- `heartbeat.ts`, `run-dispatch/adapters/postgres.ts`: the continuation
gets the same dispatch ownership checks as the environment retry.
- `task-interruption-presentation.ts`, shared
`TaskInterruptionPresentation.userAction`: a task that is blocked by a
`native_agent_blocker_needs_person` decision on its latest run shows
`userAction: "unblock"` with `canRetry: true`. `ExecutionBlockerNotice`
shows "The agent stopped and needs your help to continue. Read its last
comment, do what it asks, then Retry." with a Retry button. The inbox
label is "The agent needs your help". The assigned-work invariant reads
it as `assignee_declared_block`.
- `blocked-question-answer.ts`: an answer to the agent's open question
still wakes it after the new hand-off.
- Runner `completion-result.ts`: the "Paperclip continues this blocker
automatically" message now matches the server's environment lane.
- I did not change `readChatControlNativeParent`. The fix removes the
self-bound blocker wakes instead of accepting them as proof, so a
self-wake loop cannot form.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-agent-blocker.integration.test.ts`
(new, embedded Postgres, real executor, finalizer and committer). Red
before the fix (3 of 3 fail with `task_wide_blocker_bound`), green
after:
- The t205 shape (system owner, `tool_output_failure`) gets one
fresh-session retry, no owner wakes, and the retry run that reports done
moves the task to `done`.
- An agent-owned blocker continues once. The same blocker in the
continuation goes to the board, with no further retry and no owner
wakes.
- A system blocker with `credential_missing` goes to the board with no
automatic retry.
- An external-owned blocker goes to the board. The task shows
`userAction: "unblock"` and `canRetry: true`. `requestTaskRetry` accepts
the Retry and `resumeTaskRetry` starts exactly one new run of the task.
- `npx vitest run src/services/native-runtime/status-arbiter.test.ts`:
new case for the once-only continuation, the hand-off, the pause hold,
the chat exclusion and the non-owner exclusion.
- `cd ui && npx vitest run
src/components/ExecutionBlockerNotice.test.tsx`: new case for the help
prompt and its Retry.
- `cd packages/paperclip-runner && npx vitest run
src/contracts/completion-result.test.ts`: new cases for the conflict
message.
- Related suites pass (11 files, 523 tests): arbiter corpus,
task-interruption presentation and query, assigned-work invariant,
blocked-question answer, liveness escalation, run-dispatch adapter,
failed-turn continuation, tool-receipt limit and provider capacity.
- `tsc --noEmit` passes for `server`, `packages/shared`,
`packages/paperclip-runner` and `ui` (only the existing generated
Storybook catalog errors). `scripts/check-token-gates.mjs` is clean.
- `pnpm verify` exits 2: the source SDK pin refresh is a global CI
input, so required CI supplies the broad coverage.

## Risks

- More automatic runs: an agent-owned blocker can add 1 run per chain,
and a system blocker with a new reason code can add up to 2 (the
existing lane). Both share the failure-retry counter, so a chain never
gets more than the existing maximum. The company-wide automatic-run
brake admits these runs like any other recovery run. Judge the change by
company-wide automatic run starts per hour and their success rate.
- A blocker that only a person can resolve, but that the agent labels as
its own, now takes one extra run (about 30 s later) before it reaches a
person. This is intended.
- Board-owned blockers now show an attention card and a Retry, where
before they named the agent and showed nothing. A Retry starts a new run
of the task; the person decides when.
- Previous release: no persisted shape changes. The new reason codes are
new values in existing text columns. Decisions saved by the previous
release keep their saved effects.
- Chats keep their current behavior, because the person in the chat
answers directly.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (code search, editing, shell, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 02:43:28 -06:00
DottaandClaude Opus 5.5 a665ed6a1b fix(recovery): wait out a sandbox provider outage with no Retry (#22139)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs in a sandbox environment first gets a lease from
the environment provider (for example Daytona)
> - When the provider refuses the lease with a transient error
(`environment_provider_unavailable`), Paperclip retries twice, after 30
s and 60 s
> - After those two retries the task rests on "Retry when ready". A
provider outage longer than about 90 s therefore needs a person to click
Retry
> - Recovery must just work, and the target for runs that need Retry is
0%. The fix must not multiply automatic runs into a crash loop
> - This pull request adds a provider-outage wait after the fast
retries. The task stays system-owned, one waiter per environment checks
the provider on a capped backoff, and the other waiters continue by
themselves once the provider gives out a lease again. A person gets
Retry only after a 30-minute ceiling
> - The benefit is that a provider outage of up to 30 minutes delays
work instead of stranding it, without extra load on the provider

## Linked Issues or Issue Description

Refs #22087 (it classifies Daytona connection errors as transient, so
they reach this path).

**What happened?**
In a restart-chaos wave on 2026-11-17, the Daytona API host names did
not resolve from 19:40:28Z to 19:52:17Z (182 `ENOTFOUND` errors for the
API and proxy hosts). Tasks that started in that window failed with
`environment_provider_unavailable`. The two bounded retries (30 s, then
60 s) ran inside the outage, so these tasks rested on "Retry when ready"
and needed a person to continue them.

**Expected behavior**
Paperclip continues the task by itself when the provider is reachable
again. The task shows that it waits for the provider. It shows no Retry.
Only a long outage hands the task to a person, and then Retry works.

**Steps to reproduce**
1. Make lease acquisition fail with a transient provider error (HTTP 502
or a connection errno) for longer than about 90 s.
2. Assign a task to an agent that uses that environment.
3. After the second automatic retry fails, the task is blocked with a
Retry button, although the provider becomes available again later.

## What Changed

- `server/src/services/environment-provider-wait.ts` (new): the
`environment_provider_wait` retry lane. The delays are 1 min, 2 min,
then 5 min. The lane stops at 30 minutes after the wait started, with at
most 12 attempts. It also builds the task text "Waiting for
"<environment name>" to come back. Paperclip continues this task
automatically when it does."
- `scheduleBoundedRetryForRun` (`heartbeat/retries.ts`): when a run
fails with `environment_provider_unavailable` after the fast retries are
spent, it schedules the next attempt in the wait lane. The retry context
records the environment and the time the wait started. The time is kept
across attempts. The lane is uncharged
(`execution-recovery-attempt.ts`), so it does not spend the failure
budget. Other lanes drop the wait context.
- The environment orchestrator puts the environment id on
`EnvironmentProviderUnavailableError`, and the setup-failure path saves
it in `resultJson.environmentProvider`.
- Run dispatch (`run-dispatch/adapters/environment-provider-wait.ts`,
new): a due wait stays parked while an earlier waiter for the same
environment is still in line or still acquiring its lease, and no lease
was acquired in that environment since this waiter last failed. Parked
waiters check again every 30 s. This uses database reads only, and it
adds no work to the run start path. When a lease is acquired, the parked
waiters are promoted in arrival order. The automatic-run brake then
meters their starts as before.
- The probe must be able to start. A queued earlier waiter blocks later
waiters for only 60 s after promotion, and a running one only while it
is still acquiring its lease (at most 10 minutes). If the probe's agent
is busy, the next runnable waiter becomes the probe. A waiter past the
30-minute ceiling is never parked, so it makes its last attempt and the
exhaustion path runs.
- The notify-and-update mechanics of parking are in one helper,
`parkScheduledRetryInTx`
(`run-dispatch/adapters/scheduled-retry-park.ts`). The workspace-busy
wait and the provider wait use it and only decide whether to park and
until when.
- UI: `IssueScheduledRetryCard` shows nothing for
`environment_provider_wait` (as for `workspace_busy`). `IssueProperties`
shows "Sandbox: Waiting for the provider" instead of the scheduled-retry
picker. `IssueMonitorBanner` shows a steady "Waiting for the sandbox
provider" with no countdown and no Check now.
- Task presentation: a task in the wait lane is "working", system-owned,
with no Retry, and shows the wait text. After the ceiling the existing
path applies: the task is blocked with the system-owned notice and a
Retry that works. The notice and the provider-failure text now say that
Paperclip retried for 30 minutes.
- The per-task transient cap, the automatic-run brake and the bounded
fast retries are unchanged.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/heartbeat-process-recovery.test.ts -t "transient sandbox
provider refusal"` (embedded Postgres):
- New: "resumes ten tasks by itself when a five-minute provider outage
ends, one probe at a time and within the automatic-run brake". Ten tasks
in one company share one environment. The provider refuses leases for 5
minutes of simulated time (the test shifts all stored timestamps). All
ten tasks complete with no Retry. No task is blocked and no task offers
Retry while it waits. At most one waiter per tick is refused by the
provider. While the brake is open, at most one automatic start happens
per tick. All ten finish within 5 minutes after the outage ends. Red on
master: tasks become `blocked` after the fast retries. Green with this
change.
- New: "lets one waiter per environment probe the provider and releases
the others once a lease is acquired". The second waiter stays parked
while the first one is queued or running without a lease. It is promoted
after a lease is acquired. Red without the park step, green with it.
- Updated: "waits for the provider without a Retry, and offers a working
Retry only after the 30-minute ceiling". After the fast retries the task
is working with the wait text and no Retry. The backoff grows from 1 to
2 minutes. After the ceiling the task is blocked, Retry is offered, and
Retry runs the task.
- Updated: "rests a provider deadline that never clears on a working
Retry, never on a board recovery action". It now reaches the ceiling
first.
- New: "lets the next waiter probe when the first waiter cannot start
because its agent is busy" (red before, green after).
- New: "hands a parked task to a person with a working Retry at the
30-minute ceiling even while the leader stays queued" (red before, green
after).
- `src/services/environment-provider-wait.test.ts`: backoff, ceiling and
text.
- UI: `IssueScheduledRetryCard.test.tsx` ("offers no Retry while the
task waits for the sandbox provider to come back") and
`IssueMonitorBanner.test.tsx` ("shows a steady provider wait with no
countdown"). `pnpm check:token-gates` is clean.
- Also run locally: `heartbeat/retries.test.ts`,
`heartbeat/run-completion.test.ts` (all 163 tests pass; the file-level
`afterAll` teardown timed out on this loaded machine),
`task-interruption-query.test.ts`, `stranded-notice.test.ts`,
`execution-recovery-attempt.test.ts`,
`environment-provider-unavailable-error.test.ts`,
`task-interruption-presentation.test.ts`, the automatic-run brake tests,
and `src/modules/run-dispatch`.
- Server `tsc --noEmit` passes.

## Risks

- **Automatic-run volume:** while a provider is down, each environment
has one waiter that calls the provider on the backoff (1, 2, then 5
minutes). The other waiters only read the database every 30 s. Before
this change the volume was zero after the second retry, because the task
waited for a person. The new volume per environment is at most about 8
provider calls in 30 minutes. It does not grow with the number of tasks.
The automatic-run brake still opens after 5 consecutive provider
refusals, and it limits the probe to one start per minute.
- After the outage, waiting tasks start again as the brake allows (a
probe, then a release ramp of 2, 4, 8 and more starts per minute). They
do not all start at once.
- A run that was in the old fast lane when this deploys continues in the
new lane when its second retry fails. A run that is already blocked
keeps its Retry.
- No migration. The new fields are optional JSON in `resultJson` and
`contextSnapshot`.

## Model Used

- Claude Opus 5.5 (provider Anthropic, model ID `claude-opus-5-5`), with
tool use and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 01:24:13 -06:00
DottaandClaude Opus 5.5 42858fe4e5 fix(recovery): continue a setup failure after work in seconds, with no Retry (#22128)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a run stops before it finishes, the recovery system saves its
files and then continues the task in a fresh run from that save
> - A native run that already did work can still fail during setup, for
example while the incoming server reattaches it after a graceful release
swap
> - That failure (`setup_failed`) is not a restart-class code, so the
backlog pass did not continue it. The task waited about 10 minutes and
showed Retry, although the system owned the next step
> - Recovery must be automatic and fast: a needed-Retry rate of 0%, and
recovery latency in seconds
> - This pull request continues such a task as soon as its saved work is
recovered, and shows it as "continuing" with no Retry while it waits
> - The benefit is that a setup failure after work is invisible to the
user: the task continues in seconds, not 10 minutes

## Linked Issues or Issue Description

Refs #22118 (that PR prevents the gateway refusal that caused this
instance; this PR fixes the class).

**What happened?**
In a release-swap test wave, a native run had done about 13 minutes of
agent work. It was handed to the incoming server at a graceful swap.
Reattachment failed and the run ended with `setup_failed` at 23:13:38Z.
Saved-work recovery saved the run's files 7 seconds later. The next run
started at 23:23:40Z, 10 minutes and 2 seconds after the failure. For
that whole time the task view showed `failed` with `canRetry: true`.

**Expected behavior**
A failure during setup or reattachment of a run that already did work
continues the task automatically within seconds. While the system owns
the next step, the task shows a system-owned "continuing" state and no
Retry. Retry shows only when nothing automatic is pending, and then it
works.

**Steps to reproduce**
1. Start a native run and let it do provider work.
2. Make the run fail during setup (for example a definite refusal of the
reattach resume), so it ends `setup_failed` with a restore hold on its
files.
3. Let saved-work recovery save the files.
4. See that no continuation starts until an unrelated system wake
reaches the 10-minute fresh-continuation bound, and that the task shows
Retry in the meantime.

**Paperclip version or commit**
`master` at 4afcbfc0c7.

Root causes:
- Latency: the backlog pass (`continueHeldExecutions`) selects held
records only for "recent interruptions" (`recentInterruptionSourceSql`):
restart-class codes, recovery runs, or setup failures before provider
work. A first run that failed setup after provider work matched none of
them. So nothing called the fallback after the restore hold settled. The
task continued only when a later system wake reached the fallback after
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes).
- Latency: even for an eligible record, a check made while saved-work
recovery was running was deferred to its backed-off `nextCheckAt` (1 to
4 minutes). Recovery completion did not wake the disposition worker.
- Presentation: no fresh-continuation check was recorded and no
saved-work recovery was active, so the projection fell through to
`failed` with `canRetry: true`.

## What Changed

- `continuation-cause.ts`: export `SETUP_FAILURE_CODES` (`setup_failed`
and the transient setup codes).
- New `fresh-continuation-eligibility.ts`: the held-record and
eligibility predicates move here from `fresh-continuation-fallback.ts`
with no behavior change (`heldExecutionWhere`,
`interruptionEligibility`, `recentInterruptionSourceSql`, the backlog
age constants, and the Blocked-ownership check the fallback uses). The
fallback re-exports the public names. The fallback file goes from 992 to
about 900 lines.
- Eligibility: the backlog pass now continues a system-owned setup
failure of a run that did provider work
(`executionRecovery.providerWorkStarted` is not `false`) whose files are
held for saved-work recovery or were recovered by it. A failure before
provider work, a person-owned code, and an exhausted saved-work recovery
keep their Retry.
- `expediteFreshContinuationChecks` (fallback module): makes the run's
pending `exact_recovery_pending` checks due now and wakes the
execution-disposition worker. `firstPendingAt` is kept, so the wait
bounds do not reset. `workspace-recovery.ts` calls it in the transaction
that completes saved-work recovery.
- `continueHeldExecutions` records its check in the same transaction as
the decision. A saved-work completion on the same hold then waits for
that commit, so its expedite cannot be overwritten by a stale future
check.
- `runsAwaitingSystemContinuation` (eligibility module): the held
records the backlog pass will continue by itself: same predicates,
system eligibility only, no person's wait recorded, no pending question
or approval, and a Blocked status that the failure owns (the same check
the fallback makes).
- `task-interruption-presentation.ts`: such a run shows
`freshContinuation: continuing` (system-owned, `working`, no Retry). A
pending Retry of that run keeps its own wait message. A task at the
hourly cap keeps `retry_scheduled`.
- Tests:
- New `setup-failure-after-work-continuation.test.ts` (embedded
Postgres, real saved-work recovery): the full sequence, plus no-work and
person-owned cases.
- `fresh-continuation-fallback.test.ts`: eligibility, expedite, the
completion-during-decision ordering, the cap presentation, and the
exhausted case.
- `fresh-continuation-schedule-queries.test.ts`: the oracle predicate
gets the same clause, and the fixture includes `setup_failed`.

## Verification

- Red/green:
`server/src/services/setup-failure-after-work-continuation.test.ts`,
"continues in a fresh run as soon as its saved work is recovered, and
never offers Retry". The sequence: a run with work fails setup, a
restore hold is recorded, the disposition pass runs, real saved-work
recovery completes, the disposition pass runs again, continuations are
delivered. The task view is checked at each step. On master the first
check fails (`state: "failed"`). With the fix every step shows
`responsibility: system, state: working, canRetry: false`. The successor
is created less than 5 s after the decision, with `previousRunId` and
`forceFreshSession`.
- `setup-failure-after-work-continuation.test.ts` "leaves a setup
failure before any provider work on its Retry path" guards the
provider-work gate.
- Red/green: `fresh-continuation-fallback.test.ts` "continues a setup
failure of a run that already did work once its saved work is recovered"
(master: not selected, 0 continued).
- `fresh-continuation-fallback.test.ts` "keeps a setup failure whose
saved-work recovery is still under way ...": the check waits, then the
expedite makes it due at once and the next pass continues.
- `pnpm verify` (selected files) and focused runs:
`fresh-continuation-*`, `task-interruption-*`, `workspace-recovery-*`,
`runner-lost-mid-turn-continuation`,
`heartbeat-native-remote-failure-retention`, `assigned-work-invariant`.

Recovery latency for this class, from saved-work completion to the new
run:
- Before: 10 min 2 s (the 10-minute pending bound).
- After: the next disposition pass, which the completion wakes. The
worker is throttled to one pass per 15 s, so the bound is 15 s. The test
asserts under 5 s. End to end from the failure, add the saved-work
recovery time (7 s in the observed case).

## Risks

- More tasks continue automatically: every system-owned setup failure of
a run that did provider work and whose files were saved, also when the
provider turn was in flight. The fresh run never replays the old turn.
Its opening context names the cut-off run and asks the agent to check
unverified actions before it repeats them. This is the same contract as
restart-class interruptions.
- The per-task cap (6 continuations per rolling hour), the per-server
backlog pacing (10 per minute) and the aggregate automatic-run brake are
unchanged and apply to these continuations. Judge the change by
company-wide automatic run starts per hour and their success rate.
- The task view predicts the fallback's decision from the same
predicates and Blocked-ownership check. A rarer refusal (for example a
dependency wait) can show "continuing" until the next disposition pass
records it, then Retry shows.
- The task view runs one more bounded query when listed tasks have an
unfinished latest run.
- No schema change. Records saved by the previous release are read the
same way. No work is added before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 23:49:21 -06:00
11a0ff0205 feat(connections): per-company workload identity issuer with public signing keys (#22083)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents use outside services through connections. The control plane
holds each connection's credentials and makes the calls.
> - Some clouds, such as AWS, do not need stored keys. They trust a
signed token from an identity provider that they can verify against its
public keys.
> - Paperclip has no signing key whose public half it can publish. Agent
identity keys are Ed25519, which AWS does not accept, and the agent's
run receives the private key. Agent run tokens use a shared secret.
> - This pull request gives each company its own signing key and
publishes the public half as a standard OIDC issuer.
> - The benefit is that a connection, such as the planned AWS connector,
can get its own short-lived cloud credentials with no stored cloud keys.
The connection's role in the customer's account sets its permissions.

## Linked Issues or Issue Description

Refs #19414 and #10411. Those pull requests add OIDC *sign-in* to
Paperclip. This pull request is the other direction: Paperclip *issues*
tokens that outside clouds verify.

**Subsystem affected**
Connections (tool access) and the database.

**Problem or motivation**
An AWS connector should not store a long-lived AWS access key. AWS
supports OIDC federation (`AssumeRoleWithWebIdentity`). AWS fetches the
issuer's public keys over HTTPS and checks each token. Paperclip has no
key that it can publish for this, so this method is not available.

**Proposed solution**
Each company gets an ES256 key that Paperclip creates on first use. The
vault master key encrypts the private half. The issuer is
`https://{instance}/oidc/companies/{companyId}`. Paperclip publishes a
discovery document and a JWKS there. Tokens last 5 minutes and have `sub
= connection:{id}`. Only the control plane signs them. The board can see
key state and run planned rotation or emergency rotation. The design is
in `doc/plans/2026-11-17-workload-identity-issuer.md`. The AWS connector
plan that uses it is in
`doc/plans/2026-11-17-aws-connector-and-minted-credentials.md`.

**Alternatives considered**
- Reuse agent identity keys. Rejected: AWS does not accept Ed25519, and
the agent's run receives the private key.
- Derive the key from the master key, with no table. Rejected: the
master key alone would then sign as production. Worktree environments
copy the master key on purpose. A master key rotation would also change
the signing key and break every customer trust.
- Use one issuer for the instance. Rejected: if a customer's role checks
the issuer but not `sub`, any company on a shared instance could assume
that role.

## What Changed

- `packages/db`: new table `company_workload_identity_keys` and
migration `0498`. The migration only creates a new table, so the
previous release keeps working. Partial unique indexes allow one
`active` key and one `next` key per company.
- `server/src/services/workload-identity.ts`: lazy key creation with
concurrent convergence, a private-key cache with a fail-closed key match
check, the discovery document, the JWKS, `signConnectionToken`, and
rotation steps. A key that is no longer active stays published for 24
hours. Paperclip calculates this when it reads the keys, so no
background job is necessary.
- `server/src/routes/workload-identity.ts`: public `GET
/oidc/companies/{companyId}/.well-known/openid-configuration` and `GET
/oidc/companies/{companyId}/jwks.json` (404 until a key exists). Board
`GET /api/companies/{companyId}/workload-identity` and `POST
.../workload-identity/rotation`, with an activity log entry for each
rotation.
- Token issuance writes a tool-access audit event. The event never
contains the token.
- `cli`: worktree seeds exclude the new table in the minimal mode and
the full mode. This is the same as for `agent_identity_keys`, so a
development copy never holds production's signing key.
- Docs: `docs/api/workload-identity.md`, an entry in the API overview,
and the two plan docs.
- No connector uses `signConnectionToken` yet. The AWS connector pull
request will use it.

## Verification

- `cd server && npx vitest run src/__tests__/workload-identity.test.ts`:
8 tests pass. The tests cover these items:
  - The issuer requires https with no port and no loopback host.
  - The endpoints return 404 before a key exists.
  - Six concurrent first uses create one key.
  - The JWKS contains only `kty crv x y kid alg use`.
- A token verifies against the published key and has the expected
claims.
- A company cannot sign for another company's connection, and its keys
do not verify that company's tokens.
  - Signing stops when the public address is not usable or has changed.
  - Two-step rotation and emergency rotation work.
- Board-only access is enforced, and each rotation writes an activity
log entry.
- Red/green: I put back a JWK that leaks `d` and removed the company
check on the connection. Three tests failed. I restored the code, and
all 8 tests passed.
- `cd cli && npx vitest run src/__tests__/worktree.test.ts`: 63 tests
pass.
- `npx tsc --noEmit` passes for `packages/db`, `server` and `cli`.
- I started this branch on a real dev server. Before a key existed, the
public routes returned JSON 404 (not the UI page). After an emergency
rotation from the board route, the JWKS returned one public P-256 key.
On the http dev instance, discovery stayed 404 and the board route
reported `not_https`.
- `pnpm verify` exited 2 because of the migration. The required CI gates
supply the broad coverage.
- Not run: a real AWS `AssumeRoleWithWebIdentity` call. That needs a
public HTTPS instance. It is the first step of the AWS connector work
and will run against staging.

## Risks

- The migration is low risk. It creates a new table with no backfill,
and the compatibility checker classifies it as compatible with the
running release.
- The public routes are unauthenticated. They show only public keys,
return 404 for companies without a key, and use random company UUIDs.
Each request is one small indexed read.
- The issuer URL shape is permanent after customers trust it. A change
to the instance's public address breaks those trusts until the customer
updates the identity provider. `signConnectionToken` stops on that
mismatch when the caller supplies the recorded issuer.
- Anyone who holds the production database and the master key could sign
for that instance's companies. That is the same exposure as every other
vault secret.
- Edge routing on Paperclip Cloud must pass `/oidc/...` to the stack. I
will confirm this on staging with the AWS connector work.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, file edits, web fetch of AWS documentation, a local dev server).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 23:33:47 -06:00
43b2d8be22 feat(dashboard): add an experimental areas and responsibilities dashboard (#22069)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The dashboard is the first page a person sees in a company
> - The classic dashboard shows agents, runs, costs and recent tasks
> - It does not show what the person is responsible for, or if that work
goes well
> - People think in areas, such as Fundraising or Customers, and in the
things they own in each area
> - This pull request adds a dashboard built on areas and
responsibilities, behind an experimental setting that is off by default
> - The benefit is that a person sees what they own, how it is going,
and what agents do for it, on one page

## Linked Issues or Issue Description

No public issue exists for this work. The description follows the
feature request template.

**Subsystem affected**

Dashboard, navigation, the task conversation, routines, and the instance
experimental settings.

**Problem or motivation**

The classic dashboard counts agents and tasks. It does not say what a
person owns or if it goes well. People who run a company with agents
want to see their areas, the responsibilities in each area, the bar each
one must meet, and the agents and routines that serve it.

**Proposed solution**

A person owns responsibilities. Each responsibility belongs to an area
and has a bar, a number to reach. Projects, tasks and routines link to a
responsibility, and the page shows its status from those links. An agent
can interview the person and draft the dashboard. The person accepts or
declines each change, and any change can be undone from a saved version.

**Alternatives considered**

Add the same data to the classic dashboard. That page is a fixed set of
cards for company activity, so a second, person-centred page is clearer.
The setting lets a person pick one of the two.

**Roadmap alignment**

This is an experimental feature behind `enableResponsibilities`. It is
off by default and changes nothing for a person who leaves it off.

## What Changed

- Add the setting "Areas & Responsibilities dashboard"
(`enableResponsibilities`, off by default). On, Dashboard in the sidebar
opens the new page, and the classic dashboard link is not shown in the
sidebars or the command palette. Off, nothing changes.
- Add tables and migrations 0492 to 0497 for areas, responsibilities,
links to work, metrics, status reports, saved dashboard versions, and
model-call records for summaries.
- Add the API: areas, responsibilities, links, bars, metrics, status
reports, summaries, runs, automation, and the dashboard conversation
with versions and undo. Each route checks company access and writes to
the activity log.
- Add four task conversations with an agent: about a bar, about an
automation, about a routine, and about the whole dashboard. The agent
proposes a change. Nothing applies until a person accepts it.
- Add standing checks: routines that an agent carries out on its
heartbeat and reports.
- Add the Dashboard page: search, counts, areas that open and close, a
primary New area button, and a message box at the bottom of the page.
Add the responsibility page with its bar, metrics, work and routines.
- Add the Paperclip skill sections and runner API entries that let an
agent use these routes. The runner skill inventory is regenerated (166
to 174 rows).
- Keep the new runner API tests in their own file so
`runner-api.test.ts` stays at 985 lines.
- Onboarding is not changed. The onboarding changes that build on this
work are a separate pull request.

## Verification

- Type checks pass for `packages/db`, `packages/shared`, `server` (all
layers) and `ui`.
- `pnpm --filter @paperclipai/db check:migrations` passes. A new
`drizzle-kit generate` run produces no further change.
- Tests for every changed file pass: 26 server test files (3,009 tests),
116 UI test files (1,991 tests) and 28 shared test files (432 tests).
- The runner capability inventory checks pass.
- I started the app on a local database that already held earlier data.
Master's new migrations and these migrations applied without error. With
the setting on, the Dashboard and a responsibility page loaded with no
console or server errors. The More menu and the command palette show one
Dashboard.
- I did not run `pnpm verify`, the full local suite, or the browser
suites. Required CI covers them.
- The agent conversations are tested with simulated agents. No real
model has run them.

## Risks

- Medium. The change is large, and the setting is off by default. With
it off, the new routes refuse requests and the UI is unchanged.
- Migrations only add tables and columns. Two replace a CHECK
constraint, and one builds an index `CONCURRENTLY`. These two have the
`online_compatible` override with a reason in the file, and the second
one is listed in `MIGRATIONS_WITH_TOP_LEVEL_STATEMENTS`.
- Master can add migrations before this merges. The migrations then need
new numbers.
- `responsibilities.ts` (1,663 lines), `responsibility-summaries.ts`
(1,239 lines) and one Storybook fixture file (1,305 lines) are new files
over 1,000 lines. They each keep one cohesive unit together: the service
for one table family, the model-call path for summaries, and the
fixtures that the stories share. No existing file crosses 1,000 lines.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude: Claude Fable 5.2 (`claude-fable-5-2`) wrote most of
the change, with Claude Sonnet 5.5 (`claude-sonnet-5-5`) for the final
steps. Both used tool use, code execution and browser checks.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [ ] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-17 20:45:44 -08:00
DottaandClaude Opus 5.5 4afcbfc0c7 fix(native): retry transient provider refusals while reattaching a handed-off run (#22118)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A release swap hands live native runs from the outgoing server to
the incoming server. The incoming server reattaches each run to its
original sandbox.
> - Reattachment first resumes the exact sandbox lease through the
sandbox provider plugin.
> - In a release-swap wave, one reattach resume got an HTTP 502 from the
provider gateway. The host did not retry it. The run failed with the
generic `setup_failed` code after 13 minutes of agent work.
> - The task then stayed blocked with a Retry banner for about 10
minutes, until the fresh-continuation fallback reached its bound and
started a new run.
> - This pull request retries a transient provider refusal during
reattachment, with a bounded budget. It also reads the provider status
from the plugin RPC error envelope, so the host sees the 502.
> - The benefit is that a brief gateway failure during a swap is
invisible to the user. The same run continues and no Retry banner shows.

## Linked Issues or Issue Description

**What happened?**
During a graceful release swap, the incoming server reattached a live
native run to its retained sandbox. The `environmentResumeLease` plugin
call failed with an HTML "502 Bad Gateway" body 280 ms after the
reattach started. The other 18 handed-off runs in the same swap
reattached in about 8 seconds. The host treated the 502 as an unknown
error, so the run failed with `setup_failed`. The task stayed `blocked`
with `canRetry: true` until the fresh-continuation fallback created a
new run 10 minutes later.

**Expected behavior**
A transient provider refusal (HTTP 408/425/429/5xx, a connection errno,
or a plugin RPC deadline) during reattachment is retried within a
bounded budget. The same run continues hands-off. A definite refusal
(for example HTTP 409) or a confirmed missing sandbox still fails at
once with its own named reason.

**Steps to reproduce**
1. Start a remote native run on a reusable sandbox.
2. Send SIGTERM to the server so it hands the run to the next server.
3. Make the first `environmentResumeLease` call on the incoming server
fail with HTTP 502.
4. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**
`master` at ca34aca777 (release-swap wave) and ea154ce55e (this branch
base).

Root causes:
- Why setup failed: `environmentRunOrchestrator.acquireForRun` called
`resumeRunLease` once for a reattachment. Acquisition already retries
transient resume refusals, but reattachment did not. Also,
`transientSandboxResumeFailure` read `status` and `statusCode` only from
the thrown error. A plugin refusal arrives as `JsonRpcCallError`, which
carries the provider status only in the validated
`paperclip/environment-resume-error/v1` data envelope. So a plugin 502
was not transient even on the acquisition path.
- Why recovery took 10 minutes: `setup_failed` on a native run with
provider work is not a restart-class code, so the same-run reattach lane
does not take it. Failed-workspace preservation recorded a restore hold.
The fresh-continuation fallback waits for that hold up to
`FRESH_CONTINUATION_PENDING_BOUND_MS` (10 minutes) when it sees no
progress. The continuation started at 23:23:40, which is 10 minutes and
2 seconds after the run finished at 23:13:38.
- Why a Retry banner showed: no fresh-continuation check row existed yet
and no system recovery was active, so the presentation fell through to
`failed` with `canRetry: true`. The banner did not stay after the task
was done. The issue rests at `done` with `canRetry: false`. The grader
reported 10482 s because it closed the banner at its final read hours
later. Its last sample that saw the banner was 272 s after the first.

## What Changed

- `environment-run-orchestrator.ts`: the exact-lease resume during
reattachment goes through `retryTransientResumeCall`. The first call
keeps the provider's configured start deadline. A first call that used
up that deadline (plugin RPC TIMEOUT) is not repeated. After a transient
refusal, a 90 s budget (`REUSABLE_SANDBOX_RESUME_BUDGET_MS`) bounds each
later wait and passes the remaining budget as `timeoutMs`. A confirmed
missing sandbox and a definite refusal are not retried.
- `packages/plugins/sdk/src/environment-resume-error.ts`: owns one retry
policy for provider failures: `isRetryableEnvironmentHttpStatus` (408,
425, 429, 500, 502-504 and gateway/edge 520-527, 529, 598, 599; never
501 or 505) and `isRetryableEnvironmentErrorCode(code, scope)`.
ABORT_ERR is never retried. ENOTFOUND is retried only in the
`known_host` scope (an exact reattach to a host this run already
reached, the DNS-outage case). Lease acquisition keeps its canonical
semantics in the `first_acquisition` scope.
- `environment-provider-unavailable-error.ts`: lease acquisition uses
the SDK policy instead of its own sets.
- `reusable-sandbox-resume-replacement.ts`: owns the resume retry helper
and `transientSandboxResumeFailure(error, scope)`, which reads only
structured evidence (status, envelope errno, plugin RPC deadline,
sandbox state transition) and a few transport phrases. The acquisition
resume loop uses the same classifier and backoff helpers, and
`resumeFailureTimedOut` (label or envelope ETIMEDOUT) for its idle-start
extension.
- `environment-runtime.ts`: `EnvironmentDriverResumeInput.timeoutMs`
caps the sandbox plugin driver's resume RPC. Other drivers keep their
own deadline.
- Cloudflare bridge client: a bridge request deadline throws
`CloudflareBridgeTimeoutError` with code `ETIMEDOUT`. Daytona:
`SandboxLivenessTimeoutError` carries code `ETIMEDOUT`. Both reach the
host as a structured errno in the resume envelope.
- `tests/restart-chaos/staging-mix.ts`: `bannerShownMs` counts a visible
Retry banner at most 2 minutes past the last sample that saw it. Before,
it counted the whole unsampled gap until the final read.

Previous-release story: this changes no persisted state, launch
configuration, or pack identity. A run saved by the previous release
reattaches the same way. Only the in-process retry around the resume
call is new.

## Verification

- `pnpm verify` (selected Vitest files), and focused runs of the four
changed server test files and the two restart-chaos test files.
- Red/green: `server/src/__tests__/environment-run-orchestrator.test.ts`
- "retries a provider gateway refusal during reattachment and keeps the
original allocation" fails on master (1 call, rejects with the 502) and
passes with the fix.
- "treats a provider HTTP %i refusal during reattachment as transient"
(500, 502, 520, 599).
- "gives the first reattach resume the provider deadline and bounds
every retry by the budget".
- "does not repeat a reattach resume that used up the provider's start
deadline", "does not retry a definite provider refusal during
reattachment", and "preserves a retained sandbox startup failure without
acquiring a replacement" (default budget, one call).
- `server/src/services/reusable-sandbox-resume-retry.test.ts`: a table
over every classifier branch, and the helper's deadline rules.
- `server/src/__tests__/environment-runtime.test.ts`: "caps the exact
resume call at the caller's remaining budget".
- `server/src/services/environment-provider-unavailable-error.test.ts`:
scope table and the extended status lists. Orchestrator reattach cases:
ENOTFOUND retried, ABORT_ERR and 501 not.
-
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`:
"applies the idle-start extension to a provider liveness deadline named
only by its errno" (red/green).
- Cloudflare `bridge-client.test.ts` and Daytona `plugin.test.ts`: the
deadline errors carry ETIMEDOUT in the resume envelope.
- Red/green integration:
`server/src/__tests__/heartbeat-native-remote-failure-retention.test.ts`
"keeps reattaching through a gateway refusal during a graceful hand-off
and continues the same run". The run's controller is lost, restart
recovery claims it to reattach, and the first resume fails with the 502
envelope. On master the run fails before the runner step (1 resume
call). With the fix the resume is retried, the same run reaches the
runner step at controller generation 2, no replacement allocation is
acquired, and no second run is created.
- `tests/restart-chaos/staging-mix.test.ts`: banner tests for a late
hide sample and for a banner still open at report time.

## Risks

- The acquisition resume loop no longer retries a refusal whose only
evidence is "timeout" or "timed out" text. Cloudflare and Daytona now
name their own deadlines ETIMEDOUT, and the plugin RPC deadline is
structured, so those stay retried. A provider that reports a timeout
only as text fails at once.
- Lease acquisition now also retries gateway statuses 520-527, 529, 598
and 599.
- A reattach resume can now take the provider deadline plus up to 90 s
when it gets transient refusals. Each retry is the same exact-lease
lifecycle resume. It cannot acquire or replace a sandbox, so a retry
cannot create a second executor.
- The aggregate recovery brake is unchanged. This change adds no new run
starts and no work before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Agentic coding with
tool use (shell, file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 22:11:04 -06:00
DottaandClaude Opus 5.5 6902c3f193 fix(native): keep recovery workers off a run its live heartbeat is finishing (#22101)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run ends in two steps: the live heartbeat exports the
workspace, then it commits the accepted result and records the run's
exit code, log summary and usage
> - Two recovery workers can also do these steps: the workspace
finalization worker and the result commit worker
> - The runner posts its accepted result before its process returns.
That post wakes both workers at once
> - The workspace finalization worker then takes the export lock while
the heartbeat still waits for the runner. The heartbeat gets "busy",
stops, and leaves the run to the workers
> - On staging, 45 of 135 recent succeeded native runs had no exit code
and no log size because of this
> - This pull request makes the workers leave a run alone while its live
heartbeat in the same process still runs
> - The benefit is that the heartbeat finishes its own run again, with
exit code, logs and usage. The workers still recover a run that the
heartbeat leaves unfinished

## Linked Issues or Issue Description

Refs #22043

**What happened?**
About a third of succeeded native runs had a null `exit_code` and a null
`log_bytes`. Their traces show a `task.workspace_finalization` span
(only the workspace finalization worker makes it) and no
`execution.native_finalize` or `completion.*` spans. The live heartbeat
lost the workspace export lock to the worker, threw
`NativeWorkspaceFinalizationBusyError` and stopped without completing
the run.

**Expected behavior**
The live heartbeat exports the workspace, commits the result, and
records the exit code, log summary and usage. The workers act only when
the heartbeat cannot finish the run.

**Steps to reproduce**
1. Start a native run that syncs its workspace.
2. Let the runner post its accepted result before its process returns.
The result post wakes the workspace finalization worker at once.
3. The worker exports the workspace first. The heartbeat gets `acquired:
false` and stops. The result commit worker then commits the run without
the heartbeat's completion data.

**Relevant logs or output**
Staging run `cb1cfc23-79eb-4528-bcdc-dcf639430d31`: the worker restored
the workspace in about 51s. The worker then committed the status
decision and projected the run as succeeded. The heartbeat wrote no
completion data.

## What Changed

- Add `server/src/services/native-runtime/live-native-finalization.ts`.
It keeps an in-process reservation for each run whose live heartbeat is
still running.
- The heartbeat (`runtime-execution.ts`) reserves a native run before it
dispatches. It releases the reservation when its execution ends. The
release wakes both worker queues for that run.
- `withNativeWorkspaceFinalizationOwnership` returns `owner_active` to
every caller except the run's own heartbeat while the run is reserved.
The workspace finalization worker already waits for a release
notification on `owner_active`.
- The result commit worker skips a reserved run. The release
notification wakes it again.

## Verification

- New regression tests, red before the fix and green after:
- `native-workspace-finalization-work.test.ts`: "leaves a result posted
mid-run to the live heartbeat, then resumes after its release". Without
the fix the worker exported the workspace while the heartbeat held the
run.
- `native-result-commit-work.test.ts`: "leaves a live heartbeat's run to
it and commits only what the heartbeat leaves behind". Without the fix
the worker committed the run while the heartbeat held it.
- Both tests also prove that the workers still finish the run after the
heartbeat releases it without finishing.
- `pnpm verify` (affected tests) passes. Server typecheck passes.
- After deploy: measure the share of succeeded native runs on staging
with a null `exit_code`. It should go down toward 0%.

## Risks

- The reservation is in memory. A worker in another server process (for
example during a deploy overlap) can still race the heartbeat. That case
behaves as before this change.
- If the heartbeat stops in an unusual way, its `finally` block releases
the reservation and wakes the workers. If the process crashes, the
reservation goes away, and startup discovery recovers the run as before.
- While a heartbeat is still running after the runner posted its result,
the workers wait. Before, they could start the export at once. The
heartbeat does the same work, so the result reaches the user at about
the same time.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 21:21:32 -06:00
Devin FoleyandPaperclip ea154ce55e Add bulk keyed issue document reads for native helpers (#22105)
## Thinking Path

> - Paperclip manages work for AI agents.
> - Native helpers read task documents through authorized API tools.
> - A scan of 146 cases needs one saved response and one download per
case.
> - The document service can read the same key across authorized tasks
with bounded queries.
> - This pull request adds a bounded bulk read with the existing task
access policy.
> - Helpers can transfer one batch and check current revisions before
fetching changed bodies.

## Linked Issues or Issue Description

**Subsystem affected**

Task document REST API, shared query contracts, and native API
discovery.

**Problem or motivation**

Per-task document reads make native sync scans cost hundreds of tool
calls. Task list timestamps do not prove that document content is
unchanged.

**Proposed solution**

Add `GET
/api/companies/{companyId}/issue-documents?key=notes&issueIds=...`. Read
up to 200 task UUIDs under current company and task permissions. Return
current bodies and revision metadata, or metadata alone with
`includeBody=false`.

**Alternatives considered**

Generic request batching would add tool execution and partial failure
rules. One bulk resource read removes the per-case relay cost with the
existing tools. Do not skip documents based only on task-row timestamps.
Keep managed skill files immutable; cleanup of copied skill test
fixtures belongs to those tests.

**Roadmap alignment**

This improves existing task documents and native API access. It adds no
new workflow or UI. The roadmap has no duplicate bulk document read
work. No duplicate open pull request was found.

## What Changed

- Add a bulk keyed document route and a company-scoped document reader.
Reuse the current batched task-read policy and completed-task read
receipts.
- Preserve request order and deduplicate IDs. Return `document: null`
for readable tasks without the key. Omit unreadable, hidden, foreign and
missing tasks without revealing which exist.
- Bound requests to 200 IDs and responses to 8 MiB of JSON bytes. Check
encoded body sizes in SQL before fetching bodies in the same read
snapshot. Build JSON within the byte budget. Oversized responses fail
with 413 and are never saved as partial artifacts.
- Admit at most two active reads per actor and eight per server
instance. Extra requests receive 429 with Retry-After. Keep the slot
until both the database work and response settle.
- Batch completed-handoff lookups and group child receipts by parent.
Extend the existing event batch writer for ordinary non-waking receipts
with atomic sequence allocation. Stored receipt shapes stay unchanged
for existing consumers.
- Expose revision-only reads without selecting document bodies. Document
native discovery, one save plus one download, and complete-inventory
checks.
- Add real HTTP, database and artifact tests for 146 open and completed
cases, 200 maximum-size bodies, metadata overhead, access failures,
query limits and JSON expansion. Cover admission limits, failure
cleanup, and concurrent event writers. Refresh the shared SDK source
pins.

## Verification

Final revision: `93dc8b029600211d79a050650f3dff82e5dad52f`.

- [CI run 44212220917, attempt
3](https://github.com/paperclipai/paperclip/actions/runs/44212220917)
passed. This includes typecheck, build, general tests, Runner checks,
serialized server suites, all e2e shards, and the final verify gate.
-
[Build](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506305267),
[e2e](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309117),
and
[verify](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134510062263)
are green.
- [Linux CI job
134506309291](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134506309291)
passed all 16 bulk document integration tests. This includes the native
transfer of all 146 documents with the saved size and hash,
completed-case receipt batching, and size-only rejection of 200
maximum-size documents.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22105#issuecomment-6965449032)
is 5/5 on the final revision. All review threads are resolved. Security
checks pass. GitHub reports CLEAN and MERGEABLE.
- Local server typecheck passed on macOS with Node 25.6.1. The focused
run below passed 177 tests; the Linux-only transfer test was skipped
locally and then passed in CI.

```sh
pnpm --filter @paperclipai/server typecheck
cd server
pnpm exec vitest run \
  src/routes/issue-document-batch.test.ts \
  src/services/native-runtime/issue-document-batch.integration.test.ts \
  src/__tests__/documents-service.test.ts \
  src/__tests__/openapi-routes.test.ts \
  src/__tests__/heartbeat-run-event-atomic-append.test.ts \
  src/__tests__/chat-completion-delivery.test.ts \
  src/__tests__/native-child-task-read-routes.test.ts
```

- Shared typecheck and build passed. From `packages/shared/`, `pnpm exec
vitest run src/validators/issue.test.ts` passed 61 tests before the
rebase; the batch query contract did not change in the rebase.
- `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
passed. Runtime and lockfile pins are unchanged.
- `pnpm verify --explain` returned exit 2 because the API docs and
shared contracts require broader CI. The successful CI run above
supplies that coverage.

Failure and recovery record:

- Both performance findings reproduced before the fixes. Scale tests now
verify a size-only query for 200 large documents and one handoff lookup
plus two receipt-write statements for 146 completed cases. Concurrent
writers retain contiguous unique sequences and decoded payloads.
- Initial local runs found missing shared exports and OpenAPI route
coverage. These are fixed. Fixtures now respect Linux-only installation,
thrown cross-company denials, and the ES2022 TypeScript target. Rebase
conflicts preserved the onboarding coverage from master.
- CI attempt 1 interrupted Build, e2e shard 7 and six general-test
shards with runner shutdown signals. Attempt 2 recovered the tests but
its [Build
job](https://github.com/paperclipai/paperclip/actions/runs/44212220917/job/134499680425)
also received a shutdown signal and exited 143. No compiler or test
assertion failure was found in those interrupted job logs. The reason
the runners stopped remains unknown. Attempt 3 recovered Build and its
dependent verify gate on the same revision. Earlier interruptions are
not counted as passes.

## Risks

- Additive API only. No migration or existing endpoint response change.
- Helpers must adopt the batch response to gain the tool-call reduction.
They must verify that every requested task is present before treating
the scan as complete. Unreadable tasks are intentionally omitted.
- Split large document batches when the route returns 413. Metadata
remains available without bodies. Concurrency limits are per server
instance and can refuse simultaneous reads with 429; callers retry after
an active read finishes.
- This does not change company-owned helper inventory expiry or cleanup
tests, installed skill permissions, or generic tool batching. It does
not deploy to a tenant.

## Model Used

OpenAI `gpt-6.1-sol` through Codex in T3 Code, with xhigh reasoning,
tool use, and code execution. Context window size was not exposed.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 17:42:18 -08:00
Devin FoleyandPaperclip bf55063567 fix(server): keep chat titles current and space verification retries (#22102)
## Thinking Path

> - Paperclip manages AI agents and their work.
> - Agent chats use persistent tasks, and remote runs verify their
provider files before execution.
> - A chat keeps the name that its agent had when the chat was created.
> - A fast gateway failure can consume both verification attempts before
the gateway recovers.
> - This change updates generated chat titles on an agent rename and
waits before a verification retry.
> - Chats show the current name, and a short gateway fault has time to
clear.

## Linked Issues or Issue Description

**What happened?**

Renaming an agent leaves existing generated chat titles unchanged.
Remote provider verification also retries a transport failure
immediately. Two fast HTTP 502 responses can fail startup before the
model starts.

**Expected behavior**

Generated chat titles follow an agent rename. A read-only verification
retry gives the remote gateway time to recover. Verification still
rejects incompatible files and stops after two attempts.

**Steps to reproduce**

1. Open a chat with an agent, then rename that agent. Inspect the task
linked from a run: its title still has the old agent name.
2. Make the remote verification transport return HTTP 502 for one
second. Start verification. Both immediate attempts fail before the
transport recovers.

Related: #22087 covers a provider connection failure during lease
acquisition. This change covers a later provider-file verification
request.

## What Changed

- Update matching generated chat titles in the agent configuration
transaction. Preserve custom chat titles, ordinary task titles, and
other agents' chats.
- Record each changed chat as `issue.updated` and publish its event
after commit so open pages refresh. Roll back the rename and queued
events if audit persistence fails.
- Derive new chat titles from the current agent name under the existing
company transaction lock. Both user-opened and proactive chats use this
path, including a create that read the old name before a rename
committed.
- Acquire that company lock before proactive chat delivery takes chat or
agent row locks, so delivery and renames use a consistent lock order.
- Wait one second before the second remote verification attempt. Keep
the existing attempt limit, transport timeouts, artifact checks, and
failure codes.
- Add route and startup-verification regressions. Document the retry
delay.
- Wait for the connection-review E2E fixture to save its Ask first
permission before it creates a task or navigates away.

## Verification

- Final acceptance on `3fb74ffaf6fa0ee2ecbe5dba3d01d252cb0c986f`: [full
CI
run](https://github.com/paperclipai/paperclip/actions/runs/44208144377)
is green, all 61 reported checks are complete with no failures, Greptile
is 5/5 on this exact commit, all review threads are resolved, and GitHub
reports the branch clean and mergeable.
- Red/green: the rename test retained the old titles before the fix. The
gateway recovery test consumed both attempts before the one-second
recovery point. Both pass with the changes.
- The live-update and concurrent-create tests failed on the first PR
revision: no title events were published, and a paused chat creation
retained the old name. The fixes queue events for publication after
commit and re-read the agent name under the company lock. Both
reproductions now pass.
- `pnpm exec vitest run
server/src/__tests__/agent-configuration-save.test.ts
server/src/__tests__/agent-chat-messages.test.ts
server/src/__tests__/agent-conversations.test.ts
server/src/services/native-runtime/remote-provider-pack.test.ts
server/src/services/native-runtime/remote-provider-pack-verification.test.ts`:
191 passed on macOS, including the concurrent rename/message regression.
- `pnpm --filter @paperclipai/server typecheck`: passed, including
source and test layers.
- Earlier CI on `de2a0a4` failed in the release-registry watchdog
self-test ([job
log](https://github.com/paperclipai/paperclip/actions/runs/44202511888/job/134460224014)).
Its 1.5-second watchdog fired during `starting the file's tests`; the
assertion expected `test "waits forever"`. Both reporter files are
unchanged from the base and current head. `node --test
scripts/ci/stalled-file-reporter.test.mjs` passed all 7 tests on macOS
with Node v25.6.1. The current revision also passed the Linux Typecheck
+ Release Registry CI check; no reporter code was changed.
- CI on `c83fcd9` failed the connection-review failure journey ([shard
log](https://github.com/paperclipai/paperclip/actions/runs/44204043415/job/134465938135)).
Navigation aborted the pending permission save, so the fixture tool ran
without creating the expected approval card. A temporary two-second
delay on that save reproduced the failure locally. With the response
wait, the same delayed-save reproduction passed. The temporary delay is
removed from the diff.
- `node scripts/typecheck-test-projects.mjs --reuse-declarations
tests/e2e`: passed using the server declarations from the same
application source revision.
- Before the rebase, the complete connection-review spec passed 4
journeys and skipped the controlled-restart journey under the default
configuration. On `95fb2aa`, rebased onto `9032f55d83e`, the
provider-failure journey, all 190 focused tests, server typecheck, and
E2E test-project typecheck passed again. The dedicated
controlled-restart suite was not run.
- A concurrent rename/message regression reproduced PostgreSQL `40P01:
deadlock detected` on `95fb2aa`. Delivery now takes the shared company
lock before the chat and agent locks. The regression uses observed
PostgreSQL blockers to stage the collision, and both operations commit
with the fix. Concurrent creation also waits behind the same lock and
retains the renamed title.
- The application tree for `3fb74ff` passed all 191 focused tests and
server typecheck. The subsequent rebase onto `96907f5518c` changed only
`.github/CODEOWNERS` compared with that tested tree. The full CI run for
this revision is now green.
- CI jobs on `95fb2aa` were interrupted after their runners received
shutdown signals: [server
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311777),
[workspace
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311921),
[typecheck/release
registry](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474310521),
and [serialized server
shard](https://github.com/paperclipai/paperclip/actions/runs/44206614135/job/134474311690).
The typecheck job is labelled failed, but its active step was cancelled.
These interruptions did not supply passing test evidence. The fresh CI
run on `3fb74ff` completed without failures.
- `pnpm verify --explain`: exit 2. The changed recovery document
requires broad CI coverage. Required CI supplies the full test and build
coverage; no full local suite or build was run.
- `git diff --check` and a Gitleaks scan of the diff passed. The diff
was also checked for personal data and private identifiers.

## Risks

- An inconclusive verification takes one extra second before its last
attempt. A persistent outage still fails with the existing named error.
No provider turn or artifact mutation is replayed by this retry.
- Only chat titles that match the previous generated title change.
Earlier stale titles and custom titles need an explicit correction.
- Proactive chat delivery now shares the existing company writer lock
with agent configuration. Concurrent writes in that company wait for the
active transaction to finish.
- No schema, session, checkpoint, launch identity, or pack identity
changes. State saved by the previous release carries forward with the
same verification checks and attempt limit.
- Revert this commit to restore the previous behavior. Existing title
updates are metadata and need no schema rollback.

## Model Used

- OpenAI Codex, `gpt-6.1-sol`, xhigh reasoning, with source inspection,
browser investigation, and local code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:59:05 -08:00
ce27dde855 Start onboarding in chat and defer execution until the user chooses work (#22077)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Onboarding connects an agent and gives the user a place to discuss
work.
> - The previous flow created an execution task and project before the
user chose a scope.
> - A standing chat can gather the goal and clear exit criteria first.
> - This pull request opens chat after onboarding and starts a live
welcome after the chat is visible.
> - The agent proposes work for acceptance, or creates a task when the
user explicitly requests one.
> - Subscription setup and shared question state must work for this
first conversation to succeed.

## Linked Issues or Issue Description

**Subsystem affected**
Company onboarding, agent chat, local subscription setup, task
numbering, and shared interaction views.

**Problem or motivation**
The first session creates work before the user has agreed to a goal.
Standing chats consume task numbers. An answered chat question can
remain pending in the inbox. Claude code entry requires an extra click,
and switching from Claude to Codex can leave an incompatible model
selected.

**Proposed solution**
Open an owner-bound chat for every new company. Start a live welcome
after the chat paints in a visible tab. Gather scope in conversation and
use the existing task acceptance flow. Keep chat records out of the task
number sequence. Publish question responses to shared caches and cancel
stale reads. Submit pasted Claude codes once and wait for verified
sign-in before connecting.

**Alternatives considered**
A fixed welcome would not use the onboarding context. A hidden execution
task would still create work and consume a task number. Keep the
existing task proposal and acceptance contract instead.

**Roadmap alignment**
This improves the shipped Agent Chat and Personal & Shared AI Accounts
flows in ROADMAP.md. It does not add a separate execution or approval
system.

Related prior work: Refs #19643 and #19512. Related route handoff work:
Refs #12557. This change replaces the automatic first task with chat; it
also repairs the subscription and shared question state paths used
during that journey.

## What Changed

- Finish onboarding in agent chat for default and team-template
companies. Create no execution task or project.
- Queue one live welcome after visible chat content paints. Pass company
goals and the initial idea as context. Preserve access, budget, and
hiring controls.
- Instruct the agent to gather a goal and exit criteria, offer an
interview, and propose tasks for acceptance. Allow direct creation on an
explicit user request.
- Leave new standing chats unnumbered. The first execution task receives
number 1. Preserve existing identifiers.
- Synchronize answered question cards across chat and inbox, including
concurrent requests and inactive cached views.
- Submit a pasted Claude authorization code once. Show connection
progress and hide the restart link while sign-in is still preparing.
Detect rejected OAuth codes promptly. Clear incompatible models when the
provider changes.
- Use `Work with [agent name]` as the default chat composer placeholder.
- Add regression coverage and update onboarding documentation and API
notes.

## Verification

- Completed a real Claude subscription journey on macOS with an isolated
local database. Authorized the account, pasted the real code, and
reached the review step without clicking Submit code or Connect. Get
started opened chat and the live welcome run succeeded. The company had
zero execution tasks, zero projects, and task counter 0.
- On rebased commit c664949858, 6 browser cases passed for
paste-to-connect-to-chat, task-free onboarding, existing-company
routing, the native connection probe gate, desktop/mobile planning, and
two-tab chat/inbox question synchronization.
- On the rebased head, 419 focused UI tests passed. The focused server
and adapter run passed 160 tests and hit one `socket hang up` in the
existing authenticated chat route test. The full 72-test conversation
suite passed on one repeat. The cause of that first socket failure
remains unconfirmed; the passing repeat does not establish its cause.
- UI typecheck and server declaration typecheck pass after the rebase.
The first server check found a missing local dependency from master; a
frozen-lockfile install repaired it. No dependency manifest or lockfile
change was needed.
- The first Greptile review found two welcome edge cases. Four new tests
reproduced both before the fix. On commit b380f14ef9, all 114 onboarding
and queue tests pass, including timeout retries and queued welcomes
replaced by a user message or chat reset. Server source and focused test
type checks pass on this revision.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/22077#issuecomment-6961533641)
gives rebased commit 2dbad34445 a score of
5/5. The current-head check passed. Both welcome review threads are
resolved, with no new actionable findings.
- CI serialized shard 8/9 found a missing OpenAPI inventory entry for
the new route file. The exact coverage test reproduced this locally.
Commit 17a72ec8b7 adds explicit mounted paths and documents both
board-only operations. All 65 OpenAPI and onboarding route tests and
server source typecheck pass after the fix. The final CI run passes on
this revision.
- Rebased onto master 23ce68a8dc and regenerated the SDK source registry
from committed inputs. Worktree preflight, SDK source validation, and
all 65 API/onboarding tests pass on final head
2dbad34445.
- On final head 2dbad34445, all 419
affected UI tests pass. UI package typecheck and server declaration
typecheck also pass. Commands: `pnpm --filter @paperclipai/ui
typecheck`; `node server/scripts/typecheck-layers.mjs
--declarations-only`; focused Vitest suites for the wizard, subscription
login, agent chat, shared interaction cache, and OpenAPI/onboarding
routes.
- [Final-head CI
run](https://github.com/paperclipai/paperclip/actions/runs/44208652564)
completed successfully on commit
2dbad34445. All required typecheck, build,
test, Runner, browser, serialized-server, and canary gates pass,
including the [final verify
job](https://github.com/paperclipai/paperclip/actions/runs/44208652564/job/134487749343).
The fresh routing gate selected the trusted AWS fleet after the
protected owner policy in #22098 merged. No source rebase was needed:
the gate reads that policy from protected master.
- The repaired OpenAPI shard passed in the earlier same-head CI run.
General server shard 8/12 hit a 20-second step timeout in the existing
`issue-recovery-live-run-agent-census` suite. This PR does not change
that suite or its fixture. All four tests pass locally on the exact
current commit in 84 seconds. The cause of the CI timeout remains
unconfirmed. Every other test job passed. A single failed-shard retry
was accepted but remained queued on its saved GitHub-hosted selection.
It was cancelled before test execution after #22098 merged. The fresh
full-CI run passed on the unchanged head under the approved AWS policy,
including [server shard
8/12](https://github.com/paperclipai/paperclip/actions/runs/44208652564/job/134481388730).
This passing execution does not establish the cause of the earlier
timeout.
- Worktree preflight, committed SDK source input validation, and diff
whitespace checks pass after the rebase.
- `pnpm verify` reports exit 2 because the new subsystem documentation
requires broader CI coverage. The final successful CI run supplies that
broader coverage. This is not a full local suite pass.

## Risks

- The initial task is now an explicit user decision. Existing onboarding
tasks and numbered records remain readable. No schema migration is
required.
- A welcome still depends on a working provider connection and the
normal run queue. Failed starts can be retried without duplicate
successful welcomes.
- Claude paste acceptance was verified with a real subscription on
macOS. This change does not claim a new live Codex or Windows acceptance
test.

## Model Used

OpenAI Codex, based on GPT-6, with tool-assisted reasoning, code
execution, terminal checks, and browser verification. The client does
not expose the exact routed build or context limit.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Scott Tong <scott@scottsmbpm5max.lan>
Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:52:35 -08:00
Devin FoleyandPaperclip 9032f55d83 fix(ssh): support all Cloud SSH harnesses in remote task workspaces (#22078)
Support durable in-place SSH task workspaces across all declared SSH-capable
Cloud harnesses. Keep controller filesystem operations off remote paths,
preserve SSH session identity, and stage provider assets on their owning
execution target. Keep managed ACP conversation identity stable across
credential-home rotation and preserve Pi JSONL history after credential cleanup.

Verify with the real SSH heartbeat matrix, adapter and session regression
checks, and typechecking. Final CI passed on 3b629da with 57 successful checks
and three skips. Greptile scored the change 5/5 with no unresolved comments.
Live paid-provider acceptance remains unqualified.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
2026-11-17 16:06:25 -08:00
Devin FoleyandPaperclip a3f86e5cc8 fix(recovery): continue a recovered preparation that failed with a transient code (#22096)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts its provider, Paperclip prepares the
sandbox: it restores the saved workspace and stages the agent files
> - When a legacy Runner preparation fails after the sandbox exists,
Paperclip keeps the sandbox, saves its files, and holds the task with
"preserve without replay"
> - A preparation that a restart or a lost process stopped continues by
itself after the files are saved. A preparation that failed with a
transient error does not: it waits for a person to click Retry, even
though its own bounded retry would continue it
> - The agent-file Git exclusion timeout also did not count as
transient. It failed with the generic `adapter_failed` code, so nothing
retried it, and the wake that started the run was lost
> - This pull request continues a recovered preparation that failed with
a transient preparation code, in the same bounded lane as a lost
preparation, and names the exclusion timeout as a transient preparation
timeout
> - The benefit is that a slow sandbox no longer stops a task until a
person notices it

## Linked Issues or Issue Description

Refs #22039, which retries the timed-out Git exclusion in place. This
change covers the case where all three attempts time out.

**What happened?**
A run failed in the preparing stage with "Could not exclude agent files
from task Git staging". No provider work started. The sandbox was kept,
and workspace recovery later saved its files with no changes. The
recovery hold recorded `replay: blocked`, so no new run started and the
wake was lost.

**Steps to reproduce**
1. Start a legacy Runner run on a sandbox that keeps its lease on
failure.
2. Make the agent-file Git exclusion time out on every attempt.
3. Let workspace recovery save the retained sandbox.

**Expected behavior**
After the files are saved, the run continues once in its bounded retry
lane, as a preparation stopped by a restart does.

**Actual behavior**
The run fails as `adapter_failed`. After recovery, the task waits for a
manual Retry.

## What Changed

- `failed-preparation-retry.ts` (new):
`isTransientWorkspacePreparationCode` and
`failedPreparationOwnsBoundedRetry` move here unchanged from
`heartbeat/retries.ts`, which re-exports them. The code list is now an
array, so SQL can use it. The move lets `cancelled-native-startup.ts`
use the predicate without an import cycle, because `retries.ts` already
imports that file.
- `cancelled-native-startup.ts`:
`isInterruptedNativePreparationCandidate` also accepts a failed
preparation that has a required workspace recovery and whose own bounded
retry owns it. All other conditions stay the same: legacy Runner, still
preparing, no provider evidence, released leases with stop receipts, and
a verified workspace repair before replay.
- `workspace-recovery.ts`: the recovery sweep lists the same runs, so
their continuation is delivered after the files are saved.
- `run-completion.ts`: a `RemoteWorkspacePreparationTimeoutError` that a
legacy run raises before it enters the adapter now records
`remote_workspace_prepare_timeout`, the same code the setup path already
records.
- `agent-instruction-files.ts`: after the last timed-out attempt, the
exclusion throws `RemoteWorkspacePreparationTimeoutError`. The message
stays "... (reason=timed_out)". `RemoteWorkspacePreparationTimeoutError`
takes an optional message for this.
- Tests: a recovered preparation that failed with
`remote_workspace_prepare_timeout` is listed by the sweep and continues
once; one that failed with `adapter_failed` still waits for Retry.
`failRun` names the timeout only when the adapter did not start. The
agent-file test checks the error type.

## Verification

- `npx vitest run` on the new and changed cases: the
retained-preparation continuation, the `failRun` classification, and the
agent-file suites. All pass.
- Each new case fails without its source change. With only the sweep
query reverted, the listing assertion fails.
- `npx vitest run` on 62 related server test files (workspace recovery
and its preparation suites, legacy execution recovery, run completion,
retry scheduling, recovery service, remote workspaces, task retry, fresh
continuation): 2361 pass, 16 skipped.
- `npx tsc --noEmit` passes for the server.

## Risks

- Low to medium. A failed preparation that kept its sandbox now
continues once its files are saved, if its failure code is transient.
The continuation uses the interrupted-preparation lane, which is bounded
to two attempts, and the existing failure budget still applies.
- A failure that is not transient (`adapter_failed`, `setup_failed`) is
unchanged: it still waits for Retry.
- A run whose failure is not a pre-provider remote preparation timeout
keeps `adapter_failed`.
- A task that is `done` or `cancelled` is unchanged. A scheduled retry
never runs on a closed task, so a wake on a closed task is still not
replayed after a failed preparation.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 16:04:19 -08:00
DottaandClaude Opus 5.5 f295e496db fix(daytona): retry lease acquisition after a provider connection failure (#22087)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs on Daytona start with a lease acquisition in the Daytona
plugin
> - When the host cannot reach Daytona, the Daytona SDK raises a
connection error that has no HTTP status and no errno
> - The host only retries a lease failure that carries structured
evidence, so this failure became `setup_failed` and the task waited for
a person to press Retry
> - This pull request makes the plugin report the SDK connection error
as a closed transport code that the host already retries
> - The benefit is that a short network blip during sandbox startup no
longer needs a Retry click

## Linked Issues or Issue Description

**What happened?**

In a restart-chaos wave (201 tasks, 6 graceful release swaps), the host
could not resolve or reach Daytona for about 12 minutes. One task needed
a Retry click. Its only run failed with `setup_failed`
(`lease_acquire_failed`) and the message `Failed to acquire lease for
environment "Daytona warm ..." (sandbox): read EADDRNOTAVAIL`. Automatic
recovery did not continue the task because `setup_failed` is not a
transient setup code. The task stayed failed for about 100 minutes until
the grader pressed Retry.

**Expected behavior**

A lease acquisition that fails because the provider could not be reached
is retried automatically with the bounded bootstrap budget, as
`environment_provider_unavailable`. A person never needs to press Retry
for a network blip.

**Steps to reproduce**

1. Use the Daytona provider.
2. Make a Daytona API call during lease setup fail at the transport
level (no HTTP response), for example with `read EADDRNOTAVAIL` or
`getaddrinfo ENOTFOUND`.
3. See that the run fails with `setup_failed` and the task shows Retry.

**Paperclip version or commit**

master `ca34aca777` (wave) and `02ee538be9`.

Root cause: the pinned `@daytonaio/sdk` converts an Axios transport
error into `DaytonaConnectionError` (or
`DaytonaConnectionTimeoutError`). It keeps neither the errno nor the
cause. The worker RPC therefore sent no provider diagnostic, and
`readTransientLeaseAcquisitionFailure` returned `null`.

## What Changed

- The Daytona plugin wraps a `DaytonaConnectionError` from lease
acquisition in an error with code `CONNECTION_FAILED`, and a
`DaytonaConnectionTimeoutError` with code `ETIMEDOUT`. The message is
unchanged. It reads the SDK class name only, never message text.
- The plugin SDK closed diagnostic set accepts `CONNECTION_FAILED`, so
the code crosses the worker RPC.
- The host treats `CONNECTION_FAILED` as a transient connection failure.
The run becomes `environment_provider_unavailable` and uses the existing
bounded bootstrap retry (30 s, 60 s) and the automatic-run brake.
- Refreshed `tests/runner-e2e/source-sdk-public-inputs.json` for the
changed SDK file.

## Verification

- Red/green: with the source change removed, the new plugin tests
(`reports a DaytonaConnectionError during setup as a transient transport
failure`, timeout variant) and the new host tests (`treats errno
CONNECTION_FAILED as transient`, `retries a provider connection failure
reported across the worker RPC`) fail. With the change, they pass.
- `cd packages/plugins/sandbox-providers/daytona && npx vitest run` (637
passed).
- `npx vitest run
server/src/services/environment-provider-unavailable-error.test.ts
packages/plugins/sdk/tests/environment-acquire-error.test.ts` (66
passed).
- `pnpm typecheck` in the Daytona plugin passes.
- `pnpm verify` asks for broad CI coverage because the SDK input
registry changed. CI supplies that coverage.

## Risks

- Low risk. A wrong Daytona API URL also produces
`DaytonaConnectionError`. It now gets two bounded retries (about 90 s)
before the same Retry state as before.
- The retries count toward the automatic-run brake, so a long outage
cannot multiply run starts across the company.
- A create that timed out on the client keeps its existing
creation-cleanup envelope. The retry is a new creation attempt with its
own identity.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-17 16:55:47 -06:00
Devin FoleyandPaperclip eaf6461cee fix(server): give a slow remote more time to exclude agent files from Git (#22039)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, Paperclip runs a short Node program in
the sandbox that keeps the reserved runtime directory out of task Git
staging
> - That program had one 15 second attempt. A freshly created sandbox is
sometimes slow, so the attempt timed out
> - The run then failed before any provider work started, with the
generic `adapter_failed` code. Nothing retried it, and the wake that
started the run was lost
> - This pull request gives each attempt 30 seconds and retries a
timed-out attempt, up to three attempts
> - The benefit is that a slow sandbox no longer stops a task, while a
real failure still fails at once and names its reason

## Linked Issues or Issue Description

Builds on #21499 (merged), which adds the closed `reason=` code to this
error.

**What happened?**
A run failed during preparation with "Could not exclude agent files from
task Git staging". The sandbox lease was acquired about 50 seconds
before the agent-file copy was registered (3 to 7 seconds on the same
stack's earlier runs). The copy failed 20 seconds after it was
registered, which matches the 15 second limit. Earlier failures with
this message all took at least 16 seconds before failing, so they also
match a timeout rather than an error from the program itself.

**Steps to reproduce**
1. Start a remote run on a sandbox that takes more than 15 seconds to
run a short command.
2. The run fails in the preparing stage and is not retried.

**Expected behavior**
A slow sandbox gets more time, and a timed-out attempt is retried.

**Actual behavior**
One 15 second attempt, then the run fails.

## What Changed

- `agent-instruction-files.ts`: add `excludeRuntimeFromRemoteGit`. Each
attempt gets 30 seconds. A timed-out attempt is retried, up to three
attempts. Any other failure throws at once. The error keeps the closed
`reason=` code from #21499.
- `agent-directory-working-copies.ts` and
`agent-instruction-working-copies.ts`: both remote call sites use the
helper, so the private instruction copy also names a reason.
- `execution-target.ts`: `runSshCommand` rejects a command its timeout
killed with `code: null` and `killed: true`, but only `ETIMEDOUT` was
treated as a timeout, so an SSH timeout was thrown instead of returned.
A killed command now returns `timedOut: true`, as the sandbox path
already does.
- Tests: a timeout followed by success stages the agent files; three
timeouts fail with `reason=timed_out`; a non-timeout failure is
attempted once. An adapter-utils test covers the real SSH timeout
rejection.

## Verification

- `npx vitest run` on the agent-directory, instruction working-copy, Git
exclusion, copy retry backoff, instruction-working-copy and remote
checkpoint lock suites: 217/217 pass.
- adapter-utils `execution-target` and sandbox suites: 173/173 pass.
adapter-utils SSH suites: 65 pass, 2 skipped. Server adapter registry
suite: 67/67 pass.
- The new cases fail without this change.
- `npx tsc --noEmit` passes for server and adapter-utils.

## Risks

- Low. A sandbox that never answers now takes up to 90 seconds to fail
instead of 15 seconds.
- An SSH shell command that times out is now returned as `timedOut`
instead of thrown. Every caller that checks the result already treats
`timedOut` as a failure. The home-directory probe and the empty-file
helper ignore the result, so for them an SSH timeout now behaves as a
sandbox timeout already did.
- The exclusion program is idempotent: it writes its file through a
temporary name and a rename, so a retry after a late first attempt
leaves the same result.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 14:28:27 -08:00
Nicky LeachandPaperclip f05b90508a refactor(core): remove retired compatibility paths (#22055)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Core manages chat authorization, AI connection defaults, navigation,
and the plugin host.
> - Core still writes old default rows and checks an Agent Chat flag
that always reads true.
> - It also carries a desktop rail that cannot open and private aliases
with canonical equivalents.
> - This pull request removes those paths and moves their tests onto the
paths Core uses.
> - The result is fewer Core database operations and less code to
maintain.

## Linked Issues or Issue Description

**What existing behavior does this improve?**

Core AI default writes, Agent Chat execution and delivery, private
plugin host and Runner setup, and board navigation.

**Subsystem affected**

Cross-cutting. Core server, board UI, private Runner helpers, and a
repository guard script.

**Current behavior**

The server writes provider-scoped and per-method AI defaults, then reads
only the provider-scoped rows. It reads settings to decide whether to
block Agent Chat, although every settings view enables it. The desktop
sidebar renders expanded while still tracking requests to open and close
a retired icon rail. The agent client retries conflicts that the server
already resolves when there is exactly one active match.

**Proposed behavior**

Write provider-scoped defaults directly. Use the existing Agent Chat
authorization and retry checks without the redundant settings reads.
Render the current sidebar directly and let the API report true agent
reference conflicts.

**Reason and benefit**

A personal non-custom connection save or explicit default selection
sends one fewer SQL statement. It also avoids the provider-default write
that the legacy trigger performed when the old statement inserted or
updated a row. A successful task-to-chat send or receipt replay avoids
two transaction settings SELECTs when the singleton row already exists.
Conversation comment, wake release, and retry paths also avoid a
transaction settings read when they previously entered the gate. Savings
in other settings paths depend on cache hits. Core-to-plugin RPC and
plugin-to-Launcher HTTP traffic stay the same.

**Breaking changes**

The public plugin SDK, host navigation, and organization-switcher fields
retain their existing interfaces and behavior. The settings API still
accepts and reports the Agent Chat compatibility key. This change needs
no schema migration or rollout ordering.

**Additional context**

Refs #10610 for prior reporting on the retired sidebar rail. Refs #21756
for adjacent Coordinator plugin slots, which remain a separate change.

## What Changed

- Stop writing per-method AI defaults. Preserve the provider-scope
upsert and first-account behavior. Keep the existing tables and
migration trigger.
- Remove Agent Chat checks that read a constant true value. Keep
conversation ownership, task scope, session, and request-retry checks.
- Remove unused lock-owner fields and private plugin lifecycle, secret
extraction, Gmail connector, PRP setup, heartbeat resolver, and ACPX
aliases.
- Use a single API request for agent references. Remove retired sidebar
rail state, hooks, and branches across the active and production
snapshot layouts. Preserve desktop resizing, mobile drawer controls,
contextual navigation, and public plugin slot inputs.
- Remove unused FeedCard and AgentDetail inputs and the test-only Git
push matcher alias.
- Keep the cleanup that erases historical plaintext chat retries from
browser storage.

Production source in the touched files is 106,800 → 105,633 nonblank
lines, down 1,167. The production Git diff adds 251 physical lines and
removes 1,480, down 1,229. Those figures exclude tests, fixtures, docs,
and Storybook. The entire Git diff adds 482 physical lines and removes
2,325, down 1,843.

## Verification

- Merged master at `b56f240f45` into the PR branch as `f73fea4009`. The
five sidebar files retain the expanded navigation cleanup and master’s
explicit session-replay masking contract. The merge also preserves
master’s account-repair recovery guard and GitHub star-prompt behavior.
- Focused UI Vitest run passed 361 tests across 11 files. This covers
analytics privacy and replay masking, sidebar and layout, GitHub star
prompt, and agent detail.
- Focused server Vitest run passed 45 tests across three files. This
covers restored configuration handling, conversation recovery, and Agent
Chat authority and receipt replay.
- `pnpm --filter @paperclipai/ui typecheck`, the Git push guard and its
14 tests, and `git diff --check origin/master...HEAD` passed.
- `pnpm verify --base origin/master` classified the Git-push guard as a
global CI input and returned exit 2. GitHub CI supplied the broader
coverage.
- GitHub CI and Runner image checks passed on `f73fea4009`. Both
required checks, `ci / verify` and `ci / e2e`, passed. [CI
workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261752)
· [Runner image
workflow](https://github.com/paperclipai/paperclip/actions/runs/44181261413)
- Standard Greptile reported 5/5 with no actionable findings or open
review threads. [Greptile
review](https://github.com/paperclipai/paperclip/pull/22055#issuecomment-6959078925)
- The optional Superagent Security Scan finished neutral with an
inconclusive workflow result and zero annotations. The final rollup had
61 successes, five skips, and one neutral, with no failed or pending
checks. [PR
checks](https://github.com/paperclipai/paperclip/pull/22055/checks)

## Risks

Agent Chat authorization and retry are sensitive to small query changes.
The final diff keeps the conversation agent field and all existing
authorization and idempotency checks. The tests cover revoked defaults,
legacy migration reads, bound-listener authority, and mobile and desktop
navigation. Installations that still run software old enough to read the
per-method AI table will no longer see new defaults in that table.
Current Core reads the provider-scoped table.

## Model Used

OpenAI Codex, GPT-6.1 Sol (`gpt-6.1-sol`), with source inspection and
shell-based tests.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 14:00:14 -08:00
Devin FoleyandPaperclip f5f60cfcb2 fix(server): name why agent files could not be excluded from task Git staging (#21499)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, Paperclip prepares the agent's files in
the sandbox and keeps the reserved runtime directory out of task Git
staging
> - A short program in the sandbox writes the Git exclusion. When it
failed, the run error said only "Could not exclude agent files from task
Git staging"
> - The error dropped the exit code, the timeout state and the program
output. Many staging runs failed with this message, and none of them
showed the cause
> - This pull request adds a closed reason code to the error
> - The benefit is that the next failures show what to fix, without
leaking sandbox paths or user names

## Linked Issues or Issue Description

**What happened?**
Many staging runs failed in the preparing stage with "Could not exclude
agent files from task Git staging". The error had no cause.

**Steps to reproduce**
1. Start a remote run where the Git exclusion program fails (for
example, it times out, Node is missing, or the exclusion file is
unsafe).
2. Read the run error.

**Expected behavior**
The error names why the exclusion failed.

**Actual behavior**
The error gives no reason, so the failure cannot be diagnosed.

## What Changed

- `agent-instruction-files.ts`: add
`instructionGitExcludeFailureReason`. It maps the result to a closed
reason: `timed_out`, `unsafe_directory`, `unsafe_file`, a Node error
code such as `enoent` or `eacces`, `node_unavailable`, `signal_<name>`,
`exit_<code>` or `unknown`.
- `agent-directory-working-copies.ts`: append `(reason=<code>)` to the
existing error message.
- The reason never copies sandbox output, paths or user names.
- Tests: run the real exclusion program against failing fixtures and
check the reason.

## Verification

- `pnpm vitest run
server/src/__tests__/agent-instruction-git-exclusion.test.ts`: 16/16
pass.
- Working copy suite: 136/136 pass.
- These local results are from the earlier master base. The rebase onto
the current master had no conflicts. CI verifies the current head.

## Risks

- Low. The change adds a suffix to one error message. It does not change
when the error is thrown.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code execution
in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
2026-11-17 13:56:09 -08:00