Commit Graph
7128 Commits
Author SHA1 Message Date
DottaandClaude Opus 5.5 fe72bc5f81 fix(native): roll back a warm rotation cut off before its new control plane once its runner is proven dead (#20218)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A warm follow-up rotates authority before it attaches. It archives
the previous epoch (suspended runner state and settled controller) under
`authority-epochs/`, then writes its own control plane
> - #20146 taught recovery to roll back a rotation whose new control
plane held only an undelivered `run.attach`, once its runner was proven
dead
> - A hard kill can land one step earlier, after the archive and before
the new control plane exists. The root then holds only the archive, with
no active control plane and no runner state. The executor read "no
durable identity" and failed with `runner_state_identity_mismatch:
durable_identity_unreadable`
> - This pull request applies the same proven rollback to that earlier
point of the rotation
> - The benefit is that a hard kill anywhere in the rotation window
resumes the follow-up from the previous turn

## Linked Issues or Issue Description

Refs #20146

**What happened?**
Local mixed restart-chaos wave w-20261109T081456-s9901 (master
09e24dd12c), t15 (codex M1):
- Run `4b85497f` started at 08:29:32 as a warm follow-up and requested
its process, then K1 killed the server.
- Recovery stopped the previous run's warm runner (pid 60249) and
claimed `resume_dead_runner`.
- The executor failed with `runner_state_identity_mismatch:
durable_identity_unreadable`.

The quarantined root held:
- `authority-epochs/epoch-c2842347…` with the previous owner `dc7d447b`:
runner `suspended`, every command completed;
- no `control-plane/` state and no `runner/runner-state.json`, only
provider state.

A Retry ran fresh. Phase 3 also saw the same signature on Daytona;
#20214 covers that remote shape.

**Expected behavior**
For a local runner proven dead (`resume_dead_runner`), a root that holds
only the archive of a suspended previous owner of this session restores
that epoch, and the follow-up continues from the previous turn. Without
a dead-runner claim it still fails closed.

**Steps to reproduce**
1. Start a warm follow-up and hard-kill the server after the rotation
archives the previous epoch, before it writes the new control plane.
2. Restart. Before this change, the follow-up fails with
`durable_identity_unreadable`.

**Paperclip version or commit**
09e24dd12c (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`:
- `undeliveredAttachRotationSource` becomes `interruptedRotationSource`,
and the new `latestSuspendedArchivedEpoch` shares the epoch checks:
named by its identity digest, a matching suspended runner state.
- Besides #20146's undelivered-attach shape, it accepts a root with no
`runner-state.json` and no active control plane (absent or empty). In
that case the epoch must belong to this session (`normalizedSessionId`)
and to a different run.
- The rollback moves an active control plane aside only when one exists.
- `native-recovered-owner-followup.test.ts`: #20146's rotation test adds
the "cut off before its new control plane" shape. With a dead-runner
claim it is admitted from the restored previous owner; without the claim
it still fails closed.

## Verification

- Red/green: on master, the new admitted case fails with exactly
`durable_identity_unreadable`. With the change it is admitted. The
fail-closed cases are unchanged.
- `vitest run` of the recovered-owner, restart-recovery and bootstrap
test files and `native-session-executor.test.ts`: 8 files, 1627 tests
passed (14 skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk:
- The rollback still requires a `resume_dead_runner` claim on a local
runner, an absent runner state, and an archived epoch of this session
whose runner is suspended and whose name matches its identity.
- Restoring only moves files back within the root, and nothing is
deleted.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:20:23 -06:00
DottaandClaude Opus 5.5 ad57d5cc12 fix(native): restart an unclaimed remote follow-up that carries only its inherited checkpoint (#20214)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote per-turn follow-up inherits the previous run's session
checkpoint at admission, rebound to its own identity. Its session root
still holds the previous owner's remote marker until the follow-up
rotates authority
> - If a restart lands before the follow-up's controller ever claims
execution, recovery should restart its bootstrap: nothing of this run
reached any runner
> - Recovery's unclaimed-bootstrap check required no checkpoint and no
provider evidence. The inherited checkpoint failed both, so recovery
claimed `reattach_remote_runner`
> - Even under a bootstrap claim, the executor skipped the per-turn
projection restore for every restart claim. It therefore never
rehydrated the previous owner's external marker, and failed with
`runner_state_identity_mismatch: durable_identity_unreadable`
> - This pull request does not count the inherited checkpoint as
evidence for an unclaimed remote run, and lets a bootstrap claim restore
the previous owner's projection, as a fresh follow-up start does
> - The benefit is that a follow-up drained in its first second on
Daytona restarts on its own

## Linked Issues or Issue Description

Refs #20153

**What happened?**
Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503
(revision 9046884811), t26 (codex C1): follow-up run `f1ddc43d` started
at 07:54:18 and was drained by SIGTERM at 07:54:19, during workspace
staging.
- It had not requested a runner process, and its coordinator was never
claimed (provider attempt 0).
- Recovery claimed `reattach_remote_runner` (`hasCheckpoint: true`,
`hasProviderEvidence: true`, both from the inherited checkpoint).
- The executor failed with `runner_state_identity_mismatch:
durable_identity_unreadable`, because the session root held the previous
run's external marker. A Retry succeeded.

**Expected behavior**
An unclaimed remote run whose only checkpoint is the admission-rebound
one, with no durable controller record and no provider events, restarts
its bootstrap (`remote_execution_never_claimed`). The executor
rehydrates the previous owner's projection and starts the follow-up from
it.

**Steps to reproduce**
1. Start a remote per-turn follow-up whose session root holds the
previous owner's external marker. Drain the control plane before the
follow-up's controller claims execution.
2. Restart. Before this change, recovery claims `reattach_remote_runner`
and the executor fails with `durable_identity_unreadable`.

**Paperclip version or commit**
9046884811 (wave). Branch rebased on current master.

## What Changed

- `native-restart-recovery.ts`:
- #20153's admission-rebound checkpoint test moves earlier. A new
`inheritedCheckpointOnly` holds for a remote sandbox run (not a plugin
task) with only that checkpoint, no durable controller record, no staged
attachment and no provider events.
- The unclaimed-bootstrap path and its classification then treat that
checkpoint as neither a checkpoint nor provider evidence.
- `native-inherited-runner-retry.ts` / `native-restart-recovery.ts`: a
warm continuation carries its settled predecessor's runner process
identity until it launches its own runner, and t26's row still held pid
596 at the restart.
- The new `carriesInheritedRunnerIdentity` reuses the same-run retry
path's proof (the exact pid, group and start time a settled predecessor
recorded).
- The restart classifier then ignores that inherited identity on the
unclaimed path, for a run that carries only its inherited checkpoint.
  - A process identity that is the run's own still holds the run.
- `native-session-executor.ts`: `restorePerTurnRunnerdController` now
also runs for a `bootstrap_incomplete` claim, but only when the root's
reference identity belongs to a previous run. A claim on the run's own
root returns early, as before.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: an unclaimed
remote follow-up with only its inherited checkpoint restarts, with no
process identity and with its predecessor's runner identity. It stays
held with its own runner identity, or when a provider event was
projected;
- `native-recovered-owner-followup.test.ts`: a bootstrap claim over the
previous owner's external marker is admitted, rehydrating the marker,
keeping the backup and quarantining nothing.

## Verification

- Red/green: both new tests fail on master. The classifier test claims
`reattach_remote_runner`, and the executor test fails with exactly
`durable_identity_unreadable`. Both pass with the change, and the held
case holds.
- `vitest run` over the restart-recovery, bootstrap, recovered-owner and
failed-workspace-recovery test files and
`native-session-executor.test.ts`: 9 files, 1715 tests passed (14
skipped). After the inherited-identity change, the restart-recovery,
bootstrap, recovered-owner, failed-workspace-recovery and
inherited-runner test files: 8 files, 407 tests passed (14 skipped).
Without the identity check, the inherited case is not restarted.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk:
- The inherited checkpoint stops counting only when the database shows
no controller record and no provider event for the run, and the existing
unclaimed-bootstrap conditions all still hold.
- The executor change runs the existing fresh-follow-up projection
restore, with all of its own checks, only for a root that still belongs
to the previous run.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:10:23 -06:00
DottaandClaude Opus 5.5 98e0f1fc0d fix(native-runtime): a stopped run cannot close a task another run continues; deliver a pre-restart comment at once (#20216)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When the board stops a run and then posts a comment asking the agent
to continue, that comment starts a new run, and only that new run should
decide when the task is done
> - In restart chaos wave `w-20261109T081456-s9901`, t16 (claude S1, K1)
closed the task as done without the line the user asked for, and t10
(codex S1, K6) never woke a run for the continue comment, so it needed a
Retry
> - In t16, the stopped run's saved "done" result was accepted 10 s
after the continue run started. In t10, a restart killed the server 21
ms after the comment committed, before its wake was written, and the
comment backstop then waited out its 30 s grace
> - This pull request stops a run from closing a task that another run
now holds, and makes the backstop deliver at once a comment that
predates the server process
> - The benefit is that a user's latest direction after a Stop is
neither overridden by the stopped run nor lost to a restart

## Linked Issues or Issue Description

Refs #19831. Restart chaos wave `w-20261109T081456-s9901` (master
`09e24dd12c`), tasks t16 and t10.

**t16, from the kept database (CHA-20):**
- 08:29:19: the board stopped run 1 (`6e33d123`). Its saved result
reported done, but finalization failed on the stop
(`finalization_failed_claim_preserved`, a retryable failure).
- 08:29:41: the user's continue comment started run 2 (`2cb71c5b`). Run
2 held the task's execution and saw the comment.
- 08:29:51: the finalization reconciler retried run 1's saved result and
committed `completion_claim_policy_accepted` for run 1. The task closed
as done.
- From then on, every Paperclip tool call from run 2 was refused
(`paperclip_runner_tool_binding_not_authorized`), so it could not add
the `CONTINUE-…` line. Its result (blocked) was then preserved as
`terminal_status_preserved`.
- Run 2 ending `succeeded` with `native_session_lease_lost` follows from
this: its result was accepted before the lease was lost. The comment was
delivered; the model saw it.

**t10 (CHA-14):**
- The Stop moved the task to blocked. The user's comment reopened it to
todo at 08:26:28.633, and K6 killed the server at 08:26:28.654, before
the route wrote the comment's wake. No wake request exists for that
comment.
- This is not caused by the Stop, #20066 or #20180: the wake is written
after the comment commits, and the kill landed in between.
- The I4 backstop ("a user comment without a wake") matched the comment,
but treated it as possibly still getting its normal wake for 30 s. The
pass interval then pushed the check to 02:27:16, and the harness clicked
Retry at 02:26:59.

## What Changed

- `status-arbiter.ts`:
  - New `heldByAnotherRun(issue, runId)`.
- `supersedeStaleOwnerCompletion` also keeps status, with reason
`completion_run_superseded`, when the issue's `executionRunId` is a
different run. A run that was reassigned away still gets
`completion_owner_superseded`.
- `native-run-finalizer.ts` and `native-status-reassessment.ts` pass the
run id to that guard.
- `status-decision-committer.ts` rechecks the same rule under the issue
lock: a done decision for a non-conversation issue held by another run
raises `NativeStatusRaceError`. The finalizer then re-arbitrates and
keeps status.
- `ownership-invariant-report.ts`: the comment-without-wake grace now
only applies to comments created after this server process started
(`getServerProcessStartedAt`). Separate live server replicas are not
supported, and the wake is written by the process that accepted the
comment, so an older comment is owed at once. The startup pass now
delivers it. `runOwnershipReconciliationPass` takes an optional
`processStartedAt`.

## Verification

Red/green:
- `native-stale-owner-completion.test.ts`: three new tests.
- The unit policy: another run holding the task keeps status; the same
run, or none, still closes it.
- "does not let a stopped run's saved done result close the task a newer
run is continuing" (the t16 shape).
- "does not close the task when a newer run takes it after arbitration
but before the status commit" (only the committer recheck covers this).
  - On master: 3 failed. With this PR: 13 passed.
- `ownership-reconciliation-report.test.ts`: "finds at once a young user
comment whose wake a restart lost after the board stopped the run" (the
t10 shape). It fails on master and passes with this PR; the same comment
in a process that accepted it still waits for its grace deadline. The
file: 27 passed.

Regression and static checks:
- 16 nearby native finalization, status projection, workspace finalizer
and arbiter suites: 439 passed.
- `tsc --noEmit` passes for `server`, and for the two test files.

## Risks

- A done result now waits when a different run holds the task. That run
reaches its own decision; the earlier result stays as history. If the
holder is stale, the existing execution-lock recovery clears it, and a
later reassessment can then close the task.
- After a restart, a user comment newer than the task's last run, with
no wake, is delivered at the startup pass rather than 30 s later. A
replica that is still live and writing the same wake is not a supported
topology.
- No schema change and no change to persisted run state.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 03:03:15 -06:00
DottaandClaude Opus 5.5 088bdf7cfa fix(daytona): wait out a sandbox state change before starting it for workspace realize (#20212)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Daytona, every run realizes its workspace in the leased sandbox.
Realizing it first makes sure the sandbox is started
> - Right after a control-plane restart, the sandbox can still be
mid-transition, for example stopping or starting from the previous
lifecycle step
> - The plugin called `sandbox.start` straight away. Daytona rejects
that with "Sandbox state change in progress", so the run failed
`setup_failed` and needed a Retry
> - This pull request waits out a sandbox's state change before starting
it, and retries a start the provider rejected for that reason, within
the existing start deadline
> - The benefit is that a run whose sandbox is briefly transitioning
starts on its own

## Linked Issues or Issue Description

Refs #20211

**What happened?**
Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503
(revision 9046884811), t07 (claude A1): run `5aabeb55` failed at
07:45:27 with `setup_failed` and "Failed to realize workspace for
environment … (sandbox): Sandbox state change in progress". The sandbox
was mid-transition after the restart, and realize neither waited nor
retried. A Retry succeeded.

**Expected behavior**
Realize waits for a transitional sandbox state to settle, then starts
the sandbox. If the provider still reports the state change, it retries,
all within the existing start deadline. Any other start error fails at
once, and a transition that outlasts the deadline still fails.

**Steps to reproduce**
1. Realize a workspace while the Daytona sandbox is `stopping` or
`starting`, or while `start` returns "Sandbox state change in progress".
2. Before this change, realize fails at once with that error.

**Paperclip version or commit**
9046884811 (wave). Branch rebased on current master.

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`:
`ensureSandboxStarted` now polls `refreshData` while the sandbox is in a
transitional state.
- The transitional states are `starting`, `stopping`, `restoring`,
`archiving`, `creating`, `pending_build`, `pulling_snapshot`,
`building_snapshot` and `resizing`.
- It polls every second, until the start deadline from `timeoutMs`. Each
read keeps the existing liveness bound.
- If the start fails with "state change in progress", it refreshes and
tries again, within the same deadline.
- The wait, every retry and the start share one deadline. Each read is
bounded by the time left, and the start receives only the remaining
seconds.
- As soon as any state other than `started` is observed, the lease's
stored session is dropped before waiting, so a restart that happened
outside the plugin never reuses a dead session.
  - Starting a settled sandbox (`startSettledSandbox`) is unchanged.
- `plugin.test.ts`:
  - realize waits for a stopping sandbox, then starts it;
  - realize retries a start that was rejected for a state change;
  - realize gives up once the transition outlasts the deadline;
  - another start error fails at once;
  - the start receives only the budget the wait left;
- a restart outside the plugin that the wait sees through opens a fresh
session.
  - The tests stub the API key with `vi.stubEnv`.

## Verification

- Red/green: on master, the wait and the retry cases fail with "Sandbox
state change in progress". With the change they pass, and the deadline
and other-error cases hold.
- `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14
files, 614 tests passed (8 skipped).
- `tsc --noEmit` for the Daytona plugin package: passed.

## Risks

- Low risk. The wait is bounded by the start deadline the plugin already
used. Only transitional states and the provider's state-change rejection
are waited out, and every other error behaves as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.9
2026-11-09 03:00:14 -06:00
DottaandClaude Opus 5.5 6383eb8033 fix(authz): require board authority for agent budget changes on every route (#20213)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Budgets stop agent spending. The budget hard stop is a control-plane
invariant
> - The budget routes (`PATCH /api/agents/:id/budgets`, `PATCH
/api/companies/:id/budgets`) are board-only
> - `PATCH /api/agents/:id` also accepts `budgetMonthlyCents`, and the
configuration service writes the agent budget policy from it. That route
had no board check for the field
> - So an agent, such as a CEO, a manager or the agent itself, can
change a monthly budget without a board user. This is a second route
around a board-only control, like the pause bypass that #20164 fixed
> - This pull request applies the budget route authority to every route
that writes an agent budget, and gives agents the final board-only
denial with a link to the budget page (#20151)
> - The benefit is that only the board changes budgets, and an agent
that asks gets a final answer that it can pass to the person

## Linked Issues or Issue Description

Refs #17857 (earlier fix, now in conflict with master; this PR rebases
it and adds the budget page link). Refs #20164 and #20151 (same pattern
for PATCH status and board-only denials).

**What happened**

An agent can send `PATCH /api/agents/{id}` with `budgetMonthlyCents`.
The route accepts it, and the configuration service writes the agent
budget policy. This works for the agent's own budget and, with update
permission, for other agents. The same field also reaches budget policy
through a configuration rollback, direct agent creation, an immediate
hire and a built-in agent provision. A chat eval
(`budgets-costs.resolve-budget`) found the route.

**Expected behavior**

Only a board user can change an active budget, as on the dedicated
budget routes. An agent gets `403` with code `board_access_required`.
The denial says it is final and links the agent's budget tab
(`/<prefix>/agents/<id>/budgets`) or the company budgets page
(`/<prefix>/activity/budgets`). An agent can still propose a budget in a
hire that waits for board approval. The approving board user activates
it.

**Steps to reproduce**

1. On master, authenticate as an agent of a company (for example the
CEO).
2. Send `PATCH /api/agents/{agentId}` with body `{"budgetMonthlyCents":
5000}`, for its own id or for another agent.
3. See `200`. The agent budget policy changes with no board action.
4. The new route tests reproduce this: on master source they fail with
`expected 200 to be 403`.

## What Changed

- `PATCH /api/agents/:id`: a changed `budgetMonthlyCents` requires a
board actor. An unchanged amount with other edits still works.
- Configuration rollback: a rollback that changes the budget requires a
board actor.
- `POST /api/companies/:id/agents` and an immediate `agent-hires`: a
positive budget requires a board actor. A hire that needs board approval
keeps the proposed budget and does not activate it before approval.
- Built-in agent provision: a budget override requires a board actor.
- `prepareAgentConfiguration` rejects an agent-originated budget change
again (defense in depth). An agent rollback with an unchanged amount
does not rewrite the policy.
- Board writes keep the actual operator identity. The initial hire
budget is saved in the hire transaction.
- `assertBoard` and `boardAccessRequired` take an optional board page.
Budget denials link the agent's budget tab or the company budgets page,
not the general agent profile.
- Agent access map updated.
- This includes the four commits of #17857 (f970f6b01f, 8c06aba41c,
e5682d22fa, b6cd49d4b6), rebased onto master with the conflicts
resolved, then squashed.

## Verification

- Red on master source, same tests: 12 budget tests fail. The PATCH
tests fail with `expected 200 to be 403` for the agent's own budget and
for a CEO on another agent. Built-in provision fails with `expected 200
to be 403`. Creation and immediate hire fail with `expected 201 to be
403`.
- Green with the fix: `cd server && npx vitest run
src/__tests__/agent-permissions-routes.test.ts
src/__tests__/built-in-agent-routes.test.ts
src/__tests__/built-in-agents.test.ts
src/__tests__/agent-hire-idempotency-routes.test.ts
src/__tests__/board-access-denial.test.ts
src/__tests__/agent-budget-configuration.test.ts`: 222 passed.
- `src/modules/agent-lifecycle/__tests__/lifecycle.test.ts` (embedded
Postgres): 76 passed.
- `agent-skills-routes`, `agent-adapter-validation-routes`,
`openapi-routes`, `agent-cross-tenant-authz-routes`: 275 passed.
- `node scripts/check-maps.mjs`: passed. `git diff --check`: passed.
- Server `tsc --noEmit`: no errors in changed files. One error in
`native-session-executor.ts` comes from a linked runner build in the
local worktree, not from this change. CI runs the full typecheck.

## Risks

- Agent clients that set an active budget directly now get `403`. They
must propose the budget in an approval-gated hire, or ask a board user.
- Agent creation with no budget or a zero budget is not changed. Pending
hire budgets do not activate before approval.
- Trusted internal plugin provisioning keeps its prior behavior and gets
no board identity.
- No schema or migration change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution, in Claude Code. The rebased commits from #17857 were first
written with OpenAI `gpt-6.1-sol` through Codex.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:50:16 -06:00
DottaandClaude Opus 5.5 b1049b174e fix(native): retire a finished remote owner whose suspend never finished once its sandbox leases ended (#20211)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a per-turn follow-up starts a new runner, it retires the
previous turn's finished owner. Retirement requires the owner's
controller journal to be settled
> - A local runner that is proven stopped can no longer act on its own
lifecycle commands, so an unfinished `runner.suspend` from it does not
hold the follow-up
> - On a remote per-turn sandbox the same thing happens when a restart
or drain lands while the runner is suspending. But the remote path had
no equivalent rule, so the unfinished `runner.suspend` held the
follow-up even after the previous run's sandbox lease had ended
> - This pull request treats a remote owner's lifecycle commands as
settled once every sandbox lease of that run has been released or
expired, and none of them is the sandbox the follow-up now uses
> - The benefit is that a C1 follow-up after a restart on Daytona
continues on its own

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Daytona per-turn restart-chaos K13 storm w-20261109T074128-s9503
(revision 9046884811, image 01246d43): 60 of 65 passed. Three C1
follow-ups failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(runner.suspend.pending)`:
- t04 codex, run `c7446d0c`;
- t16 claude, run `4b2c770f`;
- t39 claude, run `3ef7d934`.

For t04, the previous owner `fc585314` committed its `run.terminal`, and
its journal kept only `runner.suspend` pending. Its per-turn lease
(`5576f27d…`) expired at 07:43:52, and the follow-up ran on a new
sandbox. Each Retry succeeded.

**Expected behavior**
Once the previous run's sandbox leases have all ended, and the follow-up
runs on another sandbox, the previous runner cannot act on its suspend.
The finished owner is retired and the follow-up continues.

The owner stays held while any of these holds:
- a lease is active, retained or pending cleanup;
- the follow-up runs on the same sandbox;
- no lease is recorded.

**Steps to reproduce**
1. Finish a remote per-turn run whose controller journal keeps
`runner.suspend` pending after its committed run terminal, and let its
sandbox lease end.
2. Start a follow-up on a new sandbox. Before this change, retirement
refuses with `journal_commands_unsettled(runner.suspend.pending)`.

**Paperclip version or commit**
9046884811 (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`:
- the new `priorRemoteSandboxLeasesEnded` checks the previous run's
`environment_leases`: at least one lease, every one `released` or
`expired` with `releasedAt`, and none on the current target's provider
lease;
- `inspectFinishedPerTurnRunnerdOwner` passes the result as
`remoteRunnerGone` to `readSettledRunnerdProjection`, for the live root
and for quarantined copies. That function then ignores unfinished runner
lifecycle commands (`runner.suspend`, `runner.drain`,
`runner.shutdown`), exactly as it already does for a stopped local
runner.
- `native-recovered-owner-followup.test.ts`: a red/green test with lease
variants. A lease that ended or was released on another sandbox is
admitted. A retained lease, an ended lease on the same sandbox, or no
lease record stays held.

## Verification

- Red/green: on master, the two admitted cases fail with exactly the
production error; with the change they are admitted. The three held
cases hold on both.
- `vitest run` of the recovered-owner, remote-warm and restart-recovery
test files and `native-session-executor.test.ts`: 7 files, 1634 tests
passed (14 skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk. Only runner lifecycle commands become settled, and only when
the database proves every sandbox of the previous run has ended and the
follow-up runs elsewhere. Unfinished turn, tool or result commands, a
missing run terminal and unanswered tool input still hold.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:40:13 -06:00
DottaandClaude Opus 5.5 fc5c79a327 fix(guidance): tell agents who acts and which page to link for person-only steps (#20206)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents in chat read the API catalog (`search_api`) and
`connections_search` to learn what they can do and who must act
> - Some requests need a step that only a person can take: connect an
assistant, allow a quarantined app action, turn on an experiment, enter
an API key, or finish their own connection request
> - For these steps, no catalog entry or search result said who acts or
which page to link, and none of them returns a board-only 403, so the
denial guidance from #20151 and #20182 does not reach them
> - The agent knew a person had to act, searched for the page until its
turn ran out, and never replied
> - This pull request puts the fact and the exact page link where the
agent already looks, following #20152 and #20170
> - The benefit is that the agent replies at once with the right page
instead of timing out

## Linked Issues or Issue Description

Refs #20151, #20152, #20170, #20182

**What happened**

In chat, five requests ended with no reply. The agent searched the API
until the turn deadline:
- "Connect my assistant so it can see our tasks." `GET /api/mcp/setup`
is refused for agents. Its catalog entry had no description, so the
refusal text "Its catalog description says who can use it" pointed at
nothing.
- "Use the app's new export feature." The new action was quarantined. No
catalog entry said where a person reviews it.
- "Let me chat with Morgan over iMessage." `connections_search` hides
chat channels while the Chat connectors experiment is off, so the agent
could not find iMessage or the setting that hides it. `POST
/api/companies/{companyId}/chat-endpoints` also accepted a draft
endpoint for the turned-off provider.
- "Connect my account using an API key." The service was the person's
own MCP server. `connections_search` returned no result and did not name
the secure "Connect your own MCP server" page.
- "I will finish Alex's connection request for them." No catalog entry
said that only the addressee can resolve a connection request card.

**Expected behavior**

The agent says who must act, links the exact page, and replies in the
same turn. It does not search further, does not ask for a key in chat,
and does not create another request.

**Steps to reproduce**

1. Ask a chat agent one of the five requests above, in the matching
state (for example, Chat connectors off, or a quarantined catalog
entry).
2. Watch the run events. The agent calls `search_api` and `call_api`
until the turn deadline and posts no reply.

## What Changed

- `server/src/routes/openapi.ts`: add catalog descriptions that say who
acts, link the page, and say not to search further:
- `GET /api/mcp/setup` and `POST /api/mcp/requests/{id}/consent` link
[Assistant connections](/<prefix>/apps/assistant-connection) and say the
consent screen offers only access within the person's own role.
- `GET /api/tool-connections/{connectionId}/catalog`,
`.../review-counts` and `GET
/api/companies/{companyId}/tools/apps/attention` say a quarantined
action waits for a person on
`/<prefix>/apps/<connectionId>/permissions`.
- The four `/api/connection-intents/{interactionId}/...` routes say only
the addressee can resolve the card, and link the task that holds it.
- `server/src/services/connection-intents.ts`:
- While Chat connectors is off, a search that names a chat-only channel
(for example iMessage) returns it as `unavailable`, with a reason that
links `/<prefix>/company/settings/instance/experimental`. Channels that
also offer tools, such as Slack, keep their current result.
- A single `unavailable` result now always gets the "explain the reason,
do not bypass" instruction.
- When no listed service matches and the person gave the service's own
server URL, the instruction links `/<prefix>/apps/byo`, the secure
"Connect your own MCP server" page, and says never to ask for a key in
chat.
- `server/src/routes/chat-channels.ts`: `POST
/api/companies/{companyId}/chat-endpoints` returns 403 with the
Experimental settings page while the provider's experiment is off (Chat
connectors, or GitHub review bots for GitHub). AgentMail is unchanged.
The catalog entry says the same. The UI already hides the setup pages in
this state.
- `GET /api/issues/{id}/interactions` says only the addressee resolves a
connection request card.
- A hidden channel that an aggregator can serve (for example Discord)
keeps its aggregator routes instead of the unavailable result.
- Tests: `person-only-page-guidance.test.ts` checks the catalog text and
that every named page is a route that `ui/src/App.tsx` mounts.
`connection-intents-service.test.ts` covers the hidden channel (off and
on) and the own-server page.

## Verification

- `cd server && npx vitest run
src/__tests__/person-only-page-guidance.test.ts
src/__tests__/connection-intents-service.test.ts` passes.
`chat-channels.integration.test.ts` passes with `-t` for every test that
posts to the create route.
- These related suites pass: `openapi-routes`,
`connection-aggregator-fallback`, `generic-mcp-connection`,
`assistant-connections-guidance`,
`company-model-service-settings-guidance`, `board-access-denial`,
`email-channels.integration`, and the eight suites that import
`runner-api-catalog`.
- `npx tsc --noEmit -p server` shows no errors in the changed files.
- End to end: the full-stack chat evals for the five requests ran on a
branch with this change. The run results are in the comments below.

## Risks

- Low risk. The change is text in the API catalog and in
`connections_search` results, plus one result for a channel that was
hidden before.
- A hidden channel now shows in search as `unavailable` with no methods.
It cannot be set up from the result, so the gate stays in place.
- Chat endpoint creation is now refused while the experiment is off.
Callers that created endpoints with the setting off now get 403. Three
integration tests that relied on that now turn the setting on first, and
a new test covers the refusal.
- The single-`unavailable` instruction now also applies in suggestion
mode. Before, that mode told the agent to "respect unavailable states"
with no reason.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:36:57 -06:00
DottaandClaude Opus 5.5 9c18cf61da fix(runner): keep discarded OpenCode token deltas from delaying tool calls (#20209)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native OpenCode runs use a TypeScript app-server proxy between
OpenCode and runnerd
> - runnerd reads the proxy's frames in order, only after the controller
acknowledges earlier events
> - For each frame, runnerd rewrites its whole durable provider state
with fsync, including every tool result that the turn retains
> - The proxy sent one `item/delta` frame for each reasoning or
tool-output token, and runnerd discards `item/delta`
> - A Paperclip tool call request therefore waited behind hundreds of
discarded frames, and the wait grew with each tool result in the turn
> - This pull request stops the proxy from sending the token deltas that
runnerd does not consume
> - The benefit is that tool calls reach runnerd without that queue, so
agent turns finish faster

## Linked Issues or Issue Description

**What happened**

In the chat eval `plugins.configure-plugin` (runner-opencode,
deepseek-v4.1-flash, GitHub Actions run 42574640038),
`paperclip_search_api` calls in one turn slowed from about 1.4 s to 9-15
s each. The server handled every call in 50-250 ms (`tool.execution`
spans). For one `call_api` call, 6.3 s passed between the server's 403
and the next agent activity. Time to first token was about 4.4 s.

The run events show the cause. Events that OpenCode streams reach
runnerd in bursts. In one burst, the usage events for three different
model requests arrived within 0.5 s, which no model can do. Each burst
ends with the next tool call request. 143 of 203 recorded
runner-opencode runs that we checked show such bursts, and 108 show a
gap of 8 s or more. Many chat eval cases that time out use this runner.

The mechanism:

1. OpenCode streams reasoning one token at a time. The proxy forwarded
each token as its own `item/delta` frame.
2. runnerd reads proxy frames in order and only after the controller
acknowledges earlier events. For each frame it calls `save_state`. That
serializes the whole provider state, with the tool results retained in
the turn, and fsyncs it.
3. runnerd's Codex facade has no mapping for `item/delta`, so it
discards the frame after the save.
4. A tool call request is one more frame in the same ordered stream. It
waits until runnerd works through every earlier delta frame. The wait is
about (reasoning tokens) x (save cost), and the save cost grows with
each retained tool result.

**Expected behavior**

A Paperclip tool call reaches runnerd soon after OpenCode makes it. Its
delay does not grow with the reasoning length or with the number of
earlier tool calls in the turn.

**Steps to reproduce**

1. Run a native OpenCode chat turn with a reasoning model that makes
many sequential Paperclip tool calls, for example `search_api`.
2. Compare each tool's `item.completed` tool_result time with the
`tool.execution.completed` time and with the next `item.started`
tool_use time in the run events.
3. See that the gap grows during the turn, and that events for several
model requests arrive within less than one second.
4. For a local model of the mechanism: run the pinned OpenCode binary
with a controlled OpenAI-compatible provider that streams 300 reasoning
tokens before each tool call. Pass the driver events through the proxy's
mapping and batching into a reader that consumes one frame at a time
with a save cost that grows by 4 ms per completed tool result. Before
this change, 8 sequential calls waited 1.7, 3.1, 4.2, 5.4, 6.7, 7.9, 9.2
and 10.5 s behind 142-261 queued frames. After this change, they waited
64-93 ms behind 0 frames.

## What Changed

- `shouldForwardOpenCodeProxyItem` now takes the event type and payload.
For `item.delta`, it forwards only assistant text (`agentMessage` or
`text`), which runnerd maps to `item/agentMessage/delta`.
- The proxy keeps reasoning and tool-output token deltas. Completed
reasoning and tool items still go to runnerd as `item/completed`. Tool
progress still goes as `item/updated`.
- New regression test: a step with 1,000 streamed reasoning and
tool-output tokens puts the same number of frames before the tool call
request as a step with 1 token, and no `item/delta` frame is written.
- New test: assistant text deltas are still forwarded. Completed
reasoning items are still forwarded.

## Verification

- `pnpm exec vitest run src/cli/opencode-proxy-events.test.ts
src/cli/opencode-proxy-output.test.ts` in `packages/paperclip-runner`:
15 passed.
- Red/green: with the predicate change reverted, the two new tests fail.
With the change, they pass.
- `tsc --noEmit -p tsconfig.test.json` in `packages/paperclip-runner`
has no errors in the changed files. Local errors exist only in
`src/eval/workflow-harness.ts`, which needs a built
`@paperclipai/paperclip-eval-kernel`.
- Local before/after measurement with the real pinned OpenCode binary
(`opencode-ai@1.18.34`), described in Steps to reproduce, step 4.
- Not run locally: the suites that need the `paperclip-runnerd` debug
binary or the bundled provider entrypoints
(`runnerd-opencode-streaming-tools`, `runnerd-codex-transport`,
`opencode-proxy-thread-read`). They fail the same way on the unchanged
base in this worktree because those build outputs are missing. CI covers
them.

## Risks

- Low risk. The changed frames were discarded by runnerd. No persisted
state shape changes, so no previous-release story is needed.
- If runnerd later maps a reasoning delta method, the proxy must forward
it with batching, as it does for assistant text. The test names this
boundary.
- This change does not change the per-frame durable save in runnerd.
Mapped frames (assistant text, tool items) still pay it. A runnerd
change that skips the save when a notification changes no state would
protect other proxies too. That is a separate change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), used through Claude
Code with tool use (shell, file edits, local benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 02:23:44 -06:00
DottaandClaude Opus 5.5 09e24dd12c fix(authz): say every plugin management step is board-only in plugin denials (#20207)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People ask a chat agent to do work, and some of that work is
board-only, such as plugin management
> - When an agent gets a board-only denial, the denial tells it to stop
and reply with a link to the board page
> - The denial only spoke about the one route that was denied. An agent
that read the plugin list first was not told that plugin configuration
and secret binding are also board-only
> - The agent then searched the API catalog for an agent route to
configure the plugin. It searched for 110 seconds, and its reply came
after the chat window closed
> - This pull request adds one fact to every board-only denial on a
plugin route: all plugin management steps have no agent route
> - The benefit is that the agent replies at once with the plugin
settings link, and does not search for a route that does not exist

## Linked Issues or Issue Description

Refs #20182 (the plugin denial link and "reply now" text that this
builds on).

**What happened**

In the chat eval `plugins.configure-plugin`, the person asked the agent
to finish configuring the preview plugin. The agent called `GET
/api/plugins` and got the board-only denial. Its reasoning said: "Denial
final. But ... perhaps the preview plugin is configured through a
different route accessible to agents". It then made 14 more catalog
searches for plugin config, credential and secret-binding routes. The
run did not reply before the 150 second chat window closed.

**Expected behavior**

After a board-only denial on any plugin route, the agent replies now. It
says that a board user must configure the plugin, and it links the
plugin settings page.

**Steps to reproduce**

1. Install a plugin that has a required secret setting, and do not
configure it.
2. In chat, ask an agent: "Finish configuring the preview plugin."
3. The agent calls `GET /api/plugins` and gets 403 "Board access
required".
4. Before this change, the denial does not say that configuration and
secret binding are also board-only, and the agent searches for another
route.

## What Changed

- `server/src/routes/authz.ts`: a board-only denial for an agent on any
`/api/plugins` route adds one sentence. The sentence says that listing,
installing, configuring (including choosing a secret for a plugin
setting), enabling, disabling, upgrading and removing plugins have no
agent route.
- Other board-only denials do not change. Plugin tools that are assigned
to an agent are not mentioned, so tool use is not discouraged.
- `server/src/__tests__/board-access-denial.test.ts`: a regression test
for the plugin list and plugin config denials, and a negative check for
a non-plugin denial.

## Verification

- `cd server && npx vitest run
src/__tests__/board-access-denial.test.ts` passes (26 tests).
- Without the `authz.ts` change, the new test fails.
- The chat eval `plugins.configure-plugin` is rerun on a case branch
that includes this commit. The result is in the PR comments.

## Risks

- Low risk. The change only adds text to the 403 body that agents get.
People and board actors get the same body as before.
- The sentence is a fact about the current routes. Every plugin
management route uses `assertBoard`, `assertBoardOrgAccess` or
`assertInstanceAdmin`. If a plugin management route becomes
agent-accessible later, this sentence must change too.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
nightly/v2026.1109.0-nightly.0 canary/v2026.1109.0-canary.8
2026-11-09 01:41:57 -06:00
DottaandClaude Opus 5.5 6fd46c0025 perf(budgets): judge Paperclip's share of restart recovery; report the provider's session load (#20183)
## Thinking Path

- Speed budgets protect the time to first visible agent activity, and
restart recovery is one of the judged metrics
(`native-speed-budgets.ts`, wave grader).
- In the B1 concurrency-6 chaos wave (`w-20261109T045320-s9807`), Codex
`restart_recovery` was p50 8.6 s and p95 11.2 s, against a budget of 6.5
s and 8.0 s. At concurrency 4 (K13) it was 3.8 s and 5.0 s.
- The preparation spans on a copy of the kept database break it down:
- Paperclip's share (ready → claim → prepare → runtime stage → runner
launch) is a steady 1.2–2.0 s.
- `provider.session.load`, the Codex app-server boot plus
`thread/resume` of long saved sessions, takes 3.0–6.3 s, 65–75% of each
recovery.
- Resumes run concurrently, and the worktree lock and the startup
reconciler are not on the path.
- Judging the full latency therefore mixes our code with provider and
session-size effects. Budget the part we control, so a regression in our
code shows up regardless of session size or concurrency, and report the
rest.

## Linked Issues or Issue Description

**What happened?**
`restart_recovery` breached its budget whenever many long provider
sessions resumed at once, although Paperclip's own recovery work stayed
within about 2 s. The breach did not point at anything Paperclip could
fix, and a real regression in our code could hide inside the
provider-load variance.

**Steps to reproduce**
1. Run a local chaos wave at concurrency 6 with long-running Codex tasks
and K1/K2/K4 faults (for example B1, seed 9807).
2. Compare `restart_recovery` in `speed.json` with
`provider.session.load` in the recovered runs' run logs.

**Expected behavior**
The budget judges Paperclip's share of a recovery. The provider's
session load and the full latency are reported but not judged.

## What Changed

- `server/src/services/native-runtime/native-speed-budgets.ts`:
- New budgeted metric `restart_recovery_control_plane`: from `ready` to
the start of `provider.session.load`, or to the next provider event when
nothing reloads (for example a reattached runner).
- Its budget is 2.5 s / 4.0 s for claude, codex and opencode. Measured:
1.2–2.0 s per run at concurrency 6.
  - The full `restart_recovery` stays in the table for reporting.
  - Breach payloads carry `restartTotalMs` and `providerLoadMs`.
- `server/src/services/native-runtime/native-run-trace.ts`:
- The per-run check records when `provider.session.load` starts, keeps
the recovery measurement, and judges it when the run finishes.
- It judges after the deferred diagnostics drain, so a provider span
committed after the provider event still counts.
  - It emits `restart_recovery_control_plane` breaches only.
- `tests/restart-chaos/ttft-baseline-stats.ts`: each recovery sample
gains `controlPlaneMs` and `providerLoadMs`. The provider-load start is
the span event's write time minus its duration.
- `tests/restart-chaos/wave-speed.ts`:
- Three restart rows per adapter: `restart_recovery` (report-only),
`restart_recovery_control_plane` (budgeted) and `restart_provider_load`
(report-only).
  - All three have outage time removed.
- `tests/restart-chaos/ttft-baseline.ts`: the recovery table adds a
"Paperclip share" column.
- Docs: `doc/run-log-events.md` (metric and payload) and the
speed-budget table in
`doc/plans/2026-11-07-durable-ownership-and-reconciliation.md`.

## Verification

- `server`: `npx vitest run
src/services/native-runtime/native-speed-budgets.test.ts`, 18 passed.
Two new tests:
  - a recovery dominated by provider load does not breach;
- a slow Paperclip share breaches with `restartTotalMs` and
`providerLoadMs`, even when the provider span is committed after the
provider event.

The existing recovery tests moved to the new metric. All six were red
before the trace change.
- `tests/restart-chaos`: `npx vitest run --config
tests/restart-chaos/vitest.config.ts wave-speed.test.ts
ttft-baseline-stats.test.ts`, 29 passed. New tests cover the sample
split and the grader rows (a judged share, a report-only provider load,
zero breaches).
- Server typecheck: no errors in the touched files.

## Risks

- **Lower sensitivity to provider-side slowdowns.** A provider
regression in session load no longer breaches. It stays visible in the
report-only rows and in breach payloads.
- **Runs with no `provider.session.load` span** (reattach, or an adapter
that doesn't emit it) are judged on the full latency against the
control-plane budget. That's conservative.
- **Run-log consumers that filter on `metric: "restart_recovery"`** see
`restart_recovery_control_plane` from now on. The doc notes the change
date.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`, Anthropic), 1M context window, tool
use and code execution, via Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:22:28 -06:00
DottaandClaude Opus 5.5 42bc238aa5 fix(runner): keep a restart continuation when the lost turn's terminal is delivered again (#20196)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records it
as `provider_turn_lost_on_restore` and continues in a same-conversation
continuation turn
> - #20118 taught the event consumer to skip a stale restart-lost
terminal of another turn once a continuation is running. It applied only
when the checkpoint already recorded the interruption
> - When the checkpoint instead names the lost turn as active (the drain
hit right after the first turn was submitted), the consumer starts the
continuation on the first lost terminal. If the same lost terminal is
then delivered again, it was taken as the run's terminal and failed the
run
> - This pull request applies the skip to any recovered run once its
continuation has started. For a checkpointed interruption it still waits
for the startup decision first
> - The benefit is that a turn lost on restore always continues, however
often its lost terminal is delivered

## Linked Issues or Issue Description

Refs #20118

**What happened?**
Local restart-chaos K13 regression storm w-20261109T055902-s9402 (master
9b71d03215): 94 of 95 passed. In t90 (claude M1 parent), run `e5eab3cb`:
1. The run submitted its first turn at 06:28:25, and a graceful drain
followed at 06:28:28.
2. After restore at 06:28:41, the ACPX runner replayed the turn's accept
and start, then reported `turn_lab_1abbeca2…` as
`provider_turn_lost_on_restore`.
3. The controller started continuation `turn_lab_a3d009c6…`; the saved
checkpoint names it as the active turn.
4. The run failed at 06:28:45 with `native_provider_terminal_failed: The
runner restored the conversation after process loss, but the previous
turn is no longer active.` and needed a Retry.

**Expected behavior**
Once the continuation is running, a later delivery of the lost turn's
terminal is history, not the run's terminal. The run finishes on the
continuation.

**Steps to reproduce**
1. Recover a run whose checkpoint names the interrupted turn as active
(not yet recorded as terminal).
2. Deliver that turn's restart-lost terminal, which starts the
continuation, then deliver it again before the continuation completes.
3. Before this change, the second delivery ends the run as failed.

**Paperclip version or commit**
9b71d03215 (wave). Branch rebased on current master.

## What Changed

- `native-session-runtime.ts`: `supersededByRestartContinuation` applies
to every recovered run.
- Once a continuation has started, a restart-lost `turn.failed` for any
turn other than that continuation keeps the consumer going.
- With a checkpointed interruption, it still waits for the startup
decision first. Before this change it excluded the checkpointed
interrupted turn itself, which could also end a run if that turn's
replayed terminal arrived before the startup continuation began.
- `native-session-runtime.test.ts`:
  - a lost terminal delivered again after the continuation starts;
- the checkpointed interrupted turn's own terminal arriving before the
continuation starts.

## Verification

- Red/green: both new cases fail on master (the run ends failed or on
the stale terminal) and pass with the change. The #20118 cases still
pass.
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 389 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm run typecheck:typescript`
(`packages/paperclip-runner`): passed.

## Risks

- Low risk. The skip still requires a recovered run, a recoverable
`provider_turn_lost_on_restore` failure, a continuation that actually
started, and a terminal for a different turn than that continuation. A
loss of the continuation itself, real provider failures, and runs that
started no continuation keep their current handling.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.7
2026-11-09 01:08:19 -06:00
DottaandClaude Opus 5.5 ad2304dc88 fix(chat): recover a held conversation turn's answer after a restart (#20204)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in Agent Chat. The agent's final message
becomes the chat reply.
> - A chat turn can end while its task stays held: a pending approval,
an execution stage or a dependency blocker keeps the status.
> - #20179 made live run completion post that held turn's answer. It
added `conversationTurnAnswered()`.
> - Background presentation recovery still required a
`conversation_turn_finished` decision and a `done` result.
> - If the server restarts between the status commit and the live
presentation, recovery skips the held turn. The person never gets an
answer.
> - Restarts are product failures. A committed answer must not be lost.
> - This pull request makes recovery use the same answered rule as the
live path, through one shared helper.
> - The benefit is that a held turn's answer reaches the chat exactly
once, with or without a restart.

## Linked Issues or Issue Description

Refs #20179 (live fix; this PR closes the "Known gap" in its Risks
section).

**What happened**

In an Agent Chat, the agent resubmitted a hire approval and ended its
turn with a yielded result. The finalizer committed
`governed_response_waiting` and moved the task to `in_review`. The
server restarted before run completion posted the reply. Background
presentation recovery found the run as a candidate, but declined it,
because it accepted only `conversation_turn_finished` with a `done`
result. No reply was posted. The run stayed a pending presentation
candidate.

**Expected behavior**

After a restart, recovery posts the held turn's final answer exactly
once. It never posts a second copy when the live path already presented
the answer. When the turn's own question or confirmation card presents
it, recovery posts nothing and records that decision, as the live path
does.

**Steps to reproduce**

1. Open an Agent Chat. Link a pending hire approval to the chat task.
2. Let the agent end the turn with a yielded result and a
`response_wake` continuation. The status decision commits as
`governed_response_waiting`.
3. Stop the server after the status commit and before run completion
presents the reply.
4. Start the server. Presentation recovery runs.
5. See that the chat shows no reply.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: add
`conversationTurnPresentsAnswer()`. It is the one rule for whether a
successful conversation turn's prose is its answer:
`conversation_turn_finished`, or a committed decision with no pending
card of the turn (`conversationTurnAnswered()`).
- `server/src/services/heartbeat/run-completion.ts`: the live path calls
the shared helper. No change in behavior.
- `server/src/services/native-runtime/native-run-finalizer.ts`
(`recoverCommittedNativeChatResponse`):
- A current conversation turn no longer needs
`conversation_turn_finished`. The turn must still be the unsettled
current turn: same agent, user, session generation, `conversationState:
active`, and no other execution run. A `conversation_turn_finished` turn
still needs `in_progress`.
- A conversation turn with a `yielded` result is accepted next to
`done`.
  - The resolver gets `conversationTurnFinished` from the shared helper.
- When the turn's own card presents it, recovery records
`presentationDecision: none` (reason
`committed_conversation_interaction_presented`) and settles the
conversation turn, like the live path. The run then leaves the pending
presentation set. Before, such a run stayed a candidate.
-
`server/src/services/native-runtime/native-response-presentation-query.ts`:
the candidate prefilter matches the same turns: any committed reason for
an active conversation turn, and `yielded` conversation results.
-
`server/src/services/native-runtime/agent-chat-response-recovery.test.ts`:
restart regressions with the real finalizer and a real database.
- `doc/execution-semantics.md`: state that recovery applies the same
rule.

## Verification

- Failing first. With only the new tests applied, 5 of 7 new cases
failed: pending approval, revision-requested approval, execution stage
(all yielded), pending approval with `done`, and the own-card case. The
run was a candidate, but recovery posted nothing and did not settle it.
- After the fix: `cd server && npx vitest run
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/services/heartbeat/run-completion.test.ts
src/__tests__/idle-native-finalization.test.ts
src/services/presentation-ownership-read.test.ts
src/services/chat-monitor-continuation.test.ts
src/services/native-runtime/native-preserved-monitor-wait.test.ts
src/services/native-runtime/native-response-publication-retry.test.ts
src/services/heartbeat/run-lifecycle.test.ts
src/__tests__/scoped-deadline-work.test.ts
src/services/native-runtime/paperclip-control-plane-port.test.ts
src/__tests__/heartbeat-run-summary.test.ts
src/__tests__/agent-conversations.test.ts`: 12 files, 627 tests passed.
- The new cases check:
- Held turns: the run is a candidate, the check posts the exact final
answer once, the presentation decision is recorded, the run leaves the
candidate set, and a second recovery returns false with one comment.
  - Live already presented: no candidate, no second comment.
- Own question card: no comment, `presentationDecision: none`, the card
stays pending, the conversation settles to `waiting`, and the run leaves
the candidate set.
- Previous-release live decision `none` (`yielded_control_plane_wait`):
kept, not re-presented.
- `tsc --noEmit -p server`: no errors in changed files. Four unrelated
errors remain in this local worktree (`esbuild` not linked in
`native-harness-prestage.test.ts`, and a linked-package type mismatch in
`run-dispatch/adapters/postgres.ts`).
- Turn start: no change. The helper runs only in run completion (after
the provider turn ends) and in background presentation recovery. Nothing
new runs before `turn.start`, and no files move. The live path keeps the
same queries: it short-circuits on `conversation_turn_finished`, and
otherwise runs the same card read as before.

## Risks

- Low to medium. Recovery can now post the answer of a conversation turn
whose task is held. This matches the live path since #20179.
- A card-presented turn found by recovery is now settled with
`presentationDecision: none`. Before, it stayed a pending candidate and
kept the presentation worker (and idle-sleep checks) busy.
- A `conversation_turn_finished` turn keeps the `in_progress` check.
Held turns accept any status their gate committed. The unsettled-turn
checks (active conversation state, session generation, execution run)
still block stale or superseded turns.
- Previous-release state: no persisted shape changes. A run committed by
the previous release with no saved presentation decision (the restart
window) is recovered by the new code. The test fixture uses the
unchanged finalizer, so it saves exactly the previous release's shape. A
run whose live path already recorded a decision keeps it. A held turn
that the previous release recorded as `yielded_control_plane_wait` is
not re-presented, so no late duplicate or stale reply appears.
- Not changed: generic stranded-work recovery in `recovery/service.ts`
still skips only `conversation_turn_finished` runs during the
publication window.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:08:10 -06:00
DottaandClaude Opus 5.5 bf67e2fbd9 test(chaos): honor 429 retry delays on harness reads (#20194)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness collects each wave's speed from the board
API after the tasks rest, including the M1 subtasks' runs
> - The server rate-limits concurrent issue-list requests per client and
answers 429 with `retryAfterSeconds`
> - After a 95-task storm, the harness requested every M1 parent's
subtask list at once. The server returned 429, speed collection failed,
and the wave exited 1 with no speed grade
> - This pull request makes harness GET requests honor a 429's retry
delay with a bounded number of attempts, and collects subtask speed one
parent at a time
> - The benefit is that a busy wave keeps its speed grade, and the
harness behaves like a polite API client

## Linked Issues or Issue Description

Refs #20071, #20147

**What happened?**
Wave `w-20261109T055902-s9402` (K13 storm, A1/C1/M1, 95 tasks) finished
its tasks. Speed collection then failed with: `GET
/api/companies/…/issues?view=index&parentIds=…&limit=20 returned 429:
{"error":"Too many concurrent issue-list requests for this
actor/client","retryAfterSeconds":1}`. `main.ts` ran `childSpeedTasks()`
for every observer in parallel.

**Expected behavior**
- The harness waits and retries a rate-limited read.
- It does not flood the issue-list endpoint while collecting speed.
- A busy wave still gets graded.

## What Changed

- `api.ts`: `ChaosApi` retries a GET that returns 429.
- It waits for the delay from `Retry-After` or the body's
`retryAfterSeconds` (default 1 s, capped at 10 s), scaled by the attempt
number.
  - It makes at most 5 attempts by default.
- Writes (POST/PUT/PATCH/DELETE) are never retried, so a user action is
never repeated.
- `rateLimitDelayMs` is exported for tests.
- `main.ts`: subtask speed is collected one observer at a time instead
of with `Promise.all`.
- New `api.test.ts` against a real local HTTP server:
  - a GET that gets 429 twice, then 200, succeeds;
  - a POST that gets 429 is sent once;
  - retries stop at the bound;
  - delay parsing (header, body, cap, default).

## Verification

- `npx vitest run` in `tests/restart-chaos`: 216 passed.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- A persistently rate-limited read now takes up to about 15 s before it
fails, instead of failing at once.
- Collecting subtask speed one parent at a time adds a few seconds to
the end of large M1 waves.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 01:06:41 -06:00
Nicky LeachandClaude Opus 5.5 990de1e0bf perf(environments): release a stopped plugin task once (#20203)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugin environment drivers with `supportsTasks` run agents as
provider tasks, and warm runs keep a parked task for the next run
> - A follow-up run can find that its parked task already stopped. Core
then retires that task before it acquires a new one
> - Core sent `stop`, `status`, and `complete` in the settle step. Then
the plugin release hook sent `stop`, `status`, and `complete` again for
the same task. Every call was on the critical path of the new run
> - This pull request removes the repeated calls and keeps every call
that does work no other call does
> - The benefit is a faster start for a run whose parked task already
stopped

## Linked Issues or Issue Description

No issue exists. This follows the enhancement template.

**What existing behavior does this improve?**

The release of a plugin task lease that has already stopped. This occurs
when a follow-up run finds that its parked warm task stopped.

**Subsystem affected**

Server: plugin environment drivers and the plugin Runner task lifecycle.

**Current behavior**

Core sends `stop`, `status`, and `complete` in the settle step. Then the
plugin release hook sends `stop`, `status`, and `complete` again for the
same task. Core also sends `stop` after resume already saw that the task
stopped. On a hosted test deployment, 8 of 40 follow-up runs took this
path. From the resume to the new acquire, these runs spent P50 1014 ms
and P90 1080 ms. These 8 runs are the full overall P90 (1275 ms).

**Proposed behavior**

Core does not call the release hook when settle confirmed the stop and
completed the same task. After a resume with no lease, Core sends
`status` first, and sends `stop` only when `status` does not confirm the
stop.

**Reason and benefit**

The repeated calls do no new work. Removing them saves about 355 ms,
plus the `stop` share of about 187 ms, for each such run. These numbers
are modeled from trace timings.

**Breaking changes**

None for the plugin API. A task driver must release task-scoped
resources in its `status` or `complete` handling. The doc now says this.

**Additional context**

The settle `status` stays. Providers release task storage there, and it
records the stop that gates the credential writeback and `complete`.

## What Changed

- `settleDispatchedEnvironmentTask` now returns `"completed"` when it
confirmed the stop and completed the task in this call, `"stopped"` when
the lease was already settled, and `false` otherwise.
- The plugin driver release does not call `environmentReleaseLease` when
settle returned `"completed"`. In all other cases it calls the hook as
before.
- When `environmentResumeLease` returns no provider lease, the release
sends `status` first. It sends `stop` (and a second `status`) only when
that `status` does not confirm `executionStopped: true`. Startup and
startup-cleanup leases keep the old `stop` first order.
- The cleanup sweep (`retryPendingSandboxTeardown`) and
`destroyRunLease` do not change. A settle failure still leaves the lease
in `pending_cleanup`, and the sweep runs `stop`, `status`, `complete`,
and the release hook.
- `doc/plugins/PLUGIN_RUNNER_EXECUTION.md` tells drivers about the new
order and that task-scoped resources must be released in `status` or
`complete`.

### Dispatch count for a stopped parked task (resume to new acquire)

| | Calls |
| --- | --- |
| Before | resume, stop, status, (credentials), complete, release hook
(stop, status, complete), acquire: 9 provider calls (10 with Codex
writeback) |
| After | resume, status, (credentials), complete, acquire: 4 provider
calls (5 with Codex writeback) |

The modeled saving is about 540 ms for each such run: about 355 ms for
the repeated release, and about 187 ms for the settle `stop` and
`status`. This pull request keeps the settle `status`, so the real
saving is about 355 ms plus the `stop` share of the 187 ms. These
numbers are modeled from trace timings. They are not measured on this
branch.

### What covers each removed call

- Release hook `stop`, `status`, `complete`: settle already sent `stop`
and `status` and checked `executionStopped: true` for the same task id
(`lease.providerLeaseId`), then sent `complete`. The skip uses settle's
success as its condition, not the code path.
- Settle `stop` after a resume with no lease: the next `status` must
report `executionStopped: true`. If it does not, Core sends `stop` and
`status` as before.

### What each kept call protects

- Settle `status`: providers release task storage and cancel
preparations here. It also writes `runnerTaskExecutionStopped`, which
the credential writeback and the `complete` bookkeeping need.
- Credential writeback: Codex can rotate its token. The writeback must
land before the new task installs credentials. It still runs inside
`complete`, before the new acquire.
- Settle `complete`: it retires every run lease that owned the task and
clears the dispatch marker.
- Release hook when settle did not complete a dispatched task in this
call: the provider may still hold the lease.

## Verification

- `pnpm --filter @paperclipai/shared --filter @paperclipai/plugin-sdk
build`
- `pnpm --filter @paperclipai/server typecheck`
- `npx vitest run src/__tests__/plugin-warm-tasks.integration.test.ts
src/__tests__/plugin-task-credential-writeback.test.ts
src/__tests__/environment-task-runtime.test.ts`
- Related suites: `environment-runtime`,
`plugin-runner-task.integration`, `task-creation-recovery.integration`,
`heartbeat-plugin-environment`, `native-startup-cleanup-recovery`,
`plugin-environment-driver-ready-recovery`,
`environment-task-authority.integration`, `native-workspace-warm-reuse`,
`remote-workspaces.integration`, `legacy-workspace-restore-recovery`,
`plugin-task-source-recovery.integration`,
`managed-plugin-environments`.

Locally, all of these pass except `heartbeat-plugin-environment`
"inherits the instance default environment across companies…". That test
also fails locally with the `origin/master` service files, so this
change does not cause it.

The existing `plugin-runner-task.integration` same-run replacement test
now expects no release hook after a completed settle.

New tests:

- The stopped parked path sends exactly resume, `status`, `complete`,
acquire. The `status` goes to the old task.
- A resume with no lease for a task that still runs sends `status`,
`stop`, `status`, `complete`.
- A failure at `status`, `complete`, or `stop` leaves the lease in
`pending_cleanup`, and the sweep then sends `stop`, `status`,
`complete`, and the release hook.
- With an observed stop, the Codex writeback runs before `complete`. A
writeback failure leaves the lease for cleanup, and the retry writes
back and completes.
- A startup-cleanup lease still sends `stop` first.

## Risks

- A third-party task driver that releases resources only in
`onEnvironmentReleaseLease` would not get that call after a completed
settle. The doc now states the requirement. The cleanup sweep still
calls the hook after any failure.
- A driver that returns no lease from resume while the task still runs
costs one extra `status`. The behavior stays correct.
- No migration. No API change for plugins.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use and
code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:00:49 -08:00
Nicky LeachandClaude Opus 5.5 9046884811 perf(native-runtime): claim a parked Runner while the follow-up run prepares (#20195)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Paperclip Runner executes native agent runs. A host can park an idle
Runner (release it and freeze it) between runs, and resume it for the
next run on the same task.
> - A follow-up run that reuses a parked Runner must claim it first. The
claim is a bootstrap ticket, a checkpoint flush, and the plugin resume
that thaws the Runner.
> - Core made the claim only inside `attachRun`, after all of its
per-run preparation. The claim does not depend on that preparation, so
the two ran in series.
> - This pull request starts the claim while Core still prepares the
run. `attachRun` then waits for the claim that is already in progress.
> - The benefit is a shorter start for every follow-up run on a parked
Runner. The model below gives about 285 ms per run at P50.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

**What happened?**

A follow-up run that reuses a parked Runner claims the Runner only in
`attachRun`. Core calls `attachRun` after backend setup: runner
environment, assigned MCP tools, tool authority and the plugin task
owner. The claim waits for all of this, but it uses none of it.

**Expected behavior**

The claim runs at the same time as backend setup. Everything that needs
the new run's tool authority stays at attach.

**Steps to reproduce**

1. Run a Codex or Claude task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. Before this change, the plugin resume starts only
after backend setup ends.

**Paperclip version or commit**

`master` at 7086019d4b.

**Deployment mode**

Hosted, with parked Runner tasks.

## What Changed

- **Transport** (`runnerd-codex-transport.ts`):
- New `beginRunAttachment({ runId })`. It runs the attach admission
checks, then starts the existing `#claimReleasedRunner` as the held
attachment operation. It returns `false` and does nothing when the
Runner is not released and confirmed, a claim already exists, or
admission fails.
- The admission checks move into `#assertRunAttachmentAdmission`.
`beginRunAttachment` runs them first, and `attachRun` runs them again.
- The next `attachRun` for the same run waits for the early claim. An
`attachRun` for a different run, `release` and `turn/start` still fail
with `native_runner_run_attachment_in_progress` while the claim is held.
- After an early claim, `attachRun` fails with
`native_runner_resume_reconciliation_expired` before it registers if the
claim deadline has passed.
- **Pass-throughs:** `HarnessNativeSession`, `CodexHarnessSession` and
the session, harness and transport contracts get an optional
`beginRunAttachment`. Each layer does the same identity,
protocol-integrity and busy checks that its attach does.
- **Core** (`native-session-executor.ts`):
- The executor records when the in-memory warm entry it reuses was
parked (`released`).
- `createRunnerdBackend` gets `beginRetainedRunAttachment`. Core gives
it only for that parked in-memory warm entry.
- The plugin task owner preparation moves into `preparePluginTaskOwner`.
On the early path, Core prepares and rebinds the retained owner at the
same time as the runner environment and the assigned MCP tools, then
starts the claim.
- Cold, adopted, remote warm recovery and restart paths keep the old
order.

### What stays at attach, and why

- Registration, the quiescence probe, `run.attach` and the authority
rotation stay in `attachRun`. Checkpoint registration needs the new
run's tool authority epoch, and Core creates that epoch during backend
setup.
- The admission checks run again at attach, so a state change during
preparation still blocks the attachment.

## Verification

- `vitest run src/live/runnerd-codex-transport-release-resume.test.ts`
(runner): 21 passed. Five new modes:
- `early claim`: `attachRun` waits for the held claim and does not route
the new run before the claim ends;
- `early claim abandoned`: a close after the claim ends fails the
waiting `attachRun` with `closed`, and the new run is never routed;
- `early claim cancellation`: a close during the claim cancels it, and
the next `attachRun` fails with `closed`;
- `early claim expired`: the deadline passes during the claim, and
attach fails before it registers;
- `early claim stale ticket`: the deadline passes after the claim,
before attach.
- `vitest run plugin-runner-release-resume.integration.test.ts
plugin-runner-eviction.integration.test.ts` (server, with real Runner
processes from `cargo build --workspace --bins`): 2 files, 51 passed,
none skipped. New modes:
- `cached held claim` (Codex and Claude): the plugin resume is held, and
the test proves Core's tool preparation runs while the claim is in
progress;
- `cached cancellation` (Codex): a cancellation after an early claim
retires the Runner and the run stops cleanly.
- `pnpm run typecheck:typescript` (runner) and `pnpm typecheck`
(server): pass.
- Evidence (modeled, not measured live): traces of 8 resumed runs on
parked Runners in a hosted test deployment, on current `master`. The
claim takes about 300 ms, and it fits inside backend setup in 7 of 8
runs. The modeled saving is about 285 ms per follow-up on a parked
Runner at P50 (range 208–331 ms). I did not run this change on a hosted
deployment.
- Not run: the full server suite (CI runs it), smoke tests, and the
hosted park, claim and attach loop with this change.

## Risks

- **New failure mode:** Core now claims the parked Runner before backend
setup ends. If that run then fails preparation, or is cancelled before
attach, the existing retire path closes the claimed session. The Runner
is retired, so that agent's next follow-up starts cold. Before this
change, a preparation failure happened before the claim, while the
Runner was still parked.
- **Ordering:** the retained plugin task owner is rebound to the new run
earlier, before Core creates tool authority. Nothing in the owner
preparation reads that authority. A failure in tool authority setup now
happens after the rebind, and the retire path handles it.
- **Scope:** only an in-memory warm entry that was parked takes the
early path. Cold launches, adopted Runners, remote warm recovery and
restart recovery are unchanged.
- **Deadline:** the claim deadline now starts earlier. A very slow
backend setup can make the claim expire before attach. Attach then fails
with `native_runner_resume_reconciliation_expired` before it registers,
the same failure as a slow claim today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:08 -08:00
Nicky LeachandClaude Opus 5.5 b4a77b8915 perf(plugins): read runtime identity in one statement without a row lock (#20192)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Lifecycle plugins read the agent runtime identity through the plugin
host
> - The identity read used the full-state path. That path locks the
agent row and runs 11 statements
> - Plugins call the identity read many times as a freshness fence, so
each round trip adds to the run latency
> - This pull request makes the identity read one statement on the
request pool, without a row lock, and loads instance secrets with their
versions in one statement
> - The benefit is fewer round trips per read (11 to 4) and no wait
behind writers that hold the agent row

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The plugin host read of the agent runtime identity
(`pluginAgentRuntimeIdentity`).

**Subsystem affected**
Server: plugin agent runtime state and instance secrets
(`server/src/services/plugin-agent-runtime-state.ts`,
`server/src/services/secrets.ts`).

**Current behavior**
The identity read shares the full-state transaction. It opens a
transaction on the file-storage pool and locks the agent row with `FOR
UPDATE`. It reads the plugin, the company settings and the latest
lifecycle event in separate statements. Then it reloads each instance
secret and its version one at a time. One read costs 11 statements. It
also waits behind any writer that holds the agent row, for example
managed file publication.

**Proposed behavior**
The identity read is one statement on the request pool. The statement
joins the plugin and the company settings, and reads the latest
lifecycle event id in a subquery. The agent fields, the authorization
inputs and the revision come from one snapshot. Instance secrets load
together with their latest versions in one statement. One read costs 4
statements.

**Reason and benefit**
A resumed agent run makes about 7 identity reads on its critical path in
a hosted deployment. With a 4 ms round trip, the modeled saving is about
330 ms per run at P50.

## What Changed

- `pluginAgentRuntimeIdentity` reads the agent, plugin, company settings
and latest lifecycle event id in one statement on the request pool. It
does not take a row lock.
- The authorization check moves into `assertOwningLifecyclePlugin`. The
identity path and the full-state path both use it, so both fail with the
same forbidden error.
- A missing agent still returns `null` before authorization runs.
- The full-state path (`readState`) keeps its transaction and its `FOR
UPDATE` lock. It no longer has an identity-only branch.
- `instanceSecrets.read` loads each secret with its latest version
through a left join in one statement. `resolveSecretValueInternal`
accepts these preloaded rows and skips the reload when the secret and
the version match. It still reloads when they do not match.
- The two audit writes per resolved secret run concurrently, as before.
- Tests:
- `agent-runtime-hooks.test.ts` compares the identity read with the full
read across revisions, lifecycle changes and secret changes, and across
every forbidden case. It confirms that forbidden reads record no secret
access. It confirms that the identity read does not wait for a held
agent row lock.
- `agent-instance-secrets.test.ts` confirms that a read of rotated
secrets uses one select and returns the latest values. It confirms that
each read records an access event. It confirms that a revoked version
fails with `version_inactive` and a missing version fails with
`version_missing`.

### Why dropping the lock is safe

The lock exists for the full-state path. That path projects instruction
files under the agent lock. The identity path does no projection. It
also released the lock before it returned, so no caller could rely on
the lock.

### Unchanged behavior

- Authorization: the same plugin, settings, capability and
required-plugin checks, with the same forbidden error.
- Decryption: the same provider resolution from the version material.
- Scope checks and version status checks: the same checks on the same
rows.
- Audit: the same `lastResolvedAt` update and access event for each
resolved secret.

## Verification

- `pnpm --filter @paperclipai/shared --filter @paperclipai/plugin-sdk
build`
- `pnpm --dir server run typecheck`: pass.
- Targeted tests: pass (6 files, 174 tests).

  ```
  pnpm --dir server exec vitest run \
    src/__tests__/agent-runtime-hooks.test.ts \
    src/__tests__/agent-instance-secrets.test.ts \
    src/__tests__/project-instance-secrets.test.ts \
    src/__tests__/secrets-service.test.ts \
    src/__tests__/agent-runtime-assignments.test.ts \
    src/__tests__/agent-runtime-skills.test.ts
  ```

- CI runs the full suite.

### Benchmark

A local benchmark ran against the embedded test Postgres. A delay proxy
simulated network round trips.

| Condition | Before | After |
|---|---|---|
| Statements per read | 11 | 4 |
| P50, no added delay | 2.38 ms | 1.47 ms |
| P50, 4 ms round trip | 104.0 ms | 32.2 ms |
| P50, 4 ms round trip, prepared statements | 51.9 ms | 16.3 ms |

The new read costs about 0.3× the old read in round trips. In a hosted
deployment, a resumed agent run makes about 7 of these reads on its
critical path. The modeled saving is about 330 ms per run at P50.

## Risks

- Low risk.
- The identity read no longer runs in the same snapshot as a concurrent
writer that holds the agent row. It can return the state from before
that write commits. The `revision` field still comes from the same
snapshot as the agent fields, so callers that use it as a freshness
fence see a consistent pair.
- The identity read now runs on the request pool, not the file-storage
pool. It adds one short query to the request pool and removes one
transaction from the file-storage pool.
- The preloaded secret path applies only when the secret and the version
match the rows from the join. Any other case falls back to the existing
reload.
- No migration. No API change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:49:43 -08:00
DottaandClaude Opus 5.5 3964ee1fca fix(native): a governance pause Stop with no turn result stays a cancellation (#20186)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the company; the pause's lifecycle Stop
cancels the agents' running native turns
> - When a Codex turn is interrupted this way, it ends without a
semantic result
> - The native executor treated that missing result as a retryable
interruption, so it rewrote the cancelled run to `failed` and scheduled
an exact resume
> - The pause held that recovery, and Raise budget and resume continues
only runs that are still cancelled, so the task stayed blocked with no
new run
> - This pull request makes the executor treat an acknowledged
governance pause Stop like a person's Stop: the run stays cancelled
> - The benefit is that Raise budget and resume continues the task
(#20144), with no manual Retry

## Linked Issues or Issue Description

Refs #20144 (resume continues budget-stopped work), #20143 (governance
pause stops are not failures), #20113

**What happened?**
Restart-chaos wave w-20261109T051242-s9802 (H1, seed 9802), task t03
(Codex, fault K2). The company budget crossed its hard limit at
05:19:39.9. The pause's lifecycle Stop ("Agent lifecycle stop
requested") cancelled run 59e976bc at 05:19:40.05, and Codex
acknowledged the interrupt. The turn ended without a semantic result,
and at 05:19:41.27 the executor rewrote the run to `failed` /
`native_session_interrupted`, with an agent-owned `active_run_watchdog`
recovery (exact checkpoint resume). The task went to `blocked`. After
`raise_budget_and_resume` at 05:20:20, no run started for 14 minutes.

**Expected behavior**
A pause's Stop is intentional. The run stays cancelled with its pause
record, and resuming continues the task once.

**Steps to reproduce**
Pause a company for budget while a Codex native turn runs, so the Stop
interrupts the turn before it records a result. Before this change, the
run becomes `failed` with a scheduled exact resume instead of staying
`cancelled`.

## What Changed

- `native-session-executor.ts`: the no-semantic-result branch now also
accepts an acknowledged governance pause Stop (`isGovernancePauseStop`
with an acknowledged native Stop receipt) as an intentional stop. It
settles the coordinator as `native_retry_cancelled` and raises
`NativeCancellationPendingRecoveryError`, as it already does for a
person's Stop and for a reassignment. The run is not rewritten to
`failed`, and no recovery action is created.
- The Stop's cancellation record is written by its fence before
dispatch, so the check does not depend on whether the terminal write has
landed.

## Verification

- `native-session-executor.test.ts`: new cases for "Agent lifecycle stop
requested" with a `pauseStop`, and for "Cancelled due to budget pause".
Each was red before the fix (it reached the retryable-failure write) and
is green now. A control case shows that a system lifecycle Stop without
`pauseStop` (termination) still takes the retry path.
- Full `native-session-executor.test.ts`: 1309 passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun. Phase 3 runs it after merge.

## Risks

- A governance pause Stop that lands during a turn no longer schedules
an exact resume of that turn. The task continues on resume through a new
Retry (#20144) instead.
- Termination and cleanup use the same lifecycle Stop reason. They are
excluded because only the pausing path records `pauseStop`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:42:25 -06:00
DottaandClaude Opus 5.5 7c65a2f479 fix(authz): link plugin settings and say reply now on board-only denials (#20182)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Chat agents run with agent credentials, and plugin management routes
are board-only
> - A board-only denial tells the agent the denial is final and links
the board page that holds the control, when the page is known (#20151)
> - Plugin routes had no known page, so the denial said "link a page
only if you already know its route"
> - In a chat eval, an agent asked to finish configuring a plugin got
this denial and kept searching the API for about two minutes, so it did
not answer in time
> - This pull request names the plugin manager, or that plugin's
settings page, in the denial
> - The benefit is that the agent can hand the step to the person with a
working link right away

## Linked Issues or Issue Description

Refs #20151

**What happened**
An agent that called a board-only plugin route (for example `GET
/api/plugins` or `POST /api/plugins/:pluginId/config`) got the generic
remediation with no page. In the `plugins.configure-plugin` chat eval,
the agent kept calling `search_api` for another plugin route after the
403 and did not reply within the 150-second window.

**Expected behavior**
The denial names the page where a board user does the step: the plugin
manager (`/<prefix>/company/settings/instance/plugins`) or the plugin's
settings page (`/<prefix>/company/settings/instance/plugins/<id or
key>`). No link is given when the operator hides plugin management
(`PAPERCLIP_HIDDEN_SETTINGS=instance.plugins`).

**Steps to reproduce**
1. Install a plugin.
2. As an agent, call `POST /api/plugins/<pluginId>/config`.
3. See a 403 `board_access_required` remediation without a plugin page.

## What Changed

- `server/src/routes/authz.ts`: board-only denials on `/api/plugins`
routes link the plugin manager, or the plugin's settings page when the
path names a plugin UUID or key. The plugin settings page accepts
either. No link when plugin management is hidden.
- `server/src/routes/authz.ts`: every agent board-only denial now says
to stop working on the denied step and reply now. With the plugin link
alone, the eval agent still searched the API for two more minutes.
- `server/src/routes/authz.ts`: any manifest-format plugin key
(including dot-free keys such as `my-plugin`) gets a direct link; plugin
manager endpoints (`examples`, `install`, `tools`, `ui-contributions`)
link the plugin list.
- `ui/src/pages/PluginSettings.tsx`: custom settings slots match by
plugin key as well as plugin id, so a page opened by key shows the
plugin's own settings.
- `server/src/__tests__/board-access-denial.test.ts`: cases for list,
install, config by id, disable by key, a dot-free key, manager
endpoints, and the hidden-settings case.
`ui/src/pages/PluginSettings.test.tsx`: opening the page by key mounts
that plugin's settings slot.
- Merged `master`, keeping its project access-members denial cases.

## Verification

- `cd server && npx vitest run src/__tests__/board-access-denial.test.ts
src/__tests__/agent-permissions-routes.test.ts
src/__tests__/environment-routes.test.ts
src/__tests__/execution-workspace-runtime-lease-route.test.ts`: 241
passed after merging master.
- `cd ui && npx vitest run src/pages/PluginSettings.test.tsx`: 9 passed;
the new key-route case fails without the `PluginSettings` change.
- `cd server && pnpm typecheck`: passes.

## Risks

- Low risk. Only the remediation text of agent board-only denials on
`/api/plugins` routes changes. Board users and authorization do not
change.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:39:39 -06:00
DottaandClaude Opus 5.5 492316e576 fix(plugins): accept secret_ref bindings in string secret-ref config fields (#20181)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Plugins declare their settings in an `instanceConfigSchema`, and the
server validates each save against that schema
> - Plugin credential fields use `type: "string", format: "secret-ref"`
> - The settings form, the plugin secrets handler and the plugin SDK all
use the `{ type: "secret_ref", secretId, version }` binding object for
these fields, and the secrets handler rejects a bare secret id
> - Schema validation ran on the raw config, so the binding object
failed with "must be string" and no value could be saved
> - This pull request validates the binding's secret id in place of the
binding object
> - The benefit is that an operator can bind a company secret to a
plugin credential field from the settings page or the API

## Linked Issues or Issue Description

**What happened**
`POST /api/plugins/:pluginId/config` returned 400 "Configuration does
not match the plugin's instanceConfigSchema" with
`{"field":"/apiKey","message":"must be string"}` when the config held `{
"apiKey": { "type": "secret_ref", "secretId": "<uuid>", "version":
"latest" } }` for a field declared `{ "type": "string", "format":
"secret-ref" }`. A bare UUID string passes the schema but the secrets
handler then rejects it ("must use { type: \"secret_ref\", secretId,
version? }"). So no value could be saved for a required secret-ref
field.

**Expected behavior**
The binding object that the settings form submits and that
`ctx.secrets.resolve()` accepts is saved. Other schema rules still
apply.

**Steps to reproduce**
1. Install a plugin whose manifest has
`instanceConfigSchema.properties.apiKey = { "type": "string", "format":
"secret-ref" }`.
2. Create a company secret.
3. On the plugin settings page, pick that secret for the field and click
Save Configuration (or POST the binding object to
`/api/plugins/:pluginId/config`).
4. See the 400 "must be string" response.

Found while getting the chat eval cases `plugins.configure-plugin` and
`plugins.disable-plugin` to run. Their setup and the human's
settings-page save both hit this error.

## What Changed

- `validateInstanceConfig` now replaces each valid `secret_ref` binding
object at a schema `format: "secret-ref"` path with its secret id before
Ajv validates. The stored config and the config sent to the worker do
not change.
- The walk follows array `items` (single and tuple) and objects inside
arrays as well as object properties and `allOf`/`anyOf`/`oneOf`, so
secret pickers inside array fields can save too.
- A binding is accepted only when `envBindingSecretRefSchema` (the
secrets handler's own parser) accepts it, so validation and the secrets
handler agree on what is a valid binding.
- Malformed bindings and objects in ordinary string fields still fail
validation.
- Added `server/src/__tests__/plugin-config-validator.test.ts`.

## Verification

- `cd server && npx vitest run
src/__tests__/plugin-config-validator.test.ts
src/__tests__/plugin-routes-authz.test.ts
src/__tests__/plugin-sandbox-provider-config-validation.test.ts`: 48
passed.
- The new test fails on `master` without the fix (3 of 3 fail) and
passes with it.
- `cd server && npx vitest run
src/__tests__/plugin-config-validator.test.ts`: 6 passed after the
array/parser follow-up; the 3 follow-up tests fail on the previous
commit.
- `cd server && pnpm typecheck`: passes.

## Risks

- Low risk. Only configs that held a valid binding object at a declared
secret-ref path change result, from a 400 to accepted. The secrets
handler still checks that the secret belongs to the company.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:37:00 -06:00
Nicky LeachandClaude Opus 5.5 92a18bb0f3 perf(runner): restart the ACPX provider when a turn settles (#20190)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Paperclip Runner drives Claude through the ACPX sidecar. The
sidecar's MCP endpoint lasts for the whole session, so it cannot tell
which turn sent a late tool callback.
> - Because of this, the Runner reaps the sidecar and provider of a
generation that ran a turn before it releases call-ID tombstones and
rotates receipt epochs. Today this restart runs at the start of the next
turn.
> - Every follow-up turn in a run pays that restart on its critical
path: about 1.5 s with real Claude. #20145 moves the restart to park,
but a second turn in the same run, after the first turn that follows an
attach, still restarts cold.
> - The restart does not have to wait for the next turn. When a turn
settles and the host has its terminal result, the Runner is idle.
> - This pull request restarts the provider at that point. The next
turn, or the next park, then finds a fresh generation.
> - The benefit: same-run follow-up turns start without a restart, park
does no restart after a settled turn, and the late-callback guarantee is
unchanged.

## Linked Issues or Issue Description

Refs #20145, which moved the restart to park. This pull request builds
on it. No public issue exists, so the problem is described here.

**Subsystem affected**

Paperclip Runner, ACPX provider (Claude).

**Current behavior**

`AcpxProviderSession::start_turn_timed` restarts the provider before any
turn whose generation already ran a turn. A follow-up turn in the same
run waits for `session.suspend`, sidecar shutdown, a new sidecar,
`initialize`, `session.open` and `run.attach` before `turn.start`. Since
#20145, park does the restart instead, but only for the first turn after
an attach.

**Proposed behavior**

When a turn settles and the controller has ACKed every committed event,
the Runner restarts the provider. The next `turn.start` uses the fresh
generation and only rotates receipts. Park reuses the fresh generation
and does no second restart.

**Reason and benefit**

The restart leaves the turn's critical path, both for same-run
follow-ups and for the turns after a park. The order of reap and
rotation stays the same, so a late callback from the reaped generation
still cannot reach a later turn.

## What Changed

- **Runner loop** (`durable/runner.rs`): new
`CommandExecutor::prepare_next_turn` hook, a no-op by default. The
runner calls it after the idle provider poll, only when no control frame
is waiting and the controller has ACKed every committed event
(`acked_source_seq >= highest_source_seq`). So the restart always runs
after the terminal result reaches the host.
- **ACPX turn-end restart** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- `restart_session_after_turn` restarts a Claude session that has a
completed turn, nothing pending, no buffered events, a dormant goal and
its whole event tail ACKed. It first settles an abandoned park
observation. If that settlement reads more events, it skips this round,
and the next idle round restarts after those events are ACKed.
- `AcpxProviderSession::restart_after_turn` does the same suspend → reap
→ start → bootstrap → identity check as the turn-start restart. It
records the reaped process id in `restarted_after_turn_from`. This field
is the explicit "fresh because of a turn-end restart" marker.
- The session does not poll a fresh generation until its next turn or
rebind, the same as a prepared one. Its startup output is published with
the next turn, as today. Polling it early would add events after the
host computed its park cut.
- **Next turn** (`acpx_provider_session.rs`): `start_turn_timed`
restarts only when the generation ran a turn. When
`restarted_after_turn_from` is set, it does the post-restart receipt and
ledger rotation without a second restart. `session.reconciled` reports
the reaped process as `previousProcessId`.
- **Park**: `prepare_next_generation` restarts (with the 20 s park
deadline) only if no turn-end restart ran. Otherwise it marks the
generation for `run.attach` rebind and returns `false`, so the runner
does not ask for a second park proof. The park-time receipt rotation
moves to the next turn start, where the turn-end path does it too.
- **Restart failure**: the executor sets the same state that a failed
turn-start restart leaves (`closed`, no session, provider exit
confirmed). The next `turn.start` fails with the same `ACPX turn start
failed closed: …` error that the turn-start restart gives today.
- **Fake sidecar**: new `pending-input` fault, which opens an input
request in a turn that then completes.

### Races

| Race | Handling |
| --- | --- |
| A turn or steer arrives during the turn-end restart | The runner loop
is single-threaded. The command waits in the socket until the restart
returns, then starts on the fresh generation without a restart. Only one
generation is live. |
| A turn arrives before the hook runs (for example, in the same read as
the ACK) | Control frames go first, so the turn restarts at its start,
as today. The hook then restarts after that turn. |
| Park arrives during the turn-end restart | Host control is read at the
top of the loop, so park waits. Park then reuses the fresh generation.
No event is published in between, so the host's park cut still matches.
|
| Park arrives before the hook runs | Park restarts with the 20 s
deadline, as in #20145. The hook then finds a fresh generation and does
nothing. |
| Eviction, `runner.suspend` or retire after the restart | They reap the
fresh generation through the ordinary paths. |
| The restart fails | The session is `closed`. The next turn fails with
the same error and the same durable state as a failed turn-start
restart. Park returns conflict, as it does for a closed session today. |
| Cancellation | `run.cancel`, `turn.interrupt` and `turn.stop` after
the restart return `already_settled`, as after any settled turn. A
cancellation that retires the runtime closes the session, so no restart
runs. |
| Interrupted, cancelled or failed turn | `completed_turn_observed` is
false, so no turn-end restart runs. The next turn restarts at its start,
as today. |
| Pending permission or input requests at the terminal | The reducer
ends them at the terminal, so the restart runs. A later
`request.resolve` fails with `no active turn`, with or without the
restart. |
| Active session goal at the terminal | No restart, because the goal can
keep working after its prompt. |

### Removed as redundant, and what stays

- **Removed:** the park-time restart and dormant verify when a turn-end
restart already made the generation fresh. The runner's
`host_park_ready` proves dormancy just before `prepare_for_park`, and
nothing runs in between.
- **Removed:** the runner's second park proof in that case, because the
provider did not change.
- **Removed:** the park-time receipt and ledger rotation. The next turn
start does it once, for both paths.
- **Kept: the turn-start restart.** Without it, a turn queued before the
hook, a turn after an interrupted or cancelled turn, a turn after an
active goal, and any non-Claude ACPX turn would run in the generation
that ran the previous turn. Tests cover the queued-turn and interrupted
cases.
- **Kept: the park-time restart.** Without it, a park that arrives
before the hook would leave the next run to restart at attach. A test
covers this.
- **Kept: the settlement of an abandoned park observation.** Without it,
the restart drops events that the old generation sent before the
abandoned response. A test covers this, and it fails with the settlement
removed.
- **Kept: the goal-dormancy check.** Without it, the restart kills an
active goal's autonomous work. A test covers this, and it fails with the
check removed.
- **Kept: the ACK gate.** Without it, the restart can run before the
host has the terminal result. A runner test covers this.
- **Kept: deferred polling of the fresh generation.** Without it, its
startup event lands after the host computed its park cut, and the park
conflicts.
- **Kept: the kept failure message.** Without it, the next turn fails
with `ACPX session is not open`, and the restart error is lost.

## Verification

- `cargo test -p paperclip-runner-core --lib` and every
`tests/acpx_*.rs`, `native_provider_backend` and `durable_recovery`
suite: all pass (debug).
- New tests (`tests/support/acpx_turn_end_restart.rs`, run from
`acpx_park_preparation`) prove these points:
- A settled turn reaps the old pid before it spawns the new one, and no
predecessor is still live. A same-run follow-up then sends only
`turn.start` and reports the reaped pid in `session.reconciled`.
- Park after a turn-end restart sends no sidecar command and returns
`false`. Attach rebinds the same pid. The run's second turn restarts at
its end, and the third turn starts without a restart.
  - The reaped generation's late callback never reaches the next turn.
- A turn that arrives before the hook restarts at its start. A park
before the hook restarts once.
- A failed turn-end restart gives the same next-turn error, retry error,
durable state and process count as a failed turn-start restart.
- Cancellation after the restart is `already_settled`. `runner.suspend`
reaps the fresh generation.
- An active goal skips the restart. An input request open at the
terminal behaves the same with or without the restart.
- `acpx_provider_turns`: a session-level turn-end restart drops the old
generation's late `issues.delete` callback, which reuses the previous
turn's call ID. The next turn then admits that call ID from the fresh
provider. Interrupted and retired turns do not restart at turn end.
- `acpx_park_readiness`: a turn-end-restarted generation is park-ready.
An abandoned park observation tail is delivered before the restart.
- `durable::runner`: the hook runs only after the terminal is ACKed and
when control is idle. It never runs during ACK debt, a waiting control
frame or a disconnect.
- Server integration tests (`plugin-runner-release-resume`,
`plugin-runner-eviction` and `plugin-runner-task-warm`) with this
branch's Runner: 49 passed. The provider process counts do not change:
each released run still adds one process start, now at turn end instead
of at park.
- `cargo fmt --check`: clean.
- Fake-sidecar timing (release build, 7 runs, medians):

  | | before (`master` with #20145) | this PR |
  | --- | --- | --- |
| same-run follow-up `turn.start` → `turn.started` | 51.5 ms | 22.0 ms |
| second turn after an attach, `turn.start` → `turn.started` | 270.3 ms
| 23.5 ms |
  | turn-end restart, after the terminal is ACKed | — | 29.0 ms |
  | park preparation after a settled turn | 34.1 ms | 0.03 ms |

- Real Claude: not run for this pull request. With real Claude, the
restart took about 1.5 s on the turn-start path, and this change takes
it off that path. That number is not measured on this branch.

## Risks

- A run that ends after one turn without a park now starts one extra
sidecar that no turn uses. The Claude CLI itself starts only at the
first prompt, so the cost is the sidecar's start and bootstrap.
- The restart blocks the Runner loop for its length (about 1.5 s with
real Claude), as the turn-start restart does today. It has no deadline,
as before. A park that arrives during it waits, inside the host's 30 s
park wait.
- The fresh generation is not polled until its next turn. If it exits
while idle, the next turn finds out at `turn.start` and fails closed.
Park finds out at its observation.
- The turn-end restart is for Claude only, the same as park preparation.
Other ACPX agents keep restarting at turn start.
- If the Runner crashes during the restart, the durable state still says
`session_open`. Recovery is the same as for a crash while idle.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with tool use
(shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [ ] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:34:26 -08:00
DottaandClaude Opus 5.5 7d92b6a7cd fix(api): use a neutral catalog pointer in the call_api protocol refusal (#20191)
## Thinking Path

- Paperclip agents reach REST routes through the `call_api` tool;
protocol endpoints (MCP, OAuth, WebSocket) are refused there with
guidance.
- #20170 changed that refusal to tell the agent the endpoint's catalog
description says who can use it.
- Review on #20170 found that some protocol endpoints' descriptions (for
example the WebSocket entries) do not say who may call them, so the
refusal could send the agent to look for information that is not there.
- The fix (e7ff8ba610) was pushed three minutes after #20170 merged, so
it never reached master. This PR carries it.

## Linked Issues or Issue Description

Refs #20170

**What happened:** after #20170, a `call_api` refusal for a protocol
endpoint says the catalog description names who can use it. Some
descriptions, such as `GET /api/companies/{companyId}/events/ws`
("WebSocket upgrade. Not callable through call_api."), do not, so an
agent following the refusal looks for an answer the catalog does not
give.

**Expected behavior:** the refusal points the agent at the catalog
description neutrally ("Read its catalog description for details.")
without promising it names who may call the endpoint.

**Steps to reproduce:**
1. As an agent, call `call_api` with `GET
/api/companies/{companyId}/events/ws`.
2. Read the refusal text: it claims the description says who can use the
endpoint.
3. Read the catalog entry: it does not.

## What Changed

- `server/src/services/native-runtime/runner-api-client.ts`: the
protocol-endpoint refusal now says "Read its catalog description for
details." instead of promising the description names who can call it.
- `server/src/__tests__/assistant-connections-guidance.test.ts`: adds a
case on the WebSocket events endpoint asserting the neutral wording.

## Verification

- `cd server && npx vitest run
src/__tests__/assistant-connections-guidance.test.ts` (4 passed).
- Commit was reviewed on #20170's branch by Ponytail (the finding that
prompted it).

## Risks

- Low: wording-only change to one refusal message, plus a test. Agents
may read slightly less specific guidance for endpoints whose
descriptions do name the caller; the description itself is unchanged.

## Model Used

- Provider: Anthropic. Model: Claude Opus 5.5 (`claude-opus-5-5`), via
Claude Code, with tool use and code execution. The change was authored
by a Claude subagent on #20170 and carried over here by Claude.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:31:49 -06:00
DottaandClaude Opus 5.5 7086019d4b fix(api-catalog): say Agent Chat is always on instead of offering a retired toggle (#20169)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agents learn what the API can do from the API catalog at
`/api/openapi.json`
> - PR #19360 made Agent Chat standard. The `enableAgentChat` key is now
retired: it always reads true, and the app ignores written values
> - The catalog still lists `enableAgentChat` as an ordinary boolean on
the experimental settings operations, with no note that it is retired
> - An agent that was asked to "Enable Agent Chat on this instance" read
the catalog and told the person to turn on a toggle in Instance Settings
→ Experimental. That toggle no longer exists
> - This pull request adds a note to both experimental settings
operations: Agent Chat is always on, the key is retired, and agents must
tell people it is already on
> - The benefit is that agents give a correct answer and do not send
people to a control that is not there

## Linked Issues or Issue Description

Bug report (no public issue exists).

**What happened**
An agent was asked "Enable Agent Chat on this instance." It read `PATCH
/api/instance/settings/experimental` in the API catalog and found
`enableAgentChat`. It replied that an admin must open Instance Settings
→ Experimental and turn on Enable Agent Chat. That toggle was removed in
#19360, and Agent Chat is already on for everyone.

**Expected behavior**
The agent says that Agent Chat is already on (the person is using it),
and that nothing must be enabled.

**Steps to reproduce**
1. Open a chat with an agent on any current instance.
2. Send "Enable Agent Chat on this instance."
3. The agent reads the experimental settings operation in the API
catalog and sends you to a settings toggle that does not exist.

Refs #19360

## What Changed

- `server/src/routes/openapi.ts`: the GET and PATCH
`/api/instance/settings/experimental` operations now say that Agent Chat
is always on, that `enableAgentChat` is a retired compatibility key that
always reads true and ignores written values, and that agents must tell
people it is already on instead of sending them to a toggle.
- `server/src/__tests__/openapi-routes.test.ts`: a regression test
checks this note on both operations.
- The note says Agent Chat "needs no setup" instead of "there is nothing
to enable", so it does not match unrelated keyword searches. The runner
API no-results test (`nothing-zzzzzzzzzz`) had matched the word
"nothing".

## Verification

- `cd server && npx vitest run src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` passes (2108 tests), and
the server `tsc --noEmit` passes.
- The chat eval `instance-operations.instance-settings` found this
defect. Its run log shows that the agent reasoned from the catalog
schema ("The setting is `enableAgentChat` on PATCH
/api/instance/settings/experimental").

## Risks

- Low risk. This changes only catalog description text. It does not
change behavior, validation or the schema.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution, used through Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:11:13 -06:00
DottaandClaude Opus 5.5 b07399c241 fix(authz): link the project's access settings when an agent cannot share a project (#20176)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Some actions are board-only. When an agent calls one, the 403 denial
tells it that the denial is final and, when known, links the board page
that holds the control
> - Project sharing (`/api/projects/:id/access-members`) had no known
page, so the denial told agents to link a page only if they already knew
its route
> - An agent asked to "Let Riley see the acquisition project" got this
denial. It told the person to look for "access/sharing settings" and
gave no link, so the person had to search for the control
> - This pull request maps project access-member routes to the project's
Configuration page, where the Manage access control is
> - The benefit is that agents give people a direct link to the control
that shares a private project

## Linked Issues or Issue Description

Bug report (no public issue exists).

**What happened**
An agent tried to add a person to a private project. `POST
/api/projects/:id/access-members` returned 403 "Board access required"
without a page link. The agent told the person to find the project's
access or sharing settings in the web app and gave no link.

**Expected behavior**
The denial links the project's Configuration page
(`/<prefix>/projects/<id>/configuration`), where Manage access is. The
agent gives that link to the person.

**Steps to reproduce**
1. Create a private project and give an agent access to it.
2. In chat, ask the agent to let another company member see the project.
3. The agent's access-member request is denied, and its reply has no
link to the sharing control.

Refs #20151

## What Changed

- `server/src/routes/authz.ts`: board-only denials for
`/projects/:id/access-members` (and its member subpaths) link "the
project's access settings" at `/<prefix>/projects/<id>/configuration`.
The link uses the resolved route project ID, so short references such as
`/api/projects/acquisition/access-members` also get the link. Without a
resolved UUID, the denial falls back to the generic instruction.
- `server/src/__tests__/board-access-denial.test.ts`: cases for add and
remove access-member denials, plus a short-reference case that goes
through a route param resolver.

## Verification

- `cd server && npx vitest run
src/__tests__/board-access-denial.test.ts` passes (17 tests). The
short-reference test fails on the previous matcher. The server `tsc
--noEmit` passes.
- The chat eval `projects.share-project` found this gap: in one run the
agent linked the project, and in the next run it gave no link.

## Risks

- Low risk. This changes only the remediation text in an agent's 403
response. It does not change authorization.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution, used through Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:11:08 -06:00
DottaandClaude Opus 5.5 2fc0293586 fix(api): tell agents only the person can revoke assistant connections, and where (#20170)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect external assistants to Paperclip, and each person
manages their own assistant connections in the browser
> - Chat agents find API routes through `search_api` and call them with
`call_api`
> - The catalog entries for `/api/mcp/connections` did not say who can
use them or which page holds the control
> - `call_api` refuses these routes as protocol endpoints, and its
message did not say the refusal is final
> - A chat agent asked to "disconnect my old assistant" searched for an
agent route for its whole turn and did not reply
> - This pull request names the person-only boundary and the page in the
catalog, and makes the refusal final
> - The benefit is that the agent replies at once with a link to the
page where the person revokes the connection

## Linked Issues or Issue Description

No public issue exists. This is a bug report.

**What happened**

A person asked the chat agent: "Disconnect my old assistant, but leave
the website task running." The agent found `DELETE
/api/mcp/connections/{id}`. `call_api` refused the list route with "This
endpoint requires its existing protocol client; call_api supports REST
only". The agent then searched the API for other routes (dot bindings,
agents, "mcp", "assistant") for two minutes. The turn timed out with no
reply. The agent wrote in its reasoning that there "must be a supported
path".

**Expected behavior**

The agent tells the person that only they can revoke the connection,
links the Assistant connections page, and says the delegated task keeps
running.

**Steps to reproduce**

1. Connect two assistants to a company as one person. Let the first
assistant create a task for an agent.
2. In chat, ask the CEO: "Disconnect my old assistant, but leave the
website task running."
3. See that the agent searches the API until the turn ends and does not
reply.

## What Changed

- `server/src/routes/openapi.ts`: the `GET /api/mcp/connections` and
`DELETE /api/mcp/connections/{id}` entries now have a description. It
says only the signed-in person can list or revoke their assistant
connections, links `[Assistant
connections](/<prefix>/apps/assistant-connection)`, notes that the
connected assistant itself can revoke its own token through OAuth token
revocation, says not to search for another route, and says that tasks
the assistant already created keep running.
- `server/src/services/native-runtime/runner-api-client.ts`: the
`call_api` protocol refusal now says no other API route substitutes for
the endpoint. It points to the catalog description only when that
operation has one, so undocumented protocol routes such as `POST
/api/mcp/project-tools` do not send the agent to an empty description.
- `server/src/__tests__/assistant-connections-guidance.test.ts`: new
regression test. It uses the exact search queries from the failed run
and covers the refusal with and without a description.
- `server/src/__tests__/public-mcp.test.ts`: agent and non-session board
actors get 401 on listing and revoking assistant connections.

## Verification

- `cd server && pnpm exec vitest run
src/__tests__/assistant-connections-guidance.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` passes (2111 tests).
After the review fixes,
`src/__tests__/assistant-connections-guidance.test.ts
src/__tests__/public-mcp.test.ts
src/services/native-runtime/runner-api.test.ts` passes (2157 tests), and
the server `tsc --noEmit` passes.
- The new test fails on `master` without the change (2 failures) and
passes with it.
- The end-to-end check is the
`assistant-connections.revoke-personal-grant` chat eval with this commit
cherry-picked.

## Risks

- Low risk. The change is catalog text and one error message. No route,
permission or data behavior changes.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-09 00:08:31 -06:00
Devin FoleyandPaperclip 9b71d03215 fix: recover interrupted native preparations and unblock upgrades (#20177)
## Thinking Path

> - Paperclip manages AI-agent companies and their work.
> - A native run prepares its workspace before a runner PID or
checkpoint exists.
> - A server crash can interrupt that stage and leave its controller on
another host.
> - Recovery needs a durable owner proof, a delivered recovery wake, and
the saved execution contract.
> - The old paths missed controller lease registration and boot proof,
skipped checkpoint-free preparations, and counted retained work as live
upgrade blockers.
> - This PR connects preparation to the existing controller lease and
recovery paths, and separates retained work from the upgrade census.
> - The result is bounded recovery without a second executor or loss of
saved work.

## Linked Issues or Issue Description

**What happened?**

A native preparation interrupted by a server crash could remain running
indefinitely with no runner PID or checkpoint. Native controller stamps
did not wake the shared instance lease. That lease was disabled on
volume-backed Railway services, and native takeover omitted the boot
binding needed for its cross-host proof. Scoped recovery selected
checkpoint-backed runs but skipped these preparations. The upgrade probe
then counted the retained row as current execution and could block every
later upgrade.

**Expected behavior**

Track the preparation's controller before a runner starts. Recover it
through the normal classifier after proving its owner stopped or fencing
that boot. Keep live or unprovable owners held. Deliver recovery wakes
to preparations without checkpoints. Preserve unresolved historical work
and report it separately from current execution during upgrade
observations.

**Steps to reproduce**

1. Save an observed native preparation with no runner process, result,
provider-start evidence or checkpoint.
2. Replace its controller with a process in another PID space. Let its
owner lease expire and send a scoped recovery wake.
3. Before this fix, recovery skips the checkpoint-free preparation or
cannot prove its foreign controller stopped. Upgrade readiness continues
to count that old row as a live blocker.

Related work: #18037, #19651, #19868, and #20168. The foreign-PID guard
from #19868 remains. The queued first-turn input fix from #20168 is
included through the current base.

## What Changed

- Register the existing instance lease when native execution and restart
claims are written. Keep passive identity reads free of ownership
effects. Keep that lease enabled on volume-backed Railway services
unless explicitly disabled.
- Preserve the existing PID/start-time proof within the same readable
PID space. Bind foreign or unknown-space takeover to its saved boot and
last write. A live lease or grace period keeps the run held with a
scheduled check. Repeated evidence waits preserve their timestamp.
- Include explicitly fenced, checkpoint-free preparations in the
existing finite recovery candidate inventory. Normal launch, checkpoint,
provider evidence and admission checks still decide whether work may
resume.
- Expose the existing board stop-confirmation operation for historical
foreign-host owners. Bind confirmation to the exact run, boot, PID,
process start and host. Reject changed ownership.
- Report narrowly defined old, unowned preparations as
`retainedBlockedRuns` outside the live upgrade census. Local executions,
pending admission, drain state, truncation and positive checkpoint proof
retain their barriers.
- Update the execution contract and add PostgreSQL, real-route, race and
preservation regressions.

## Verification

- Genuine red/green regressions cover the old upgrade census, missing
native lease registration, the volume-backed default, foreign-host
confirmation, and scoped recovery skipping checkpoint-free preparation.
Test bodies ran against real PostgreSQL before their corresponding
production changes.
- Combined verification before the final idle correction (head
`055be6c84d9`): **459 passed, 22 skipped, 11 files passed**. Covers
controller lease/identity, native restart classification and
integration, host confirmation, heartbeat recovery, instance route
authorization, upgrade readiness, and queued first-turn input. The
focused command used `pnpm exec vitest run` with those 11 suites and
`--maxWorkers=1 --no-file-parallelism`. The 22 skipped cases require the
Rust debug fake-Codex fixture, which was not built locally; the
real-process CI coverage must pass.
- `pnpm --filter @paperclipai/server typecheck`: passed on head
`055be6c84d9b058064daad0af71bb5d893ab9529`.
- Actual route proof uses the real heartbeat service and PostgreSQL.
Unauthorized access returns 403. The retained run and owner records
remain unchanged. Recovery integration admits one bootstrap claim across
two replacement controllers, keeps a live owner held, and preserves
local execution.
- `pnpm verify`: exit 2, requests broad CI because the execution
contract changed. Full local suite not run. Current-head CI must supply
broad coverage.
- Local source checkout on macOS ARM64, Node v25.6.1, pnpm 9.15.4. Tests
use isolated PostgreSQL. No live provider restart is claimed by those
fixtures.
- CI found an idle-query regression in the first version of the lease
wiring. Two focused PostgreSQL regressions reproduced the misplaced
identity-read notification and the missing claim notification. The fix
moves registration to actual ownership writes. `env -u npm_execpath pnpm
exec vitest run server/src/__tests__/maintenance-server-routes.test.ts
--maxWorkers=1 --no-file-parallelism`: **17 passed**, including 90 real
seconds and two qualified hourly windows with no extra idle queries. An
initial local setup attempt inherited npm metadata and did not run
tests; clearing that metadata selected the correct pnpm helper. Three
focused controller/native suites also passed **168 tests, 22
fixture-gated skips**. Server typecheck passed after the correction on
head `4a527326dcd5b3a9e286f0e4c3e761d3056af4d9`.
- The older CI server shard also exposed a same-space graceful-restart
regression. The new boot fallback had overridden local process-exit
proof. Seven suspension cases and a focused unit regression reproduced
it; the fix limits boot fallback to foreign or unknown PID spaces. Fresh
verification on `acf1065d98150210c1e227c7f3b05a1eb489730b`: **251
passed, 22 fixture-gated skips, 4 files passed**, including all **40**
graceful-restart suspension tests. Server typecheck passed. The local
regression failed before this correction, and the same-space and
foreign-space paths pass together after it.
- Current-head CI completed successfully on
`acf1065d98150210c1e227c7f3b05a1eb489730b` (run 42568109784). Greptile
reviewed that exact head at 5/5 with no unresolved review threads.
- Secret and private-data scans passed. No workflow, lockfile or
migration changes.

## Risks

- The instance lease now applies to volume-backed Railway services. Its
existing 30-second renewal, quiescence, grace and self-fence behavior
applies while work is held. Idle servers retain the existing no-query
behavior. An explicit `PAPERCLIP_CONTROLLER_INSTANCE_LEASE=off` remains
available.
- A controller lease proves only the server owner's state. It does not
prove an agent process stopped or authorize replay, resource release or
volume removal. The existing classifier and provider/runner checks
remain responsible for those decisions.
- Previous-release records keep their shape and saved state. A missing
boot lease stays unprovable and requires the scoped stop-confirmation
path. A changed confirmation target is rejected.
- The upgrade observation does not recover a held task by itself. Source
tests do not qualify a hosted replacement. Deployment and the affected
task's useful result must be verified separately.
- This repairs recovery after a crash; it does not claim to identify or
fix every source of heap exhaustion.

## Model Used

OpenAI `gpt-6.1-sol`, xhigh reasoning effort, through Codex in T3 Code,
with repository, shell, and GitHub tools. Context window size is not
exposed by the runtime.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1109.0-canary.6
2026-11-08 21:53:13 -08:00
Nicky LeachandClaude Opus 5.5 2d93413085 perf(runner): prepare the next Claude provider generation at park (#20145)
## Thinking Path

> - Paperclip runs agents through Paperclip Runner, and a host can park
an idle Runner between runs and resume it for the next run on the same
task.
> - For Claude (ACPX), a follow-up run on a parked Runner pays about
1.5–2.3 s at `run.attach` before the provider can start the turn. The
attachment closes the Claude sidecar and spawns a new one: initialize,
`session.open` with `session/load`, `run.attach`, then a dormant verify.
> - The respawn is required. The tool-bridge MCP endpoint lasts for the
whole session and can't tell which turn a late callback came from, so a
new turn must not run in a process that ran an earlier one.
> - Nothing requires the respawn to happen at the *start* of the next
run. A turn that ends 2 s later is a better experience than one that
starts 2 s later.
> - This pull request does the respawn while parking. The next run's
`run.attach` then only rebinds the run id on a fresh, never-used
generation.
> - Follow-up Claude runs on a parked Runner reach the provider about
1.5–2 s sooner, and the late-callback guarantees are unchanged.

## Linked Issues or Issue Description

No public issue exists. The issue is described inline.

### What happened?

On a hosted deployment with parked Runners, Claude follow-up runs
reached the provider about 1.8 s later than Codex follow-ups.
`runner.session.resume` medians were 5.2 s for Claude and 3.4 s for
Codex. The extra time is the ACPX respawn inside `run.attach`. Codex
keeps its process in place.

### Expected behavior

The respawn happens while the Runner parks, so the next run starts on a
prepared generation.

### Steps to reproduce

1. Run a Claude (ACPX) task on a host that parks Runners between runs.
2. Let the run finish and the Runner park.
3. Send a follow-up. `run.attach` closes the provider and starts,
initializes, opens and attaches a new sidecar before `turn.start`. The
Claude CLI itself starts at the first prompt on either path.

### Paperclip version or commit

`master` at e194e193.

### Deployment mode

Hosted, with parked Runner tasks.

## What Changed

- **`durable/runner.rs`:**
- New `CommandExecutor::prepare_for_park(deadline)` hook, a no-op by
default. The runner passes a deadline 20 s after the park request; the
host waits 30 s for the reply.
- The Park host-control arm calls it after `host_park_ready()`,
re-proves readiness, and only then replies `parked`.
- A failure records a diagnostic and replies `conflict`, the existing
park-failure path.
- **ACPX, park side** (`acpx_park_preparation.rs`,
`acpx_provider_generation.rs`):
- For Claude, with the session open, nothing pending and nothing queued:
reap the old generation, using the same suspend-and-reap sequence a
second turn uses.
- Start the replacement with the current run attached, and prove it
dormant.
  - Roll the tool receipt epochs and the turn-identity ledger over.
- Record a digest of the session configuration it was opened with,
ignoring the MCP binding id and the aggregate digest.
- Every sidecar request during preparation is capped at the park
deadline. Running out of time fails like any other preparation failure.
- **ACPX, attach side:**
- A prepared, never-used generation whose digest matches the next run is
rebound: the sidecar gets `run.attach` with the new run id, the reducer
and tool bridges are reset, and the snapshot is checked for the new run
id.
- Anything else takes today's cold path: a different tool set, skill,
prompt or launch; a buffered callback; or any rebind error.
- **ACPX, turn start:** the provider restarts when its generation has
run a turn, instead of when the run has settled turns.
`has_settled_turns` is removed.
- **ACPX, polling:** polling skips a prepared generation until it is
rebound or starts a turn, so nothing it emits is attributed to the
parked run.
- **Sidecar** (`acpx-runtime-sidecar.ts`): rejects `run.attach` once a
turn has started in that process.
- **Fake sidecar:** reports live predecessors at process start, and adds
the faults `reject-open`, `late-tool-after-suspend` and `pre-turn-tool`.

## Verification

- `cargo test -p paperclip-runner-core`: 1029 passed (debug).
  - New `tests/acpx_park_preparation.rs` (10 tests). These prove:
- park reaps the old pid before spawning the new one, and acks only
after the dormant verify;
- a late callback from the old generation after park is rejected, and so
is a pre-turn call from the new generation;
- a compatible attach keeps the pid, and the sidecar and snapshot carry
the new run id;
- changed instructions, tools, skills or runtime context take the cold
path;
    - a used generation is never rebound;
- a prepare failure produces a park conflict and a recoverable state;
    - Retire works after a prepared park;
- a replacement whose `session.open` stalls fails at the park deadline,
leaves the session suspended, and the next run attaches and completes a
turn.
- `cargo fmt --check` and `cargo check --workspace --locked`: clean.
- `vitest` `acpx-runtime-sidecar.test.ts` and
`acpx-sidecar-input.test.ts`: 126 passed. The new re-attach guard test
fails if the guard is removed.
- `vitest` `plugin-runner-release-resume.integration.test.ts` and
`plugin-runner-eviction.integration.test.ts` with the branch's Runner:
48 passed. Both now expect the provider process to be replaced at park
instead of at attach.
- Fake-sidecar timing (release build, 3 runs): `run.attach` took 230–260
ms cold and 40–80 ms after a prepared park.
- Real Claude (claude-agent-acp 0.73.0, release build on a hosted test
VM, 3 runs each way plus one with a 180 s cgroup freeze between park and
attach):

  | | before | this PR |
  |---|---|---|
  | `run.attach` | 1691 ms mean | 212 ms mean |
| `run.attach` start → follow-up `turn.started` | 2565 ms mean | 1065 ms
mean |
  | `turn.start` → first provider output | 2261 ms mean | 2232 ms mean |
  | park preparation added to park | — | about 1450 ms |

Session recall, an authorized tool call after the rebind, and the
absence of any tool call or event from the old run all passed in every
run, including after the freeze. The prepared generation holds only the
sidecar: the Claude CLI starts at the follow-up prompt on both paths, so
a parked Runner also holds one node process instead of the whole Claude
tree.
- Hit rate: on a hosted test deployment, all 15 consecutive Claude runs
across two tasks carried identical content-addressed instruction and
skill bundle paths, so the prepared digest would have matched.
- Not run: the hosted park, claim and attach loop with real Claude, a
native MCP binding or skills with real Claude, `cargo test --release`,
smoke tests.
- Pre-existing flake:
`unconfirmed_suspension_state_becomes_recoverable_only_after_cleanup_and_suspend`
fails intermittently on clean `master` when the `hosted acpx durable::`
subset runs in parallel. It passed in both full runs.

## Risks

- **Park takes about 1.5 s longer** before it replies with real Claude.
Preparation is capped at 20 s; the host's park wait is 30 s.
- **Configuration changes:** a change between park and the next run
(registered asset paths, tools, prompt) takes the cold path. That's
safe, but saves nothing.
- **Stale environment:** a prepared process was opened with the previous
run's environment and disk state. If credential or configuration files
change on disk while the Runner is parked, the prepared process won't
see them, whereas the cold path would.
- **Failed prepare:** a park that fails after the reap leaves no live
provider. The next turn or `run.attach` starts Claude cold, as today.
- **Scope:** only ACPX Claude prepares. Other providers park exactly as
before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, with tool use
(shell, git, GitHub CLI) and subagents.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:38:57 -08:00
DottaandClaude Opus 5.5 0ddc9f1ef1 test(chaos): count an M1 subtask stall only when its recovery needs the board (#20178)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The M1 chaos oracle checks that a parent gets exactly one
`issue_blocker_stalled` wake per stalled subtask (#20147)
> - The product wakes a parent only for a stall that needs the board.
The harness recorded a stall for any blocked subtask, whatever its
latest recovery action
> - In the Daytona M1 wave, a subtask blocked twice on agent- and
system-owned recovery that resolved by itself. The harness counted two
stalls and failed the parent for getting no wake, a false failure
> - This pull request records a stall only when the subtask's latest
recovery action needs the board, matching the product's definition
> - The benefit is that `parent_woken_once_per_stall` fails only for a
wake the product owed

## Linked Issues or Issue Description

Refs #20147, #20098

**What happened?**
In wave `w-20261109T041248-s9501`, t10's subtask 195b5fd7 was blocked
twice. Action 56d036e0 was agent-owned (`native_session_interrupted`)
and was cancelled. Action 53eb91b0 was system-owned
(`native_continuation_requires_reconciliation`) and resolved with
outcome cancelled. Neither needed the board, so the product correctly
sent no stall wake. The harness still failed
`parent_woken_once_per_stall` with 2 stalls and 0 wakes.

**Expected behavior**
A stall counts when the subtask's latest recovery action is:
- open (active or escalated) and board-owned, or escalated; or
- board-owned and resolved with outcome blocked, with the subtask still
blocked.

That is the definition the product's blocker-stall wake uses.

## What Changed

- `observeChildStalls` reads the parent's `stalledDependencies` from
`GET /api/issues/:id`. That is the product's own stalled-dependency view
(`listStalledDependencies`, built with the same `readStalls` the
blocker-stall wake uses).
- Each `relation: "child"` entry is recorded as a stall
(`stalledIssueId`, `recoveryAction.id`). The harness no longer infers
stalls from `/recovery-actions`. That endpoint lists only open actions
and selected holds, and can omit a board action resolved with outcome
blocked.
- Clicking the board Retry on a blocked subtask is unchanged: it still
happens whenever the product offers Retry, and it still counts as a
needed Retry.
- Tests:
  - a stall the product lists is recorded;
- for t10's two non-board recovery states, the product lists nothing, so
no stall is recorded, and the subtask's Retry is still clicked.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 212 passed.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- The harness checks `stalledDependencies` every 10 s. A stall that
appears and clears inside one interval is not recorded, so its wake goes
unchecked. It is never reported as a false failure.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:36:40 -06:00
DottaandClaude Opus 5.5 95bf90bd2c fix(chat): post a conversation turn's answer while an approval holds the task (#20179)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - People talk to agents in chat. The agent's final message becomes the
chat reply.
> - A chat turn can end while the task stays held. For example, the
agent resubmits a hire approval and the approval is pending.
> - The finalizer keeps the task in review for the approval
(`governed_response_waiting`). That decision is not
`conversation_turn_finished`.
> - Run completion then treats the yielded turn as a control-plane wait.
It records `presentationDecision: none` and posts no comment.
> - The person never sees the agent's answer. The chat shows nothing
after the request.
> - This pull request counts a committed conversation turn as answered
unless its own pending question or confirmation card presents it.
> - The benefit is that the person always gets the reply, whatever state
the task keeps.

## Linked Issues or Issue Description

Refs #19971 (post the full final reply). That fix covers the reply text.
This fix covers replies that were never posted.

**What happened**

In a chat, the person asked the CEO agent to change a proposed hire to a
cheaper model and send it back for approval. The agent resubmitted the
approval correctly. It called `paperclip_finish`, and Paperclip accepted
the report with "task is still waiting for approval". The run succeeded
and the task moved to `in_review`. The run event
`run.presentation.resolved` recorded `commentAction: none` (reason
`yielded_control_plane_wait` on master). The agent's final message never
reached the chat.

**Expected behavior**

A chat turn's accepted final answer is always posted to the
conversation. An approval, an execution stage or a dependency blocker
can hold the task. It must not hide the reply. Only a question or
confirmation card that the turn itself left pending presents the turn
instead of its prose.

**Steps to reproduce**

1. Open an Agent Chat with an agent. Link a hire approval in
`revision_requested` to the chat task.
2. Ask the agent to change the hire and resubmit it.
3. The agent resubmits the approval and ends the turn with a yielded
result and a response wake.
4. See that the chat shows no reply. The run is `succeeded` and its
presentation decision is `none`.

## What Changed

- `server/src/services/native-runtime/native-governance.ts`: move the
pending-card query of `pendingNativeGovernance()` into
`pendingNativeInteraction()`, with no change in behavior. Add
`conversationTurnAnswered()`. It returns false only when a pending
thread interaction (the same predicate as status arbitration) gates the
turn. It checks cards on their own, so a pending execution stage does
not hide a card.
- `server/src/services/heartbeat/run-completion.ts`: a successful
conversation run whose status decision committed now passes
`conversationTurnFinished: true` to the presentation resolver, unless
`conversationTurnAnswered()` is false. The existing
`conversation_turn_finished` path is unchanged.
- `server/src/services/heartbeat/run-completion.test.ts`: add a
regression table with a real database. A pending approval, a
revision-requested approval, a pending execution stage and a dependency
blocker now post the answer. A finished turn still posts. The turn's own
question card still owns the response, also when an execution stage is
pending.
- `doc/execution-semantics.md`: state the rule.

Other finish outcomes I checked:

- `done` or `needs_review` with a gate (`governed_gate_pending`): the
result is not yielded, so the reply was already posted.
- `blocked` (`paperclip_block`, waiting on a person or another task):
not yielded, so the reply was already posted.
- Yielded with a dependency blocker
(`durable_dependency_blocker_bound`), or with
`record_finalization_error` (`prior_status_preserved_no_live_path`,
`monitor_wait_authority_lost`): the reply was also dropped. This fix
covers them, because the rule does not depend on the reason code.

## Verification

- `cd server && npx vitest run
src/services/heartbeat/run-completion.test.ts`: 118 passed. Before the
fix, 4 new cases failed (both approval states, execution stage,
dependency blocker). The finished-turn and question-card cases passed
both before and after.
- `cd server && npx vitest run
src/__tests__/heartbeat-run-summary.test.ts
src/services/heartbeat/run-comment-selection.test.ts
src/__tests__/slack-conversation-lifecycle.test.ts
src/services/native-runtime/agent-chat-response-recovery.test.ts
src/__tests__/chat-run-milestone-governed-work.test.ts
src/__tests__/agent-conversations.test.ts`: all passed.
- `tsc --noEmit -p server`: no errors in the changed files. Unrelated
errors in this worktree come from an unbuilt local Runner package.
- An end-to-end chat eval run on the case branch is in progress. I will
add its result here.

## Risks

- Low risk. The change only affects successful Agent Chat runs with a
committed status decision. Ordinary tasks and external chat (Slack,
GitHub) keep their own paths. External chat returns before this check.
- A turn that yields while an approval is pending now posts its prose.
The approval stays pending and the task status does not change.
- One extra small read (the task row and pending gates) runs after a
conversation turn that is not `conversation_turn_finished`. It runs
after the provider turn ends, so it does not add time before the first
token.
- Previous-release state: no persisted shape changes. The fix reads
`finalizationPhase` and `finalizationReasonCode`, which the previous
release already writes. A run in flight at the deploy completes on the
new code and posts its reply. A run that completed before the deploy
keeps its recorded decision; nothing re-presents it.
- Known gap, unchanged: background presentation recovery for
conversations still requires `conversation_turn_finished` with a `done`
result. If the server restarts between the status commit and the live
presentation, a held yielded turn is not re-posted. That is a separate
follow-up.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use (shell,
file edits, test runs) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:17:36 -06:00
DottaandClaude Opus 5.5 c5c587f162 fix(issues): start a deferred children-completed wake the release drain missed (#20180)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A parent that delegates waits for its children, and is woken when
they finish (`issue_children_completed`)
> - Restart chaos t06 ended with both children done and the parent
blocked forever
> - The children finished while the parent's run was still going, so the
wake was deferred as a follow-up. That run then ended with the agent
setting the parent blocked, and the deferred wake was never started
> - The durable re-drivers for deferred wakes cover only
`issue_blockers_resolved`, so nothing else picks up a missed
`issue_children_completed`
> - This pull request makes both re-drivers cover children-completed
wakes too
> - The benefit is that a parent always hears that its children
finished, even when the release drain misses the deferred wake

## Linked Issues or Issue Description

Refs #19831 (durable ownership plan). Restart chaos wave
`w-20261109T041248-s9501`, task t06 (claude, M1, K2), Daytona per-turn,
master fa7950121d.

**Problem:** parent CHA-10 (all times UTC):
- 04:29:57: run 9cf49f52 started.
- 04:32:35 and 04:32:43: its children CHA-13 and CHA-14 finished.
- 04:32:43: `issue_children_completed` merged into the running turn.
- 04:32:53: that wake was deferred as a follow-up
(`issue_execution_deferred`, `deferred_issue_execution`, no run).
- 04:33:13: the turn ended with the agent setting the parent `blocked`,
a decision made before it saw the children's results.

The deferred wake stayed deferred for the rest of the wave, including
three later restarts. The parent sat blocked with no Retry. The
deferred-receipt worker (`deferred-receipt-work.ts`) and the
stranded-queue resume (`queue.ts`) select deferred dependency wakes with
`issueBlockersResolvedWakePayloadSql`, which matches only
`issue_blockers_resolved`.

## What Changed

- `server/src/services/issue-dependency-wakeups.ts`:
`dependencyResolvedWakePayloadSql` and `isDependencyResolvedWakePayload`
replace the blockers-only predicates. They match both
`issue_blockers_resolved` and `issue_children_completed` in
`_paperclipWakeContext.wakeReason`.
- The deferred-receipt worker and the stranded-queue resume use them, so
a deferred children-completed wake is started once the parent has no
live run, the same way a blockers-resolved one is.
- No code comments added.

## Verification

Red/green:
- `deferred-execution-receipt-work.test.ts` "starts a deferred
children-completed wake left behind when its parent's run blocked the
parent (restart chaos t06)". It sets up a blocked parent whose last run
succeeded, a done child, and the deferred `issue_children_completed`
receipt as t06 left it.
  - On master: the receipt stays `deferred_issue_execution` (red).
- With this PR: the receipt completes with exactly one
`issue_children_completed` run.
- `deferred-execution-receipt-work` and
`heartbeat-dependency-scheduling` pass (60 tests). `tsc --noEmit -p
server` is clean.

## Risks

- The workers now also select deferred children-completed wakes.
Admission is unchanged: they still pass the queue's own gates, and they
start only when the issue has no live run, the same as blockers-resolved
wakes.
- No schema change.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above (#20069, #20108)
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes (no
user docs affected)
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green (pending)
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
(pending)
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 23:14:47 -06:00
DottaandClaude Opus 5.5 5a51a7685d fix(budgets): raise_budget_and_resume continues the work its budget stopped, once (#20144)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop cancels the scope's running work; the board
resolves it with Raise budget and resume
> - In the restart-chaos H1 scenario, the cancelled tasks did not
continue after resume and each needed a manual Retry
> - Resume is the board's explicit decision to continue, so the work it
stopped should continue once, as if the board clicked Retry
> - This pull request files that Retry automatically, keyed by the
cancelled run, once the run's stop has settled
> - The benefit is that a budget raise restores the work without manual
Retries, without repeating completed work, and without charging the
run's attempt budget

## Linked Issues or Issue Description

Refs #20143 (bug B; this PR builds on it), #20113 (bug A), #20101
(superseded: split into #20143 and this PR)

**What happened?**
Restart-chaos waves w-20261108T195944-s42 and w-20261108T210118-s9802
(H1). After `raise_budget_and_resume`, CHA-3's budget-cancelled run
(enforcement stop) and CHA-1's run (agent lifecycle pause from the
budget hold) stayed cancelled until someone clicked Retry.

**Expected behavior**
Resuming continues each task its budget stopped, exactly once, with no
attempt charged, an activity entry, and no repeat of work that already
continued. A run a person stopped stays stopped.

**Steps to reproduce**
Set a budget below spend while a task runs; let the stop cancel the run;
resolve with Raise budget and resume. Before this change, no new run is
queued.

## What Changed

- New table `budget_resume_claims` (migration 0451). One row per
cancelled run, outside the run's result JSON, so a Stop that settles
during the resume cannot erase it.
- `markBudgetCancelledWorkForResume` (called by `resolveIncident` before
the incident closes): claims every task run its budget stopped since the
incident opened. That covers runs budget enforcement fenced, including
stops still in flight, and runs the agent lifecycle pause stopped for
this budget (`pauseStop`, carried from the lifecycle operation's
`pauseCause` so a resume before the Stop keeps the link). Runs a person
stopped are excluded. Two indexed passes led by `(company_id, status)`.
- `deliverBudgetResumeContinuations`: once a claimed run's stop settles,
files the resolving board member's Retry (`requestTaskRetry`,
idempotency key `budget-resume:<run>`, with the member's session source
and the chat Retry callback). The Retry path checks, under the task
lock, that the run is still the task's latest and the task is open, so
continued or completed work is never repeated. Claim settlement and its
activity entry commit together; retryable refusals and errors retry.
- Delivery runs in the always-on budget-enforcement worker after each
pass's Stop delivery. Its schedule includes deliverable claims and the
earliest stale-claim expiry. The execution-continuation worker is a
backstop.
- `agent.paused` / `agent.resumed` logging is in #20113.
`budget.cancelled_work_resume_requested` (incident) and
`budget.cancelled_work_continued` / `_not_continued` (task) are recorded
here.

## Verification

- New `budget-resume-continuation.test.ts` (Postgres, real
`resolveIncident` → claims → `requestTaskRetry` → real `resumeTaskRetry`
into a queued run):
- each task continues once, including a blocked task; repeat delivery or
resolve saves nothing new;
- a stop still in flight at resume, whose terminal write replaces the
result JSON, is still continued;
  - a 30-task scope is fully continued;
- a lifecycle-pause stop caused by the budget is continued, and one
caused by a manual pause is not;
- continued work, done tasks, a run that finished before its stop, and
person-stopped runs (including ones enforcement also fenced) are not
continued.
- `lifecycle.test.ts`: the pause records its cause on the operation and
on stopped runs, including when the budget is resumed before the pause's
Stop. `budget-enforcement-worker.test.ts`: delivery after Stop delivery,
and a wake at claim expiry.
- 287 tests across the affected suites passed. Server typecheck
(`typecheck-layers`, low-memory): no errors.
- Not run: the H1 chaos rerun (Phase 3 runs it after merge).

## Risks

- Resume now continues budget-stopped tasks automatically. A board that
wants a task stopped can Stop it again; person-stopped runs are never
continued.
- A new table and one index (migration 0451).

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:56:28 -06:00
DottaandClaude Opus 5.5 61fa13c6d8 test(chaos): add S1, a board Stop across a restart then a continue comment (#20161)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness drives real user actions across
control-plane restarts: comments, Retry, approvals and budget resumes
> - The board's Stop button on a running task was not exercised. The
cancel route stamps the stop as operator-initiated so that recovery
stands down instead of re-waking the agent
> - A restart right after a Stop is where recovery could wrongly restart
work the operator stopped, or lose the operator's later "continue"
direction
> - This pull request adds S1: stop the running run, fire a fault right
after, then comment to continue
> - The benefit is a graded check that a Stop holds across a restart and
that a later continue comment finishes the task

## Linked Issues or Issue Description

Refs #20071

**What is missing?**
The campaign's scenario list includes mid-run stops. C1 covers a mid-run
comment (a steer), but no workload clicks Stop. Without one, a
regression where recovery restarts operator-stopped work, or drops the
continue comment, would go unseen.

**Expected behavior**
- After the board stops a run and a restart follows, the stopped run
stays cancelled.
- Nothing runs until the operator comments.
- The continue comment then finishes the task.

## What Changed

- S1 workload ("Stop while running, then continue"): a brief task. At
its follow-up time, while a run is running, the observer posts `POST
/api/heartbeat-runs/:id/cancel` (the board's Stop) and records `stop`.
- New `stop` trigger, for S1 only: the planned fault fires 0–2 s after
the stop.
- The continue comment is posted only after the stop has settled: at
least 20 s after the stop, with no run running or queued and no fault in
progress. It asks the agent to continue and add a `CONTINUE-<nonce>`
line.
- The oracle requires:
  - status done;
  - the brief with the chaos and continue markers;
  - `stop_took_effect`: the stopped run ended `cancelled`;
- `nothing_ran_after_stop`: no run was created between the stop and the
continue comment.
- S1 is not a default workload. There is a README section.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 205 passed. New tests
cover:
- the S1 grade (pass, a run restarted after the stop, the stop not
taking effect, no stop);
  - the trigger-to-workload pairing;
- an observer test that clicks Stop once and posts the continue comment
only after the 20 s settle.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.
- A live S1 wave runs after the current chaos wave. Its result will be
posted here before merge.

## Risks

- If the product moves the stopped run to a status other than
`cancelled`, `stop_took_effect` fails. That check is intentional.
- The continue comment waits for the stop to settle, so a run stuck
`queued` delays it. R5 and the wave deadline still bound the wait.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:52:48 -06:00
DottaandClaude Opus 5.5 b54317d3fb fix(native): give a preparing run's first turn the user's queued messages instead of interrupting it (#20168)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A Paperclip Runner (native) run prepares its workspace and
environment before it starts its first provider turn. This can take
about a minute.
> - A user's message that arrives during that time waits as a deferred
wake behind the run. The Interrupt control then cancelled the run before
any provider turn.
> - The cancel threw away the whole preparation. A new run had to
prepare again, so the user waited longer for the first token.
> - This pull request gives the run's first turn the queued messages.
The run seals its first-turn input immediately before `turn.start`, so
the messages are part of the prompt that the turn starts with.
> - The benefit is that the user sees the message handled in the same
run, with no cancel and no second preparation.

## Linked Issues or Issue Description

**What happened?**
A Paperclip Runner run took about 54 s to prepare (environment startup,
workspace, runtime selection). The user sent a second message 32 s into
that time and clicked Interrupt 21 s later. Interrupt cancelled the run
as `operator_interrupted` ("Interrupted to send queued messages") before
any provider turn. A new run then prepared again from the start. The
runtime-selection transaction that froze the run's input completed about
5 s before the click, and the coordinator claim came 4 s after it, so
the message could not be given to the first turn at any point in the old
code.

**Expected behavior**
When a user message arrives for a task whose current run has not started
its first provider turn, the run does not stop. The message goes into
that run's first turn. A run that is in a turn is interrupted as before.

**Steps to reproduce**
1. Assign a task to a Paperclip Runner agent with a sandbox environment,
so preparation takes a long time.
2. While the run prepares, post a second comment on the task.
3. Click Interrupt on the queued message before the run starts its
provider turn.
4. See that the run is cancelled and a new run prepares again.

**Paperclip version or commit**
Staging revision b43cccbb0.

## What Changed

- `server/src/services/native-runtime/first-turn-queued-input.ts` (new):
`claimFirstTurnQueuedInput` seals a run's first-turn input in one
transaction. It locks the task, the deferred wakes, and the run in the
same order as the queue routes. It takes the responsible user's plain
queued comments in queue order (not interaction answers, interrupt
intents, independent continuations, conversations, or other users'
messages). It renders them with source-trust sanitizing and secret
redaction, then writes a `firstTurnQueuedInput` receipt
(`paperclip.first-turn-queued-input.v1`) into the run context. A second
call for the same run returns the same messages.
`runAcceptsFirstTurnQueuedInput` is true for a running native (or
preparing native) run that has no receipt and no accepted provider turn.
- `packages/paperclip-runner/src/native-session-runtime.ts`: new
`getFirstTurnQueuedInput` option. The runner calls it immediately before
the first `session.startTurn` and appends the result to `task.prompt`
(or to `events` for a resumed-session continuation envelope). It is not
called for the disposition-only recovery turn.
- `native-session-executor.ts`: passes the claim to the runner. A failed
seal commits nothing, so the executor logs it and starts the turn
without the messages. The messages stay queued.
- `queued-comment-delivery.ts`: a message in a run's receipt counts as
delivered only after the provider accepted that run's turn
(`turn.accepted` or the provider's `turn.started`), and only if the
message was not edited after the seal. Release promotion, Send, and
later first turns use this rule. `explicit-native-continuation.ts` uses
the same rule for stopped native runs.
- `routes/issues.ts` Interrupt: when the target run has not sealed its
first turn and has no accepted turn, the route does not record interrupt
intent and does not cancel the run. The response has
`executionWait.reason = "first_turn_pending"`. Other runs are
interrupted as before.
- Queue views (`routes/issues.ts` and the wake-queue adapter) hide
messages that the active run took into its first turn. Queue edits keep
those ids on the wake, so a run that fails before acceptance still
leaves them queued.
- `run-state.ts`: the issue-control run projection includes the receipt,
so the queue poll can hide taken messages.
- `doc/execution-semantics.md`: new section "Queued messages before a
native run's first turn".

## Verification

- Red before the fix, green after:
- `pnpm --filter @paperclipai/server exec vitest run
src/__tests__/issue-queued-comments-routes.test.ts -t "first
turn|already started"`: Interrupt on a preparing run and on a claimed
native run before `turn.start` does not stop the run, and the first turn
gets both messages in order. Before the fix, both cases cancelled the
run.
- `pnpm --filter @paperclipai/paperclip-runner exec vitest run
src/native-session-runtime.test.ts -t "first-turn queued input"`: the
claim runs immediately before `turn.start`, and the first turn's prompt
ends with the queued messages. Before the fix, the claim was never
called.
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/first-turn-queued-input.test.ts`: a run that
fails before `turn.start` leaves the message queued. The next run
delivers it, and after that no later run or copied retry context takes
it again. Before the delivery rule, the "exactly once" and "accepted
then failed" cases failed.
- Safety cases that pass before and after: Interrupt still cancels a run
whose provider accepted a turn (including a run saved by the previous
release with no receipt) and a run whose first-turn input is already
sealed. The existing "explicit interrupt fallback" cases for preparing
and native runs without a responsible user still pass.
- Speed: the seal is one short transaction before `turn.start`. In the
embedded-Postgres test it took 5.5 ms. It removes a full second
preparation (54 s on staging) from the message-to-first-token time.
- Related suites: `src/modules/wake-queue`,
`explicit-native-continuation`, `issue-queued-comment-queue`,
`issue-queued-comments-routes`, `queued-comment-consumption`,
`heartbeat/run-state` (888 tests), `native-session-executor.test.ts`
(1306 tests), and the full `native-session-runtime.test.ts` (238 tests)
pass. `pnpm --filter @paperclipai/server typecheck` passes.
- Not done: no live run against a real provider or sandbox. Staging was
read only.

## Risks

- Previous-release state: a deferred wake saved by the previous release
has the same shape and is taken normally. A run saved by the previous
release has no receipt. If it is resumed before its first turn, it seals
on the new release. If its provider already accepted a turn, Interrupt
stops it as before. A run saved by the previous release while it was in
`turn.start` but before acceptance counts as unsealed, so in that short
window an Interrupt does not stop it and the message is delivered at the
turn boundary.
- Rollback: if the server rolls back after a run sealed and its turn was
accepted, the old release does not read the receipt and can deliver
those messages a second time.
- A first turn that does not use the fresh-turn path (restart
continuation, disposition-only recovery) does not call the claim. Its
messages are delivered at the turn boundary as before.
- A message that arrives after the seal waits for the turn boundary.
Interrupt stops the run as before.
- No schema migration. The receipt is a key in the run context.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, test execution, read-only
browser).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.5
2026-11-08 22:44:18 -06:00
Devin FoleyandClaude Opus 5.5 32b3648758 fix(storage): back off and reduce parallel parts when S3 asks to slow down (#20165)
A workspace snapshot upload sends up to eight file-backed parts at once. The SDK never retries a part because its body is a stream, so a part that received 503 SlowDown had only the uploader's three fixed-delay attempts (0.5 s and 1 s). Under a request-rate limit the save failed and the run failed in setup.

A rate-limited part now retries with exponential full-jitter backoff (up to eight attempts, at most 20 s per wait), and the upload halves the parts it admits, once per round of throttled requests. Other part failures keep the existing three attempts. Tests cover a real-client SlowDown recovery, the single halving, the attempt bound, cancellation during a wait, and the backoff schedule through its ceiling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:42:12 -08:00
DottaandClaude Opus 5.5 098f7d370b fix(workspaces): release an unchanged remote writer without a snapshot and wake its waiters (#20171)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A remote (sandbox) task workspace has one writer run at a time. The
writer holds the workspace until its files are saved as a snapshot
> - A run that a user comment interrupts right after its claim did no
work. No provider turn started and no command ran in the sandbox
> - Paperclip still kept its sandbox, then took a full snapshot of the
unchanged workspace. On a 2 GB repository this took 3 minutes
> - The next runs failed setup with `workspace_busy` ("ended, but its
workspace save has not settled"). The retry then waited for the
one-minute settling re-check after the save. The user's message waited
more than 4 minutes
> - This pull request releases such a writer at once, at the saved
version it started from, when durable evidence proves that it changed
nothing. It also wakes parked `workspace_busy` waiters when a hold
settles
> - The benefit is that an interrupted run no longer delays the next
message by a full workspace save

## Linked Issues or Issue Description

**What happened?**
A remote-workspace run was claimed (`native.coordinator.claim`). One
second later a board comment interrupted it (`operator_interrupted`). It
made no provider turn and no tool calls. Cancelled-workspace
preservation still retained its sandbox, and workspace recovery took a
full `remote_workspace_snapshot` of the unchanged workspace (3 minutes
for a 2 GB repository). Two follow-up runs were cancelled with
`workspace_busy`. Their retry was due during the save, and it parked for
the one-minute settling re-check after the save finished.

**Expected behavior**
A writer that provably changed nothing releases the workspace at once
and keeps its existing `snapshotId`. The next run is admitted without
`workspace_busy`. When a save does run, waiters start as soon as it
commits.

**Steps to reproduce**
1. Run a task on a remote sandbox workspace that already has a saved
version.
2. Interrupt the run with a board comment right after the claim, before
the provider turn.
3. See the full snapshot operation on the interrupted run and
`workspace_busy` on the follow-up runs.

**Paperclip version or commit**
Staging served revision b43cccbb0. Reproduced on `master` by the new
regression tests.

## What Changed

- `prepareRemoteWorkspace` records `remoteWorkspaceUnchangedBaseline`
(epoch, writer run, saved version) when preparation started from a clean
saved version and added nothing to it: no recovered attachments, no
repository outside the saved version and no workspace migration.
- New `proveRemoteWorkspaceUnchanged`
(`remote-workspace-unchanged-proof.ts`). It uses only durable evidence.
The writer run must have ended without a result, and its owner fence
must be clear. It must have no process fields, no execution, provider or
tool events (the shared `executionStartedEventPredicate`), no native
result, no runner checkpoint rows (the controller writes one before
every runner command), and no provider turn in a checkpoint that names
this run. It must have no workspace operation other than a restore or a
`workspace_finalize`/`workspace_recovery` receipt, so provisioning and
saves count as writes. Every task repository must be in the saved
version. Any missing or ambiguous evidence refuses.
- New `releaseUnchangedRemoteWorkspaceWriter`. It sets the authority to
clean with no writer, keeps `snapshotId`, marks the run reference
`saved`, logs `workspace.snapshot_skipped_unchanged` and sends the same
notifications as a final save. It never waits on a writer run that
another transaction holds (`skip locked`).
- The release applies in three places: the next run's
`reserveRemoteWorkspace`, cancelled/failed-workspace preservation
(`preserveNativeFailedWorkspace`), and `saveRemoteWorkspace` with the
final boundary. The final save returns the same version on a repeated
call, so a recovery that raced the release finishes without a capture.
- The final-save commit and the unchanged release call
`wakeWorkspaceBusyWaitersInTx`. A parked waiter whose holder was that
writer is due at once, not at the one-minute settling re-check.
- `doc/execution-semantics.md` describes the release and the wake.

## Verification

- `pnpm --filter @paperclipai/server exec vitest run
src/services/remote-workspace-unchanged-release.test.ts` (13 tests,
embedded Postgres, real archive capture):
- Staging shape (claim, then interrupt with no provider work): the next
reservation is admitted. No snapshot operation is recorded. The run
reference is `saved` at the old `snapshotId`. Red on `master`: "The
workspace still has an unfinished writer or save operation."
- A later final save, behind the `workspace_finalize` receipt that
production finalizers publish first, returns the same version with no
capture, a repeat returns it again, and a parked waiter becomes due now.
Red on `master`: a new snapshot is captured.
- Negatives that still do a full save: `turn.started`, `provider.event`,
`tool.execution.started`, `native.process_start_requested`, a
provisioning command that wrote into the workspace, a recorded process,
a live owner fence, and a writer that is still running.
- Upgrade safety: no baseline (a writer prepared by the previous
release), a baseline without its saved version, and a baseline with
another schema each keep the full save.
  - A full final save wakes the parked waiter. Red on `master`.
- `native-cancelled-workspace-preservation.test.ts`: a new case shows
that a cancelled writer with a baseline and no work is released, with no
retained lease and no recovery action. Red on `master` (the
restore-recovery hold was recorded). The other 44 cases pass unchanged.
- I also ran the related remote-workspace, workspace-recovery, native
preservation and run-dispatch suites locally.

## Risks

- A false "unchanged" verdict would drop unsaved files. The proof
requires a positive baseline from preparation and refuses on any write
record. Only `.paperclip-runtime` (excluded from snapshots) is written
by runner staging before a process starts. A deploy that lands mid-run
keeps the full save, because the proof reads only durable rows.
- Previous-release state: a writer prepared without the baseline (or
with an unknown or partial one) keeps today's full save. An older server
ignores the new metadata key.
- Waiters now start sooner after a final save. Arrival order among
waiters is unchanged (the park check still admits the earliest first).
- Other ways a hold can settle (for example a stop receipt that turns an
unused reservation free) still rely on the one-minute re-check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used in Claude Code
with tool use (shell, file edits and a read-only browser for staging
evidence).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:37:52 -06:00
DottaandClaude Opus 5.5 8e5959c759 fix(agents): keep pause board-only and resume grant-gated on the PATCH status route (#20164)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board governs the agent lifecycle: pause, resume and terminate
> - The V1 permission matrix says agents may never pause an agent, and
may resume one only with a direct `agents:configure` change grant
> - `POST /api/agents/:id/pause` and `POST /api/agents/:id/resume`
enforce this, but `PATCH /api/agents/:id` with a `status` ran the same
lifecycle change after only the agent-configuration check
> - The `agents.pause-agent` chat eval caught it: the CEO agent paused
Morgan itself through `PATCH`, although the case (and the spec) expect a
board user to do it
> - This pull request applies the dedicated routes' checks to `PATCH`
status changes, as #19955 did for termination
> - The benefit is one lifecycle authority, with no second route around
it

## Linked Issues or Issue Description

Refs #19955

**What happened**

An agent that may configure another agent could pause it with `PATCH
/api/agents/{id}` `{ "status": "paused" }`, although `POST
/api/agents/{id}/pause` is board-only. The same `PATCH` with `{
"status": "idle" }` resumed an agent without the change-grant decision
and without the invalid-org-chain guard that `POST /resume` applies. In
a chat eval run, the CEO agent paused Morgan this way and reported
"Morgan is paused".

**Expected behavior**

Pause is board-only on every route (`doc/SPEC-implementation.md` section
9.3: "Pause/resume agent ... pause: no; resume: direct
`agents:configure` grant only"; "Pause, clear-error, terminate,
approval, and key-management routes remain board-only"). An agent gets
the standard final "Board access required" denial with a link to the
agent's profile. Resume through `PATCH` needs the same grant and
org-chain checks as `POST /resume`.

**Steps to reproduce**

1. As an agent with agent-configuration permission over another agent,
call `POST /api/agents/{id}/pause`. The result is 403.
2. Call `PATCH /api/agents/{id}` with `{ "status": "paused" }`. Before
this change the result is 200 and the agent is paused.

## What Changed

- `server/src/routes/agents.ts` (`PATCH /agents/:id`): `status:
"paused"` now calls `assertBoard`, the same check as `POST /pause`.
Agents get the structured `board_access_required` denial that says the
denial is final and links `/<prefix>/agents/<id>`.
- `status: "idle"` now calls `assertCanResumeAgent` (the
`agent_config:update` decision with `requiresChangeGrant`) and returns
409 with the repair guidance when the org chain is invalid, the same as
`POST /resume`.
- `server/src/__tests__/agent-permissions-routes.test.ts`: pause is
denied through both routes for an agent allowed to configure the target,
with no lifecycle change, run cancel or activity. A board user can still
pause through `PATCH`. `PATCH` idle applies the change-grant decision,
allows a granted agent, and keeps the org-chain guard.

## Verification

- `cd server && pnpm exec vitest run
src/__tests__/agent-permissions-routes.test.ts
src/__tests__/agent-cross-tenant-authz-routes.test.ts
src/__tests__/board-access-denial.test.ts`: 147 passed.
- Without the route change, three new tests fail (PATCH pause by an
agent, PATCH idle without the change grant, PATCH idle with an invalid
org chain).
- Policy evidence: `doc/SPEC-implementation.md` section 9.3 and the
`agents.pause-agent` scenario ("Pausing an agent requires board
control").

## Risks

- Non-board automation that paused agents through `PATCH` status now
gets 403. That is the intended governance boundary. Board users are not
affected.
- Agents that resumed agents through `PATCH` status now need the same
change grant as `POST /resume`. Standard agents get `agents:configure`
on creation, so ordinary behavior does not change.
- `PATCH` idle now returns 409 for an agent with an invalid org chain,
the same as `POST /resume`.

## Model Used

Anthropic Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code, with
tool use (shell, file editing).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:01:11 -06:00
DottaandClaude Opus 5.5 f89813d005 fix(api-catalog): describe the real backup and restore paths for agents (#20159)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Instance admins and operators own database backups; a chat agent
cannot create one
> - When a person asks an agent to back up the instance and check a
restore, the agent must say who can do it and how
> - The agent reads the API catalog to find the operator path, but the
backup entry had only the summary "Trigger a database backup"
> - With no real path in the catalog, the agent told the person to use
an "Instance settings > Backups" page that does not exist
> - This pull request describes the real paths in that catalog entry:
the endpoint, the CLI, the retention setting, the result file, and how
to check a restore safely
> - The benefit is that the agent sends the person to paths that exist
and keeps a restore check off the live database

## Linked Issues or Issue Description

**What happened**

A person asked the CEO chat agent: "Back up this test instance and check
that we can restore it." The agent got 403 on `POST
/api/instance/database-backups`, which is correct. It searched the
catalog for backup and restore operations and found only that endpoint
with no description. Its reply then said: "Create a backup in Instance
settings → Backups". The web UI has no such page or control.

**Expected behavior**

The agent says that an instance admin or operator must do this. It names
real paths: the endpoint for an instance admin, `paperclipai db:backup`
on the server host, and the Backup retention setting for automatic
backups. For the restore check, it says to restore into a separate empty
database, not over the live one.

**Steps to reproduce**

1. Start a local instance and open a chat with the CEO agent.
2. Ask: "Back up this test instance and check that we can restore it."
3. Read the reply: it names a backup page in Instance settings. Look for
it in the UI: there is none.

## What Changed

- `server/src/routes/openapi.ts`: add a description to `POST
/api/instance/database-backups`. It says that only instance admins can
call it (agents get 403, cloud-managed instances run backups for you),
that automatic backups run about hourly with retention set on the
company-prefixed /company/settings page, that the UI has no create or
restore control, that an operator can run `paperclipai db:backup`, what
the result file is, and that a restore check uses psql into a separate
empty database, never over the live one.
- `server/src/routes/openapi.ts`: register the success status as `201`,
which is what the route returns, instead of `200`. The published
document already showed `201` through the created-operations fixup, so
this makes the source match it.
- `server/src/__tests__/database-backup-catalog-guidance.test.ts`: new
regression test. It runs the agent's catalog searches from the failed
run and checks the description. It also checks that the published
document shows `201` and not `200` for a successful backup.

## Verification

- `pnpm exec vitest run
src/__tests__/database-backup-catalog-guidance.test.ts
src/__tests__/openapi-routes.test.ts` (from `server/`): 40 tests pass.
Without the change, the 2 search tests fail. The status test guards the
published document, which the fixup already kept correct.
- `pnpm exec vitest run
src/__tests__/instance-database-backups-routes.test.ts`: 7 tests pass.
The server typecheck passes.
- Chat eval `instance-operations.backup-and-restore` (runner-opencode),
with this commit cherry-picked onto the eval branch: before the change,
the reply named an "Instance settings → Backups" page that does not
exist. After the change (GitHub Actions run 42552777052), the eval
passed. The first reply said the agent got 403, named `POST
/api/instance/database-backups` for an instance admin and `paperclipai
db:backup --json` for an operator, linked Backup retention on the
company settings page, and said to restore with psql into a separate,
empty database, never over the live one. The second reply reported the
operator's restore receipt and its limits correctly.

## Risks

- Low risk. The change adds catalog text and makes the registered
success status match the published 201. It does not change any API or
permission behavior.
- If a backup page or restore command is added later, this text must
change. The regression test names these facts, so such a change fails
the test and shows the text to update.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 22:01:06 -06:00
DottaandClaude Opus 5.5 3f12046576 fix(tool-access): an unscoped tools:use grant does not turn on actions a connection left off (#20160)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People connect apps and choose which actions of each app agents may
call; the choice is saved as the connection's tool profile
> - Every new standard agent also gets an unscoped `tools:use` grant,
which means "this agent may use tools"
> - The access policy checked explicit grants before profiles, so this
default grant allowed every action of every installed connection,
including actions the person left off
> - The agent's effective profile showed the action off, but the gateway
and the test-as-agent summary allowed it
> - This pull request makes an unscoped grant stop at a connection
profile: when a profile governs the tool's connection, only a grant
scoped to the action, connection or application allows an action the
profile leaves off
> - The benefit is that the actions a person turns off for a connection
stay off for new agents

## Linked Issues or Issue Description

**What happened**

A person connected an app with one read action and one destructive write
action. When they finished the connection, they enabled only the read
action. They installed the connection for a new agent, Morgan. Morgan's
effective profile (`GET
/api/companies/{companyId}/tools/profiles/effective/agents/{agentId}`)
allowed only the read action. But `GET
/api/tool-connections/{connectionId}/test-agents/{agentId}/access`
reported both actions as `allowed` with reason `allow_explicit_grant`,
and a chat agent told the person that Morgan could also run the
destructive write. The same policy decision admits gateway calls, so
Morgan could call the write.

**Expected behavior**

An action that the connection's profile leaves off is off for the agent.
The access summary, the effective profile and the gateway agree. A grant
that a person scopes to that action, connection or application still
allows it.

**Steps to reproduce**

1. Connect an MCP app that has a read tool and a destructive tool. In
the finish step, enable only the read tool.
2. Create a new agent through the API (it gets the default unscoped
`tools:use` grant) and install the connection for it.
3. Call `GET
/api/tool-connections/{connectionId}/test-agents/{agentId}/access`: the
destructive tool shows `allowed`, reason `allow_explicit_grant`.

## What Changed

- `server/src/services/tool-access-policy.ts`: for an agent, a tool's
connection is governed by a connection profile when the agent's
effective profiles include the connection's own access profile (the app
wizard's `app:<connectionId>` profile or the agent's connection-intent
profile, even with no entries), or any entry that names this connection
or selects this action (including application, tool-name and risk-level
selectors). For a governed connection, `explicitGrant` counts only
grants whose scope names an action, connection or application (`allow`,
`toolName`, `catalogEntryId`, `connectionId`, `applicationId`,
`applicationKey`). A grant scoped only by context (for example an agent
or project) still covers every tool, so it does not override the
profile. Board users keep their operator grant.
- `server/src/services/tool-profile-binding-precedence.ts`: export the
existing connection-profile identity check as
`isConnectionAccessProfile`, so the policy and binding precedence use
one definition.
- `server/src/__tests__/generic-mcp-connection.test.ts`: regression
tests through the gateway's access summary and the direct policy check:
- only the read action enabled: the write stays off with the unscoped
grant, and a grant scoped to the write allows it
- every action turned off (both lists empty): both stay off; a
context-only scope stays off; a connection-scoped or application-scoped
grant allows them
- an agent the connection's profile does not cover keeps the unscoped
grant
- a general profile with an application-selector exclude (no connection
id) is not overridden by the unscoped grant
- with company-wide access, the direct policy check denies the off
action for an agent and still allows it for a board user with the
unscoped grant

## Verification

- New tests: all pass with the change. On `master`, 4 of 5 fail. On the
first revision of this PR, the all-off, application-exclude and
board-user tests fail.
- Regression suites, all pass: `generic-mcp-connection`,
`tool-access-policy-service`, `tool-profile-matching`,
`tool-profile-entry-order`, `connection-management.integration` (238
tests), and `tool-access-service`, `tool-gateway-service`,
`tool-gateway`, `tool-profile-review-service` (799 tests).
- Every enforcement path uses the one `decide` function in
`tool-access-policy.ts`, and `explicitGrant` is called only there. That
covers the gateway tool listing and calls, the access summary, delegated
and Test tab calls, the connection-intent access check, the GitHub
review bot tool, and the example smoke check.
- Chat eval `app-permissions.test-as-agent` (runner-opencode). Before
the change (GitHub Actions run 42550085273), the access summary for
Morgan reported the write as `allowed` by `allow_explicit_grant`. With
the first commit cherry-picked onto the eval branch, GitHub Actions run
42552819589 passed. The summary reported `launch_read` allowed
(`allow_profile`) and `launch_update` off (`deny_default`), which
matches Morgan's effective profile, and the agent's reply said the same.

## Risks

- Medium risk. This changes an access decision. An agent that used a
connection action only through its default unscoped grant, while the
connection's profile or an explicit exclude left that action off, loses
that action. That is the intended fix. An operator who relied on it must
enable the action in the connection or give the agent a grant that names
the action, connection or application.
- A grant scoped only by context (for example `{ projectId }`) no longer
overrides a connection profile. Risk-level and provider-type scopes are
not treated as naming an action.
- Tools that no connection profile governs (for example Paperclip's own
tools), agents without a connection profile, and board users keep the
current behavior.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Related: #17884 (scoped `tools:use` grants at hire time).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.4
2026-11-08 21:58:29 -06:00
DottaandClaude Opus 5.5 28e070f8dc feat(skills): show a line diff for a pending tracked-source update (#20155)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Skills can come from a tracked GitHub source, and an update moves
the installed copy to a newer upstream revision
> - Before an update, a person asks the agent "what changed in the new
version?"
> - The product has no diff for a pending update. The agent previews two
full SKILL.md snapshots and compares them by eye
> - In two eval runs the agent reversed the same changed line. In a
third run it fetched the files with curl and ran `diff` itself, and the
turn took too long
> - This pull request adds a read-only endpoint that returns the line
diff between the installed and the latest tracked revision
> - The benefit is that review answers come from an exact diff, and they
come faster

## Linked Issues or Issue Description

No public issue exists. Feature description:

**What happened**
An agent asked to show what changed in a pending skill update had only
`POST /skill-sources/preview`, which returns one file at one commit. It
previewed both commits and compared about 9 KB of text by eye. It
reported one changed line in the wrong direction ("every new build" ->
"every build or release" instead of the reverse), and later retracted a
real change.

**Expected behavior**
The agent can request the exact changes of a pending tracked-source
update and report them with the correct direction.

**Steps to reproduce**
1. Install a skill from a GitHub source at an old commit and set the
tracking reference to a newer commit (automatic updates off).
2. Ask the chat agent: "Show me what changed in the new accessibility
skill version before we update it."
3. See that the agent can only preview full files at each commit and
must compare them itself.

## What Changed

- Add `GET /api/companies/{companyId}/skills/{skillId}/update-diff`. For
a skill from a tracked GitHub source, it compares the installed source
revision (`currentRef`) with the latest tracked revision (`latestRef`).
For each changed package file it returns `change` (added, removed,
modified, or `not_compared`), `binary`, `executableChanged`, and a
unified line diff (`-` installed, `+` latest), or `diffUnavailable`
(`binary`, `too_large`, `omitted`, `budget`) that says why there is no
diff. The endpoint is read-only. The `skillId` parameter handler checks
company access and skill visibility, the same checks other skill reads
use.
- `hasUpdate` is false and `files` is empty when the installed revision
is already the latest one. That case reads only the latest commit and
scans nothing.
- Return `supported: false` with a reason for: a skill without a tracked
source, a disconnected source, an unknown installed revision, and a
repository package that a scan could include only in part (a subset, or
skill folders only). In the last case, files that a scan limit left out
would otherwise look removed.
- A path the scan left out of a revision (too large, a symlink, or
rejected by the audit) is listed as `not_compared` instead of added or
removed.
- One response shares a work budget (24M LCS cells), a diff-text budget
(1M characters) and a file-count budget (1,000 files), and sets
`truncated` when it reaches one. The GitHub read and both scans stop
when the caller disconnects.
- Add `server/src/services/skill-update-diff.ts` with a bounded line
diff (shared head and tail trimmed, LCS size capped). An absent side has
no lines, a final newline is not a line, and a missing final newline
shows the unified-diff marker. A replaced line reads `-old` then `+new`.
- Add the shared types `SkillSourceUpdateDiff` and
`SkillSourceUpdateDiffFile`.
- Document the endpoint in the API catalog (`openapi.ts`) so agents find
it with `search_api`.

## Verification

- `cd server && npx vitest run src/__tests__/skill-update-diff.test.ts`
(10 tests):
  - direction of `-`/`+` lines, and separate hunks with bounded context
  - exact diffs for added and removed files, and final-newline handling
  - a change to only the executable bit, and paths left out of a scan
  - the response budgets
- the snapshot comparison for the selected skill path, with the abort
signal passed to both scans
  - a commit mismatch
  - refusal of partial repository scans
- `cd server && npx vitest run
src/__tests__/skill-update-diff-routes.test.ts` (6 tests):
  - the diff response
  - no pending update (no scan)
- a caller from another company (board user and agent) gets 403 before
any lookup
  - a skill the caller cannot read gets 404 before any source read
  - every unsupported case
  - a client disconnect aborts the scans
- `company-skills-routes.test.ts` and `openapi-routes.test.ts` pass (119
tests across the four files).
- `npx tsc --noEmit -p server`: no errors from this change.
- End to end: the `skills.review-update` chat eval exercises this path.
Its grader checks the quoted hunks against the real pinned diff.

## Risks

- Low risk. The endpoint is new and read-only. Existing routes do not
change.
- When an update is pending, each call scans the source twice (installed
and latest commit). It uses the same GitHub reader, scan limits and scan
slots as preview and refresh. The diff work and response size are capped
per response.
- `/update-status` and this route both read the latest commit of the
tracking ref. They are kept separate because `/update-status` compares
with the source's last scan, while this route compares with the
installed revision.
- No UI uses the endpoint yet.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:55:52 -06:00
DottaandClaude Opus 5.5 4afdf0493d fix(api-catalog): name the Connections page for company model service settings (#20152)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A company can turn on model services such as Fast response, Decision
model and Prompt Injection Prevention
> - Only people who manage connections can change these services, so a
chat agent must send the person to the right page
> - The agent finds these operations through the API catalog search, but
their catalog entries named no page
> - The agent searched for the settings page, found nothing, and linked
the general /company/settings page, which does not have the control
> - This pull request adds a description to these catalog entries that
names the section and the company-prefixed /company/settings/connections
page
> - The benefit is that the agent links the page that actually holds the
control

## Linked Issues or Issue Description

**What happened**

A person asked the CEO chat agent: "Help me enable the company's Fast
response feature so chat acknowledges my requests faster." The agent
correctly got 403 on the fast-response API. It then searched the API
catalog for "company settings page fast responses web UI" and found no
page. It linked the general company settings page
(`/<prefix>/company/settings`) and told the person to find Fast response
there. That page has no Fast response section. The section is on
/company/settings/connections.

**Expected behavior**

The agent links /<prefix>/company/settings/connections and tells the
person to use the Fast response section there. If the section is hidden,
the agent can say that an instance admin turns the experiment on.

**Steps to reproduce**

1. Turn on the fast responses experiment. Create a shared OpenRouter API
connection.
2. Open a chat with the CEO agent and ask it to enable Fast response.
3. Read the reply: it links /<prefix>/company/settings. Open that link:
the page has no Fast response control.

## What Changed

- `server/src/routes/openapi.ts`: add one shared description to the
fast-response, decision-model and prompt-injection-prevention catalog
entries. It names the section, the company-prefixed
/company/settings/connections page, the Markdown link form, and that
agent credentials cannot change or test the service. For Fast response,
it also names the instance experiment that hides the section.
-
`server/src/__tests__/company-model-service-settings-guidance.test.ts`:
new regression test. It runs the agent's catalog search (including the
query from the failed run) and checks that the results name the
Connections page and section. It also requires every page the
descriptions name to be a route that `ui/src/App.tsx` mounts, so
renaming a settings route fails CI.

## Verification

- `pnpm exec vitest run
src/__tests__/company-model-service-settings-guidance.test.ts` (from
`server/`): 6 tests pass. Without the catalog change, the 5 search tests
fail. If the Connections route in `ui/src/App.tsx` is renamed, the route
test fails.
- `pnpm exec vitest run src/__tests__/openapi-routes.test.ts`: 37 tests
pass.
- Chat eval `companies.fast-responses` (runner-opencode), with this
commit cherry-picked onto the eval branch: before the change, the reply
linked the general settings page, which has no Fast response control.
After the change (GitHub Actions run 42549422681), the eval passed. The
reply linked the Experimental settings page and the Connections page and
said to use the Fast response section there. The scripted person clicked
the reply's links and found the Fast response control on the Connections
page, then saved and tested it.

## Risks

- Low risk. The change adds catalog text only. It does not change any
API or permission behavior.
- If the settings page moves, these descriptions become wrong. The
regression test compares every page the descriptions name with the
routes in `ui/src/App.tsx`, so a route change that does not also update
the description fails CI.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:55:48 -06:00
DottaandClaude Opus 5.5 f316562045 test(chaos): wait for new agents to be ready before the first assignment (#20158)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades each run's time to first activity
against the speed budgets
> - The performance session traced an 8–12 s "first claim" delay on
every wave's first cold runs. A brand-new agent spends about 8 s in
`lifecycleState: verifying` while its harness runs a real hello probe,
and queued runs start only when it becomes ready
> - The chaos waves create agents and assign the first tasks in the same
second, so those runs carry the one-time verification. Their cold
first-activity and boot looked over budget even though dispatch after
ready takes 40–60 ms
> - This pull request makes the waves wait for each agent to be ready
before the first assignment, records how long verification took, and
fails the wave if an agent never verifies
> - The benefit is that cold-start grades measure task startup, while
agent verification stays visible as its own number

## Linked Issues or Issue Description

Refs #20119

**What happened?**
In waves `w-20261109T021322-s9801` (G1) and the K4/K6 wave, every new
agent's first cold runs waited 8–12 s in `heartbeat.queue`. The
performance session reproduced it locally: the agent is in
`lifecycleState: verifying` while a hello probe runs, and its queued
runs start on `onReady`. `ttft-baseline.ts` already waits for `ready`
before launching tasks, so it never showed the delay.

**Expected behavior**
The chaos waves start their first task after the agent is ready, as the
baseline does, and they report the verification time.

## What Changed

- `seed.ts`: new `waitForAgentsReady` polls each new agent until
`lifecycleState` is `ready`, with a 4-minute cap.
- It records `seed_agent_ready` with `readyAfterMs` per agent on the
timeline.
- It throws, failing the wave, if an agent reports a `lifecycleError` or
never becomes ready.
- `seedChaosLab` runs it when `waitForReadyAgents` is set. `main.ts`
sets it for every wave. `ttft-baseline.ts` keeps its own readiness
logic, which skips unready adapters.
- The seed test's fake API answers agent reads as ready.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 205 passed. New tests cover
the wait, the verification-error failure and the timeout failure.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- Waves start about 8 s later, once per wave.
- A product regression that keeps agents in `verifying` now fails the
wave at seeding with a clear error, instead of showing as slow first
tasks.
- The one-time verification delay a user sees on a new agent's first
task is no longer in the cold grade. It stays in the timeline
(`seed_agent_ready`) for anyone tracking it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:32:11 -06:00
DottaandClaude Opus 5.5 43fe87795a fix(authz): make board-only denials final and actionable for agents (#20151)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People talk to their agents in Agent Chat, and the agents act
through the Paperclip API
> - Many API routes are board-only. The shared `assertBoard` check
returns 403 `"Board access required"` to an agent, with no other detail
> - The chat guidance also told the agent to "use permitted discovery to
identify the supported operator path" after such a denial
> - So the agent keeps searching the API, `call_api` and the shell for a
second route. On a slow model, this search uses the whole reply window
and the person gets no answer
> - Earlier single-route fixes (#19990, #20041, #20008) showed the
remedy: a denial that says it is final and names where the person acts.
Reply time fell from 10-24 minutes to about 2 minutes
> - This pull request applies that remedy once, in the shared board-only
check and in the chat guidance
> - The benefit is a prompt, correct handoff for every board-only action
that an agent cannot do

## Linked Issues or Issue Description

Refs #19990, #20008, #20041 (same remedy for single routes).

**What happened**
In about 30 Agent Chat evaluation cases on a slower model, the agent got
403 "Board access required" (or read `authorization.actor: "board"` in
the API catalog). Then it searched for another route until the attempt
deadline. Examples: archive a company, set up a managed agent profile,
add a company secret. In one recorded run, the agent got the 403 at
minute 1, then tried PATCH, environment variables, the CLI and UI paths
for more than 3 more minutes, and never replied.

**Expected behavior**
A board-only denial is final. The agent stops, tells the person that a
board user must do the action, links the page in the Paperclip web app
where they can do it, and ends its reply.

**Steps to reproduce**
1. Open Agent Chat with a CEO agent.
2. Ask it to archive the company (or to add a company secret).
3. The agent calls `POST /api/companies/{id}/archive` and gets `{
"error": "Board access required" }`.
4. The body says nothing about finality or where the person acts. The
agent searches for other routes and replies late or not at all.

## What Changed

- `server/src/routes/authz.ts`: `assertBoard` now throws
`boardAccessRequired(req)`. For an agent actor, the 403 body adds `code:
"board_access_required"` and a `remediation`. The remediation says the
action is board-only by design, the denial is final, and no other route
can do it. When the API path maps to a known board page, it links that
page with a `<prefix>` placeholder: company settings, company secrets,
budgets, members, tool gateways, connection settings (decision model,
prompt injection prevention), an approval, or the agent profile tab that
holds the control (API keys, permissions, budget, identity, harness
settings). Otherwise it says to link a page only when the route is
already known, and not to search for one. Board and anonymous actors get
the same body as before.
- `environments.ts`, `instance-settings.ts` and the issue force-release
route used their own copy of the check. They now call `assertBoard`, so
they return the same denial. Status and error text do not change.
- `server/src/middleware/error-handler.ts`: for this code, the handler
does not repeat `code` and `remediation` inside `details`. They are
already top-level, and the copy only made the agent's context longer.
- `AGENT_CHAT_DIRECTIVE`: the board-identity paragraph now says that a
board-only operation (catalog actor `board`, or a 403 "Board access
required") is final. The agent must stop searching, tell the person
which control in the web app to use, and link it when it knows the
route. The old instruction to "discover the operator path" is removed.
- Claude Managed Agent setup: the Paperclip Runner configuration doc and
the `POST /api/companies/{companyId}/managed-agent-profiles` catalog
entry now say that a board operator creates and qualifies the profile
(limited environment, retention acknowledgement) and that agents cannot.
Before, an agent asked for an "Anthropic managed agent" explored hiring
and adapter docs for its whole reply window.
- Tests: new `board-access-denial.test.ts` covers the agent body for
each mapped page, the agent profile tabs, the general fallback, the
unmapped fast-response route, the org-access guard, the unchanged
non-agent body, the guidance and the managed profile catalog entry. Two
exact-body tests now also expect the new code.

## Verification

- `cd server && npx vitest run src/__tests__/board-access-denial.test.ts
src/__tests__/costs-service.test.ts
src/__tests__/browser-use-route-scope.test.ts
src/__tests__/primary-agent-guidance.test.ts
src/__tests__/agent-cross-tenant-authz-routes.test.ts
src/__tests__/company-portability-routes.test.ts
src/__tests__/issue-comment-cancel-routes.test.ts
src/__tests__/error-handler.test.ts
src/__tests__/managed-agent-profile-routes-authz.test.ts
src/__tests__/companies-route-cross-company-authz.test.ts
src/__tests__/environment-routes.test.ts
src/__tests__/environment-instance-routes.test.ts`: all pass.
- Real chat journeys on the chat evaluation branch (runner-opencode,
`deepseek-v4.1-flash`). Before this change, each case below got no reply
before its deadline:
- `companies.archive-company`: passed. Reply 47 s after the request
(before: none in 270 s). The agent read the remediation and linked
`/COM/company/settings`.
- `companies.configure-decision-model`: passed. Reply 68 s after the
request (before: none in 389 s). It linked
`/COM/company/settings/connections`.
- `companies.prompt-injection-guard`: reply 85 s after the request
(before: none in 390 s), with the correct handoff and link. The case
still fails later, in the scripted settings step: the saved model is the
default, not the model the script typed. That is a separate defect.
- `navigation-preferences.sidebar-order`: reply 99 s after the request
(before: none in 150 s). The case still fails in its scripted sidebar
step (a locator was not found).
- `secrets.configure-company-secret`: reply 6.5 min after the request
(before: none in 7.5 min). The reply is correct, but a grader keyword
check reads "chat is not a safe place to send it" as a request for the
secret.
- `agents.managed-profile`: still no reply in its 150 s window. After
the catalog change, the agent found the board-only profile setup within
30 s, but then spent about 90 s deciding what "Anthropic managed agent"
means.

## Risks

- Low risk. The HTTP status and the `error` text stay the same for every
actor. Board users and anonymous callers get the same body as before.
- Agent actors now get two more top-level fields (`code`,
`remediation`). A client that compares the whole 403 body of an agent
call exactly will see a change. Two server tests did this and are
updated.
- The page map uses the API path. A wrong map entry gives a less useful
link but does not change authorization.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:28:52 -06:00
DottaandClaude Opus 5.5 fa7950121d fix(native): move an interrupted bootstrap's session aside on the run's own created sandbox lease (#20154)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When restart recovery proves a remote bootstrap reached no provider
work, it bootstraps the same run again in the same sandbox. Before
relaunching, it stops the interrupted runner (#20103)
> - The relaunch starts a fresh controller record, whose command ids
restart at `command_prp_00000001`. The sandbox still held the
interrupted runner's journal, which had already recorded that command id
with different data
> - The relaunched runnerd rejected it ("commandId was reused with
different command data") and exited while waiting for `run.prepare`.
Every same-run retry collided the same way until the retry budget ran
out and the task blocked
> - The proven interrupted-bootstrap session move (#20103) ran only for
a resumed lease. A same-run retry on a per-turn sandbox reuses the run's
own created lease, so nothing moved the interrupted session aside
> - This pull request applies that same proof-gated session move on the
run's own created lease
> - The benefit is that a same-run bootstrap retry starts its runner on
a clean journal, and the old journal is kept as evidence

## Linked Issues or Issue Description

Refs #20103

**What happened?**
Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master
8e3b64a257, image 34bc12bf):
- **t10 (codex M1):** the codex subtask's run `f8a8f483` went through
eleven same-run bootstrap retries between 01:56 and 02:07.
- Each relaunched runnerd logged `paperclip-runnerd: commandId was
reused with different command data` and exited about 12 s later while
waiting for `run.prepare`.
- The run then failed `native_session_retry_exhausted` ("Provider
startup failed on 3 attempts before any provider work began"). The
subtask blocked, and so did the parent.
- **t02:** the same "runnerd exited while waiting for run.prepare"
pattern appeared on retry run `2e13568f`.

**Expected behavior**
A same-run bootstrap retry moves the interrupted session aside, keeping
it as a `.bootstrap-restart-<time>-<uuid>` sibling, and claims a fresh
session root before relaunching. The relaunched runner then cannot see
the interrupted attempt's command ids. This already happened on a
resumed lease and now also happens on the run's own created lease.

**Steps to reproduce**
1. Let a remote bootstrap reach the runner (it journals
`command_prp_00000001`), then restart before any provider work.
2. Recovery bootstraps the same run again in the same sandbox. Before
this change, the relaunched runnerd sees the old journal, rejects the
reused command id and exits, and every retry repeats this.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`: the created-lease branch of remote
harness preparation now applies the resumed branch's proven restart. It
runs when all of these hold:
  - the session is `not_suspended` and belongs to this run;
  - there is no backup;
  - the claim is `bootstrap_incomplete`;
- the stale-runner stop proved the interrupted runner gone
(`hasRemoteBootstrapRestartProof`).

It then runs `moveInterruptedBootstrapSessionAside` and
`claimUntouchedRemoteSession`. If the move or the claim fails, it throws
`runner_harness_state_mismatch: created_not_suspended` and changes
nothing.
- An earlier revision of this PR archived only the `runner` directory.
Review showed that would leave a session the existing claim cannot
reopen on a resumed lease, so that revision was replaced.
- On a fresh command namespace: moving the session aside removes every
id the relaunched runner could see, so a separate namespace isn't
needed. It would also change the persisted command id format that saved
controller state depends on.
- `native-session-executor.test.ts`: the existing "bootstraps only an
untouched provider session" matrix runs the real move and claim scripts
against a temporary filesystem. It gains two created-lease cases:
- an interrupted bootstrap: the old session is kept as a sibling and a
fresh root is claimed;
  - a symlinked parent: refused, and nothing moves.

## Verification

- Red/green: both created-lease cases fail on master and pass with the
change. The resumed cases from #20103 are unchanged: 42 cases in the
matrix pass.
- `vitest run` over the restart-recovery and bootstrap test files and
`native-session-executor.test.ts`: 7 files, 1540 tests passed (14
skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.
- The collision itself happens in runnerd (the production log shows it
on every retry). The live check is a Daytona per-turn K2/K4 rerun.

## Risks

- Low risk. The move reuses the resumed path's proof gates, its symlink
checks and its whole-session move, which keeps the session and never
deletes it. The new branch acts only when every gate holds; otherwise
preparation continues exactly as before.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:06:26 -06:00
DottaandClaude Opus 5.5 510746c682 fix(native): restart a remote run whose controller never sent its runner a command (#20153)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A follow-up run on a remote sandbox inherits the previous run's
session checkpoint at admission, rebound to the new run's identity
> - A remote runner receives work only through the controller's Postgres
checkpoint, which the controller writes before every command. A run with
no checkpoint rows, no runner events and no recorded runner process
never sent its runner anything
> - Restart recovery still read the rebound admission checkpoint's
provider session id as provider evidence, and classified such a run
`reattach_remote_runner`. The executor then found the root still owned
by the previous run and failed with
`native_remote_recovery_lease_mismatch: stage=runner_verification
field=identity_run_id`
> - This pull request classifies that run as an incomplete bootstrap,
and lets the executor start it from the previous owner's settled root
> - The benefit is that a follow-up interrupted before its runner ever
got a command restarts on its own

## Linked Issues or Issue Description

Refs #20137

**What happened?**
Daytona per-turn restart-chaos M1 rerun w-20261109T013733-s9501 (master
8e3b64a257, image 34bc12bf), t02 (codex M1 parent):
1. Wake run `a867cb18` was admitted at 01:48:31 with a rebound
checkpoint inherited from run `dfeddafa`: identity of the new run,
`cursor: null`, no terminal turns, no active turn.
2. It recorded `native.process_start_requested` at 01:48:36, then a K4
hold froze the control plane.
3. At 01:53:11 recovery classified it `reattach_remote_runner`
(`hasCheckpoint: true`, `hasProviderEvidence: true`, from the rebound
checkpoint), although the run had no `native_runner_checkpoints` rows,
no runner-originated events and no recorded process identity.
4. The executor found the session root still owned by `dfeddafa` and
failed with `native_remote_recovery_lease_mismatch:
stage=runner_verification field=identity_run_id`. The task needed a
Retry.

**Expected behavior**
A remote run whose runner provably never received a command restarts its
bootstrap. The executor treats the previous owner's root as it would for
any follow-up start.

**Steps to reproduce**
1. Claim a remote sandbox follow-up that recorded
`native.process_start_requested` but has no controller checkpoint rows,
no runner events and no `native.process_identity_recorded`, with only
the admission-rebound session checkpoint.
2. Restart. Before this change, recovery claims
`reattach_remote_runner`, and the executor fails with `identity_run_id`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `native-restart-recovery.ts`: the new
`remoteRunnerNeverReceivedCommands` condition.
- It applies to a remote sandbox run (not a plugin task) with a prepared
remote workspace when all of these hold:
    - no durable controller record, and no staged attachment;
- only the admission-rebound checkpoint (or none), and no provider
process events;
- no runner-originated events, and no
`native.process_identity_recorded`;
    - no run process fields;
- the same claimed and dead previous-controller conditions as
`remoteRunnerNeverLaunched`.
- It yields `bootstrap_incomplete` with the reason
`remote_runner_never_received_commands`, and does not count the rebound
checkpoint as this run's checkpoint.
- `native-session-executor.ts`:
- a `bootstrap_incomplete` claim requires the incomplete-bootstrap proof
only when the root carries this run's own identity;
- a root still owned by the previous run takes the normal previous-owner
path, and may retire a finished previous owner, as any follow-up start
does.
- Policy change, called out for review: a claimed remote run that
recorded only `native.process_start_requested` was held as
`ambiguous_remote_recovery_evidence`. Under the conditions above it now
bootstraps again.
- The controller writes every command to the Postgres checkpoint before
sending it, so with no checkpoint rows no command reached any runner,
and no provider work can exist.
- A runner process that was spawned but never commanded is stopped
before the new bootstrap by the existing stale-runner stop.
- A recorded process identity, an adapter invocation, a runner event,
process fields or a provider session still hold the run.
- Tests:
- `native-runner-restart-recovery.integration.test.ts`: the
classification for a follow-up with an inherited checkpoint and for a
first run without one, plus a held case when a runner process identity
was recorded. The old held expectation for "a runner process start was
requested" moved here;
- `native-recovered-owner-followup.test.ts`: the executor admitting a
`bootstrap_incomplete` follow-up from the previous owner's settled root.

## Verification

- Red/green:
- On master, the classification test claims `reattach_remote_runner`, as
in production. With the change it claims `bootstrap_incomplete` with
`remote_runner_never_received_commands`.
  - A recorded runner process identity keeps the run off this path.
  - The executor test is admitted with the change. On master it fails.
- `vitest run` over the restart-recovery, bootstrap, recovered-owner and
failed-workspace-recovery test files and
`native-session-executor.test.ts`: 9 files, all passing after the policy
test moved; `native-runner-restart-recovery.integration.test.ts` alone:
144 passed (14 skipped).
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk:
- The new path requires positive proof that nothing reached a runner: no
controller record, no runner events and no recorded process. Any of
these keeps the existing classification.
- In the executor, a root that carries this run's identity still needs
the full bootstrap proof.
- A root that carries the previous run's identity goes through the same
previous-owner verification as any follow-up start.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 21:06:21 -06:00
DottaandClaude Opus 5.5 763adbea09 fix(execution): a pause's stop never blocks the task behind an incident with no control (#20143)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A budget hard stop pauses the agent, which stops its running work
> - In the restart-chaos H1 scenario, the stopped task went `blocked`
behind a board action that said "inspect and reconcile" and offered no
button, and it stayed there after the board resumed the budget
> - A pause is an intentional stop, not a failure, so it must not leave
the task with no way forward
> - This pull request treats a pause's stop as intentional: the task
keeps its status and no incident opens, so the normal Retry works
> - The benefit is that a budget or agent pause never strands a task
behind a control-less incident

## Linked Issues or Issue Description

Refs #20113, #20101

**What happened?**
Restart-chaos wave w-20261108T210118-s9802 (H1). At the budget pause,
task CHA-1 went `blocked` (21:02:27.870) with an active board
`active_run_watchdog` action, cause
`native_continuation_requires_reconciliation`, next action "Inspect the
original failure and reconcile the previous execution before
continuing." After raise_budget_and_resume it stayed blocked,
`canRetry=false`, with no Retry. The run was cancelled by the agent
lifecycle pause (`cancellation: { reason: "Agent lifecycle stop
requested", initiator: system, expected: true }`). The wake-queue
release (`recordNativeTerminalRecoveryIfNeeded`) blocks every cancelled
native run that is not an acknowledged user Stop.

**Expected behavior**
A pause's stop keeps the task's status and opens no incident; the task
can be continued.

**Steps to reproduce**
Pause an agent (budget hold or company budget pause) while its native
run is active. Before this change, the task is blocked behind a board
action with no control.

## What Changed

- New `server/src/services/governance-pause-stop.ts`:
`isGovernancePauseStop(run)` matches an expected system cancellation
whose reason is the budget pause stop, or the agent lifecycle stop with
a `pauseStop` stamp.
- `wake-queue/adapters/postgres.ts`
`recordNativeTerminalRecoveryIfNeeded`: returns early for such a run, so
the release keeps the task's status and opens no reconciliation
incident.
- `agent-lifecycle-driver.ts`: the `pausing` host step stamps
`resultJson.pauseStop = { holds, companyPauseReason }` on the agent's
active runs before stopping them (one update, company pause reason by
subquery). Termination and cleanup share the lifecycle stop reason but
write no stamp, so they still open the incident.

## Verification

- `wake-queue/adapters/postgres.test.ts`: new cases. A run stopped by
the lifecycle pause or by the budget pause keeps the task `in_progress`
with no incident (both red on master). Agent termination, and another
system stop (company archive), still open the incident.
- `agent-lifecycle/__tests__/lifecycle.test.ts`: "records what a pause
stopped each active run for". `driver-scope.test.ts` updated for the
stamp's single update.
- `wake-queue` (107), `lifecycle` (68+), `driver-scope`,
`companies-service`: 199 passed. Server typecheck (`typecheck-layers`,
low-memory): no errors.
- Not run: the H1 chaos rerun (after the follow-up continuation PR).

## Risks

- A native run that a pause stopped no longer opens a reconciliation
incident. Resuming it is the normal Retry, which continues the
conversation. Termination, archive and failures still open one.
- One extra update per pause attempt on the agent's active runs. It is
idempotent (skips runs that already have the stamp) and not on any turn
path.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test execution). Extended reasoning
enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:55:36 -06:00
DottaandClaude Opus 5.5 3eea8b87bd fix(native): settle a finished owner whose journal holds a definitively answered command (#20148)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a per-turn follow-up starts a new runner, it retires the
previous turn's finished owner. Retirement requires that owner's
controller journal to be settled: every command answered, then a run
terminal
> - A restart can cut off a continuation turn. The recovered runner
answers that `turn.start` with `failed` and the nested code
`execution_indeterminate` ("it will not execute twice"), and the run
continues on a later turn to an accepted result
> - Retirement counted only `completed` commands as answered. A finished
run whose journal held that answered `turn.start` never counted as
settled, so the follow-up failed with
`retirement_refused=journal_commands_unsettled(turn.start.failed)` and
needed a Retry
> - This pull request counts definitive `failed` and `rejected` answers
as settled, except for runner lifecycle commands. An unknown-outcome
`turn.start` counts as settled only once a later `turn.start` completed,
which shows the run moved past it. Every other unknown or pending
command stays unsettled
> - The benefit is that a follow-up after a restart-interrupted, then
finished, run continues on its own

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master
8e3b64a257), t07 (claude C1, K6):
- The previous run `9cb2c0b3` lost a continuation turn to a restart. The
runner answered that `turn.start` as `failed` with `result.result.code:
"execution_indeterminate"`.
- After the next restore it ran a new continuation to an accepted
result, and the run terminal was committed at 00:48:18.
- A hard kill then landed before the runner suspended, and recovery
finalized the result.
- Follow-up `372991c8` failed at 00:48:37 with
`runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(turn.start.failed)`.

**Expected behavior**
A command the runner definitively answered (`failed` or `rejected`)
leaves nothing outstanding, so it counts as settled. A `turn.start` with
an unknown outcome is never executed again by the runner, so once a
later `turn.start` completed, the run has moved past it and it also
counts as settled. The follow-up then retires the finished owner and
continues.

The following still count as unsettled:
- an unknown-outcome `turn.start` with no later completed turn;
- any other unknown-outcome command;
- a `pending` command;
- a failed runner lifecycle command, such as a failed `runner.suspend`
(the runner did not suspend).

**Steps to reproduce**
1. Finish a per-turn run whose controller journal holds a `turn.start`
answered `failed`, followed by a committed run terminal.
2. Start a follow-up. Before this change, retirement refuses with
`journal_commands_unsettled(turn.start.failed)`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `remote-warm-session-recovery.ts`: the new
`unsettledControllerCommands` decides which commands are settled.
`isSettledRemoteWarmControlState` uses it.
  - `completed` is settled.
- An unknown outcome (raw `indeterminate`, or `failed` with
`execution_indeterminate` at `result.code` or `result.result.code`) is
settled only for a `turn.start` with a later completed `turn.start`.
- Other `failed` or `rejected` answers are settled, except for
`runner.suspend`, `runner.drain` and `runner.shutdown`.
- `native-session-executor.ts`: the retirement refusal message lists
unsettled commands with the same rule.
- `native-recovered-owner-followup.test.ts`: a red/green test whose
journals are in real lifecycle order.
- Admitted: a `turn.start` answered `failed` or `rejected`; a superseded
unknown-outcome `turn.start` in the runner's stored nested shape; a
superseded raw `indeterminate`.
- Held: the same unknown-outcome answer with no later completed turn; a
failed `runner.suspend`.

## Verification

- Red/green: on master, the admitted cases fail with exactly the t07
error. With the change they are admitted. The held cases stay held. If
only `result.code` is checked, the unsuperseded case fails.
- `vitest run` of `native-recovered-owner-followup.test.ts`,
`remote-warm-session-recovery.test.ts` and
`native-session-executor.test.ts`: 3 files, 1403 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Low risk. Definitive answers to non-lifecycle commands, and
unknown-outcome turn starts that a completed later turn superseded,
become settled.
- The run terminal requirement, post-terminal live-work checks, and
every lifecycle and process proof are unchanged.
- The same rule now also applies to remote warm recovery's settled
check, where a definitively answered command is equally finished.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.3
2026-11-08 20:47:05 -06:00
DottaandClaude Opus 5.5 cedf335c98 fix(native): a failed terminal followed by an active restart continuation is not a permanent failure (#20149)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a restart loses a native run's active turn, the run records
that turn as failed with `provider_turn_lost_on_restore` and continues
on a same-conversation continuation turn
> - Restart recovery blocks a run as
`provider_checkpoint_permanently_failed` when its checkpoint carries a
failed run terminal and no pending restart interruption
> - While a continuation is running, the checkpoint keeps the lost
turn's failed run terminal and names the continuation as the active
turn. The pending-interruption check returns nothing once a turn is
active
> - So a second restart during the continuation read the run as
permanently failed, blocked replay, and needed the board
> - This pull request treats that failed terminal as the restart loss it
records whenever an active turn follows a restart-lost terminal turn
> - The benefit is that runs survive repeated restarts during their
continuations, including M1 parents and their subtasks

## Linked Issues or Issue Description

Refs #20148

**What happened?**
Local restart-chaos M1 K4/K6 rerun w-20261109T003126-s9701 (master
8e3b64a257), t05 (claude M1 parent):
1. Run `e1e37dab` lost turns to restarts at 00:41:32 and 00:47:53. Each
time it continued, and the third continuation `turn_lab_92b03dc7…` was
active.
2. At 00:48:29 the next restart blocked it:
`native_restart_recovery_blocked` /
`provider_checkpoint_permanently_failed`, with replay blocked.
3. Its saved checkpoint held `terminal.runTerminalState: "failed"` (from
the lost turn), `activeTurnId` set to the continuation, and a last
terminal turn fingerprinted `provider_turn_lost_on_restore`.

Its codex/ACPX subtask's run `f9b10d06` was blocked in the same pass
with the same shape, and its continuation had already proposed a
semantic result. Both stayed blocked. The parent's Retry was held on the
blocked child.

**Expected behavior**
A failed run terminal that records a restart loss, followed by an active
continuation turn, is not a permanent failure. Recovery claims
`resume_dead_runner`. The runtime then reconciles the active turn, and a
held result settles a lost turn as completed. A real provider failure
followed by a later turn still blocks.

**Steps to reproduce**
1. Checkpoint a run whose last terminal turn is a restart loss, whose
`terminal` is failed, and whose `activeTurnId` is a later continuation
(optionally with a semantic result).
2. Claim restart recovery. Before this change it blocks with
`provider_checkpoint_permanently_failed`.

**Paperclip version or commit**
8e3b64a257 (wave). Branch rebased on current master.

## What Changed

- `contracts/native-session-backend.ts` (runner): the new
`nativeRestartContinuationInFlight` is true when all of these hold:
  - the checkpoint has an active turn;
  - its last terminal turn is a different turn;
- that turn's fingerprint is a failed, result-less, recoverable
`provider_turn_lost_on_restore`.
- `native-session-runtime.ts`: recovery clears that stale failed
terminal for a continuation in flight, as it already does for a
checkpointed interruption. Before, it threw
`provider_checkpoint_failed_terminal` before it reconciled the active
turn.
- `native-restart-recovery.ts` (server): `checkpointFailed` now also
requires that no restart continuation be in flight. It uses the same
contract function through the vendor boundary.
- `native-session-runtime.test.ts`: an end-to-end recovery test. A
checkpoint with the stale failed terminal and an active continuation
resolves on the continuation's result without starting a new turn.
- `native-runner-restart-recovery.integration.test.ts`:
  - a codex case of the stale-terminal shape as saved in production;
  - an ACPX case with a held result;
- a held case where an active turn follows a real provider failure
(`model_not_found`).

## Verification

- Red/green: on master, the stale-terminal case is blocked with
`provider_checkpoint_permanently_failed`. With the change it claims
`resume_dead_runner`. The real-failure case stays blocked on both.
- Red/green (runtime): without the runtime change, the end-to-end test
throws `provider_checkpoint_failed_terminal`. With it, the run
completes.
- `vitest run` over the restart-recovery and failed-workspace-recovery
test files in `server/src/services/native-runtime`: 5 files, 305 tests
passed (14 skipped).
- `vitest run` over the runner restart, recovery, continuation and
terminal test files: 9 files, 385 tests passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`) and `pnpm
run typecheck:typescript` (`packages/paperclip-runner`): passed.

## Risks

- Low risk. The exception applies only when the recorded failure is
itself a recoverable restart loss and a later turn is in flight. Every
other recovery gate is unchanged: the provider-attempt budget,
controller takeover and process proofs.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:28:12 -06:00
DottaandClaude Opus 5.5 0502cc9fac test(chaos): retry a blocked M1 subtask and check the parent's stall wake (#20147)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness grades the M1 scenario: a parent task
delegates two subtasks across restarts
> - When a subtask's run failed recovery, the harness only clicked Retry
on the parent. The parent's Retry is correctly held behind the blocked
subtask (#20109), so the task waited until its deadline
> - A real board user would also click the subtask's Retry. #20098 also
added a wake for a parent whose subtask stalls, and the grade did not
check it
> - This pull request clicks the board Retry on a blocked subtask
(graded as a needed Retry) and checks that the parent got exactly one
stall wake per stall
> - The benefit is that M1 grades the full user path and proves the
stall wake from the product's own activity record

## Linked Issues or Issue Description

Refs #20098, #20109, #20094

**What happened?**
In wave `w-20261109T003126-s9701`, the M1 parent t05 rested blocked
behind a blocked codex subtask. The harness clicked Retry on the parent.
The product held that Retry ("waiting on a blocked task") as designed.
Nothing retried the subtask, and the task waited 38 minutes until the
wave deadline. The wave also could not show whether the parent got the
new `issue_blocker_stalled` wake.

**Expected behavior**
The harness acts like the board: it clicks Retry on a blocked subtask
that offers one, and it counts that as a needed Retry. The M1 oracle
checks the parent's activity for one `issue.blocker_stalled_wake` per
stalled subtask.

## What Changed

- `observer.ts`:
- For M1, the observer checks the parent's subtasks every 10 s while the
parent is open, independent of the parent's rest window.
- Each blocked subtask is recorded as a stall: the subtask plus its
latest recovery action, from `GET /api/issues/:id/recovery-actions`.
- When a blocked subtask's interruption offers Retry, the harness posts
the board Retry. It clicks once per subtask; a click that was not
accepted can be tried again on a later check.
- The parent's `issue.blocker_stalled_wake` activity entries are
collected as `stallWakes` through `GET /api/issues/:id/activity`.
- `grader.ts`: R3 fails with `retry_needed_on_subtask_<n> (<ids>)` when
a subtask needed a board Retry, unless the parent's own Retry already
failed R3.
- `workloads.ts`: the new M1 check `parent_woken_once_per_stall` is
added when a subtask stalled. Each (subtask, recovery action) stall must
have exactly one matching wake. A stall with an unknown action matches
any wake for that subtask.

## Verification

- `npx vitest run` in `tests/restart-chaos`: 203 passed. New tests
cover:
- an observer test: a blocked subtask gets one Retry click, and the
parent is still watched;
  - the R3 subtask-retry grade;
- the stall-wake check: woken once, not woken, woken twice, two stalls
of one subtask with only one woken, and no stall.
- G1, which also has subtasks, is unaffected: the existing governance
tests pass.
- `node scripts/typecheck-test-projects.mjs tests/restart-chaos`: clean.

## Risks

- The harness now acts on subtasks, which it did not before. It clicks
once per subtask and only when the product offers Retry.
- The stall-wake check reads activity through the board API. If the API
omits these entries for a board user, the check fails visibly instead of
passing.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:12:03 -06:00
DottaandClaude Opus 5.5 218eb4ec8b test(runner-e2e): hide the eval harness from local native agents (#20044)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Runner E2E evals run real agents against a real Paperclip server,
then grade the result
> - A local cell runs the product from the same checkout as the eval, as
the same OS user
> - In run 42472276601 an OpenCode agent read the eval's own scenario
YAML and grader from that checkout
> - An agent that can read the oracle can steer toward it instead of the
user's request, so the eval stops measuring the product
> - This pull request hides the eval harness from every local native
agent process and removes eval names from fixtures
> - The benefit is that a passing chat eval again shows product
behavior, not oracle access

## Linked Issues or Issue Description

Refs #19614 (the chat catalog eval harness that these graders belong
to).

**What happened**
In run 42472276601, cell
`chat-features.runner-opencode.local.access.cli-authorization`
(OpenCode, DeepSeek), the agent printed its environment and saw the
checkout path in `PATH` and `PAPERCLIP_RUNTIME_NOTIFICATION_HELPER`.
OpenCode's `read` and `grep` tools were denied by its
`external_directory` policy. The agent then used its `bash` tool with
absolute paths, which that policy does not gate. It listed
`/home/runner/_work/paperclip/paperclip`, read `docs/cli/*.md` and
`cli/src/commands/client/*.ts`, ran `grep -rn` over `tests/runner-e2e`,
and found the query in `chat-full-catalog-testops-manifest.json`. It
then read `tests/runner-e2e/chat-scenarios/access.yaml`, the manifest
entry, and the case grader
`chat-catalog-domains/identity-cli-authorization.ts`. Its reasoning
said: "This is the exact test case!" Separately, fixture company names
contained the case id, for example `Runner E2E
chat-features.runner-opencode.local.access.private-resource …`, and an
agent noticed that it was in a test.

**Expected behavior**
The agent under test cannot read the eval's scenarios, fixtures, graders
or captured evidence. It keeps everything that it needs: its workspace,
its home, the product skills, the product packages and temporary files.
The names it can see do not contain case ids, "E2E" or "test".

**Steps to reproduce**
1. Dispatch `runner-full-stack-e2e.yml` for any
`chat-features.runner-opencode.local.*` cell on master.
2. In the agent run, use bash to execute `grep -rn expected_outcome
/home/runner/_work/paperclip/paperclip/tests/runner-e2e/chat-scenarios`.
3. The command returns the scenario expectations. With this change, the
command returns `Permission denied` and exit code 2.

## What Changed

- `tests/runner-e2e/agent-filesystem-sandbox.py` (new): applies a Linux
Landlock domain that grants filesystem access everywhere except the
hidden paths, sets `no_new_privs`, checks in-process that the hidden
paths are denied, then `execv`s the command with the same PID, argv and
environment.
- `tests/runner-e2e/agent-filesystem-sandbox.ts` (new): selects the
hidden paths (`tests/`, `.manual-private-trusted/`, the private evidence
directory, `RUNNER_TEMP`). It never hides the agent's temporary root,
the checkout root or the shared temporary directory. It runs a setup
probe, logs the result and fails closed in CI. Then it wraps each
`paperclip-runnerd` spawn of the E2E server.
- Follow-up: run 42445668869 showed that the hidden list missed two
places. A fixture worker listed Playwright's compiled copy of the
harness and graders with source maps
(`/tmp/playwright-transform-cache-1001/`). An agent in another run
listed the attempt root's `artifacts-private/`, `control/` and
`selected-private-inputs/` under `/tmp`. A sandboxed attempt now creates
its root in `/var/tmp`. The reason: Landlock hides a path by leaving its
ancestors ungranted, and agents must still create files in `/tmp`. The
launcher sets `PWTEST_CACHE_DIR` to a hidden child of that root and
removes the default cache. The hidden list covers the four private
children of the root, including children that are created after runnerd
starts. The probe also checks that the default Playwright cache is not
listable.
- `tests/runner-e2e/launch.ts`: passes the selection to the server
process. The selection applies only to Linux local native cells. It does
not apply to Codex cells (Codex's bubblewrap sandbox cannot mount under
Landlock), installed-release launches, legacy adapters or Daytona.
- `tests/runner-e2e/server-entry.ts`: installs the sandbox before the
product starts. The product server itself stays unrestricted because it
loads this entry from the harness.
- `tests/runner-e2e/fixture-names.ts` (new), `catalog.ts`,
`live-fixtures.ts`: fixture company, agent, environment, secret, AI
connection and project names and descriptions use a neutral word and an
8-character digest suffix. They do not contain case ids or "Runner E2E".
No grader or harness lookup reads these names.
- `tests/runner-e2e/agent-filesystem-sandbox.test.ts` (new): tests the
selection, the launch rewrite and the server-entry order. On Linux with
Landlock, it also tests that a sandboxed `grep` over a fake `tests/` dir
is denied while the product file stays readable.
- `tests/runner-e2e/SECURITY.md`: documents the boundary and its limits.
- The trusted files `manual-private-handoff.ts`,
`manual-private-retention.ts` and `manual-private-report-assets.ts` are
byte-identical to master.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/agent-filesystem-sandbox.test.ts
tests/runner-e2e/live-fixtures.test.ts tests/runner-e2e/catalog.test.ts
tests/runner-e2e/codex-ci-sandbox.test.ts
tests/runner-e2e/server-entry.test.ts`: 151 passed, 1 skipped. The
Landlock test is skipped on macOS. `npx tsc --noEmit -p
tests/runner-e2e` shows no errors in the changed files.
- Paid run
[42480314919](https://github.com/paperclipai/paperclip/actions/runs/42480314919)
for `chat-features.runner-opencode.local.agents.change-manager` ran on
branch `chat-eval/eval-isolation-verify`, which is this commit
cherry-picked onto the chat catalog candidate. The cell passed. Its
`server.log` records:
- `probe …
"harnessFile":"denied","harnessDirectory":"denied","productFile":"readable","temporaryWrite":"allowed"`,
and `denied` for each of `…/paperclip/tests`,
`…/.manual-private-trusted`,
`…/_temp/manual-private-evidence-*/capture`,
`…/_temp/manual-private-evidence-*` and `…/_temp`. `"failures":[]`.
  - `runnerd launched inside the sandbox (pid 3548)`.
- The fixture company was `Company cc369a2e`. No captured snapshot
contains "Runner E2E".
- The probe is a regression check on every local native cell: a missing
or ineffective sandbox stops the attempt in CI before an agent starts.

## Risks

- Landlock does not hide that a path exists. `stat` of
`tests/runner-e2e` still succeeds (the probe logs `harnessDirectoryStat:
visible`). Reading and listing are denied.
- Directories above a hidden path (for example `/`, `/home/runner` and
the checkout root) cannot be listed. New top-level entries in those
directories are not accessible after launch. Agent state, `HOME` and
temporary files are in `/tmp`, which stays fully granted.
- Not covered: Codex cells, legacy adapters, installed-release launches
and macOS. An agent that can reach the Docker daemon can still mount the
checkout into a container. `ps` still shows the launcher's command line,
which includes the case id. The temporary root
(`paperclip-runner-e2e-*`) and instance id (`runner-e2e-*`) in the
agent's paths still say "e2e". The chat catalog candidate depends on
those prefixes, so this PR does not rename them.
- Python 3 must be present on Linux runners. Ubuntu runners have it, and
the harness already uses it.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), agentic coding with
tool use (shell, file edits, GitHub CLI).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:10:21 -06:00
DottaandClaude Opus 5.5 c0fc7608ee fix(agents): honor an explicit hiring denial on CEO agents (#20041)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The board controls which agents can hire, with the "Can create
agents" permission on each agent
> - A CEO agent with that permission turned off could still hire,
because the CEO role allowed hiring before the server read the saved
permission
> - In chat, a CEO agent asked to hire then created the agent through
the `hire_agent` tool, against the board's policy
> - This pull request makes the saved permission decide, and uses the
CEO role only for old rows that never saved a value
> - It also tells a denied agent that the denial is final and where a
person can act
> - The benefit is that the permissions page and the server now agree,
and the board can trust a hiring denial

## Linked Issues or Issue Description

**What happened**
A board user turned off "Can create agents" for a CEO agent. Then the
user asked the CEO in chat to hire a QA engineer. The CEO called
`hire_agent` (`POST /api/companies/{companyId}/agent-hires`). The server
returned 201 and created the agent. The legacy creator rule allowed any
agent with `role: "ceo"` before it read `permissions.canCreateAgents`.

**Expected behavior**
An explicit `canCreateAgents: false` denies hiring for every role,
including CEO. The agent gets a 403 that says the denial is final, and
it sends the person to the hire form or to its permissions page.

**Steps to reproduce**
1. Create a company with a CEO agent.
2. On the CEO's permissions page, turn off "Can create agents".
3. In chat with the CEO, send "Hire a QA engineer named Morgan."
4. See that a new agent named Morgan exists.

Found by the chat eval case `agents.hire-denied`. Related: #19614 (chat
eval harness).

## What Changed

- `canCreateAgentsLegacy` (authorization) uses the role-aware permission
normalization. A CEO keeps hiring unless the board recorded a hiring
denial.
- New `ceoHiringDeniedBy` record in `permissions` (`{ actorType,
actorId, at }`). `PATCH /agents/:id/permissions` writes it when an
authorized actor saves a CEO's hiring as off (an earlier record is
kept). The page shows the effective value, so a saved off is the board's
choice. Turning the toggle off also removes an explicit `agents:create`
grant. A saved `canCreateAgents: false` restricts a CEO only together
with this record.
- Reads report the effective value. A CEO row with no value, or with an
unrecorded false, shows hiring on. A CEO with a recorded denial shows it
off.
- Create and hire inputs cannot supply the record. Configuration updates
keep an existing record unless they turn hiring on. Company export keeps
a CEO's recorded denial, and import restores it, stamped by the
importing user.
- `PATCH /agents/:id/permissions`: an agent actor needs `agents:create`
to change `canCreateAgents`. A denied CEO cannot restore hiring for
itself or grant it to another agent.
- `assertCanCreateAgentsForCompany` adds a final, actionable sentence to
an agent's `agents:create` denial on hiring routes only (`POST /agents`,
`POST /agent-hires`, the hiring toggle on `PATCH /permissions`).
- `POST /api/companies/:companyId/imports/apply` (CEO-safe import): an
agent import that would create agents needs an `agents:create` decision
before any write. Before, any CEO agent could create agents this way.
Import also removes an existing `agents:create` grant when it restores a
CEO's denial.
- Chat guidance tells the agent to link `/<prefix>/agents/new` and its
own `/permissions` page when it cannot hire.
- Regression tests: authorization (a recorded denial is honored; an
unrecorded false keeps CEO hiring), normalization, route tests (denial
text, self-escalation, the record is written only when a CEO is turned
off), service tests for the permission and configuration write paths,
chat guidance text.

### Why a recorded denial and not a plain false

These normal paths save `canCreateAgents: false` on a CEO without a
board choice:

- The permissions page posts the full object. Before this change, a
legacy CEO row with `{}` was read as false, so saving any other toggle
stored false.
- A company import that updates an existing agent normalizes the
manifest permissions in the "stored" context. A manifest without the
field stores false. Export drops false values, so this is common.
- Low-trust creation stores false. If the trust preset later changes to
standard, the false remains.

These paths store true or nothing for a CEO: onboarding seed
(`requestHire`, create context: true), teams-catalog and import creation
(create context: true; the bundled catalog sets true only for a CTO),
built-in agents (no CEO roles), migrations and seed data (no writes to
this field).

A plain-false rule would remove hiring from those CEOs. With the record,
existing data keeps today's behavior, and no data migration is
necessary.

## Verification

- `cd server && npx vitest run
src/__tests__/agent-permissions-service.test.ts
src/__tests__/authorization-service.test.ts
src/__tests__/agent-permissions-routes.test.ts
src/__tests__/agent-conversations.test.ts` passes (244 tests).
- `teams-catalog-service`, `company-portability` and `built-in-agents`
tests pass (189 tests).
- Red check: the new authorization test fails without the
`authorization.ts` change.
- `npx tsc --noEmit -p server` passes.
- End to end: the chat eval `agents.hire-denied` on the OpenCode runner.
The baseline run (Actions run 42478870236) created Morgan through
`hire_agent`. With this fix (Actions run 42480936881), the agent got the
403, created no agent, and linked the hire form and its permissions
page. The eval clicked the rendered hire-form link.

## Risks

- Behavior change: a CEO whose hiring the board turns off after this
change can no longer hire.
- A CEO that already has a saved false keeps hiring until a board user
turns it off again on the permissions page. The page now shows such a
CEO as on, so the board can see and change it.
- A request that creates a CEO with `canCreateAgents: false` does not
restrict it. The board must turn hiring off on the permissions page.
- A CEO agent whose hiring is denied can no longer import a package that
creates agents. A board user can still import it.
- Other CEO special cases are not hiring routes: `issues.ts` lets a CEO
link approvals to issues, and `agents.ts` gives a CEO task assignment.
`teams-catalog.ts` and `llms.ts` read the normalized permissions, so
they follow the same effective value.
- The denial text is longer. No existing test compares this route's
exact error text.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-08 20:05:15 -06:00
DottaandClaude Opus 5.5 a0b9895bbd fix(native): roll back an undelivered warm attach rotation once its runner is proven dead (#20146)
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A follow-up turn can attach to the previous turn's warm local
runner. The transport first rotates authority: it archives the previous
epoch (suspended runner state and settled controller) under
`authority-epochs/` and writes a new control plane that holds only the
`run.attach` command
> - A hard kill in that window, before the runner connects, leaves the
root with the new control plane and no `runner/runner-state.json`.
Restart recovery proves the warm runner dead and classifies the run
`resume_dead_runner`
> - The executor saw a root with the run's own identity and no runner
state, called its authority indeterminate, and quarantined it. The run
failed with `runner_state_identity_mismatch: authority_indeterminate`
and needed a Retry
> - The attach was never delivered, and the runner that would have
received it is proven dead, so it cannot have taken effect. The root's
real authority is still the archived previous epoch
> - This pull request rolls back such an undelivered rotation, so the
follow-up continues from the previous turn
> - The benefit is that a restart in the first second of a warm
follow-up no longer needs a Retry

## Linked Issues or Issue Description

Refs #20140

**What happened?**
Local restart-chaos K13 regression storm w-20261108T235957-s9401 (master
4420a5adcf), task t22 (codex C1):
1. Follow-up run `7b1b1650` started at 00:05:56 as a warm attach to the
previous run's runner (pid 11436).
2. The transport archived the previous epoch (`313ee831`: runner
`suspended`, every command completed) to `authority-epochs/epoch-2c2a…`.
It wrote a new control plane with `run.attach` pending,
`connectionCount` 0, no events, no delivery counts and
`bootstrapHistory.adopted` false.
3. A K1 at 00:05:57 killed the server before the runner connected.
Recovery stopped pid 11436 and claimed `resume_dead_runner`.
4. The executor found `runner/` non-empty (provider state) but without
`runner-state.json`, returned `authority_indeterminate`, and quarantined
the root. A Retry ran fresh.

**Expected behavior**
When a `resume_dead_runner` claim proves the local runner gone and the
run's own root holds only an undelivered `run.attach` over an archived
suspended epoch, recovery restores that epoch and continues. If delivery
is uncertain (any connection, event, delivery count or other command) or
there is no dead-runner claim, it still fails closed.

**Steps to reproduce**
1. Start a warm follow-up and hard-kill the control plane after the
authority rotation, before the runner connects.
2. Restart. Before this change, the follow-up fails with
`runner_state_identity_mismatch: authority_indeterminate`.

**Paperclip version or commit**
4420a5adcf (wave). Branch rebased on current master.

## What Changed

- `native-session-executor.ts`: the new
`rollBackUndeliveredAttachRotation` runs only for a `resume_dead_runner`
claim on this run with a local runner. It acts only when all of these
hold:
  - the active control plane is this run's own exact identity;
  - its only command is a pending `run.attach`;
- `connectionCount`, `ackedSourceSeq` and the delivery counts are 0,
there are no committed events, and it was never adopted and issued no
credentials;
  - it has no warm, staged or completed transition;
  - there is no `runner/runner-state.json`;
- the latest archived epoch is named by its identity's digest, shares
the runner instance, lease and session, and holds a `suspended` runner
state for that identity.

It moves the undelivered control plane to `continuity-breaks/` as
evidence, then puts the epoch's control plane and runner state back. The
executor then sees the previous owner and takes the normal
suspended-previous-owner path.
- `native-recovered-owner-followup.test.ts`: a red/green test of the
undelivered rotation. A possibly delivered attach (`connectionCount` 1)
and a missing dead-runner claim both still fail closed.

## Verification

- Red/green: on master, the undelivered case fails with exactly the t22
error. With the change, the follow-up is admitted: the previous control
plane is restored, `authority-epochs` is gone, and the undelivered
control plane is kept under `continuity-breaks`. The two fail-closed
cases fail on both.
- `vitest run` of `native-recovered-owner-followup.test.ts`,
`native-session-executor.test.ts`,
`native-runner-restart-recovery.integration.test.ts` and
`native-session-root-entries.test.ts`: 4 files, 1508 tests passed (14
skipped); after rebasing onto #20140,
`native-recovered-owner-followup.test.ts` again: 65 passed.
- `PAPERCLIP_TYPECHECK_LOW_MEMORY=1 pnpm typecheck` (`server`): passed.

## Risks

- Medium-low risk. The rollback changes durable state, but only for a
closed shape that proves the attach never reached any runner, after
recovery proved the runner process dead.
- A crash in the middle of the rollback leaves a root that still fails
closed, as before this change.
- The test exercises admission up to tool authority, not a full
real-runnerd turn. A chaos rerun of the K1 warm-follow-up window is the
remaining check.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1109.0-canary.2
2026-11-08 19:36:44 -06:00