Jannik/PaperClipAI
1
0 Fork 0
mirror of https://github.com/paperclipai/paperclip.git synced 2026-11-11 05:11:49 +01:00
Code Issues Packages Projects Releases Wiki Activity
7,439 Commits 1,761 Branches 2,417 Tags
2eede8b3183f877f9afbf06655aa1eee07590d28
Commit Graph
7439 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
DottaandClaude 2eede8b318 fix(native): give a relaunched settled runner its admitted provider pack (#20683) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - After a server restart or a same-run retry, a remote native run
reattaches to its runner in the sandbox (`reattach_remote_runner`).
> - When the controller saved the runner artifact, the executor prepares
that run as an adoption. It skips provider-pack selection, because a
live adopted runner keeps its launched provider.
> - The reattach probe can instead find the runner settled and exited
(`verifyRemoteRunnerOrSettledExit`). Then nothing is adopted, and the
transport relaunches the runner from its durable state. That relaunch
must start the provider again, so it needs the provider-pack launch
inputs.
> - The executor did not give them. OpenCode failed with "OpenCode
executable is missing from the provider pack". ACPX failed with "ACPX
launch profile omitted its provider-pack authority or verified artifact
digests". Both ended the run.
> - This pull request selects the saved (admitted) provider-pack
authority on that settled-exit relaunch and verifies it as an exact
installed identity, as an ordinary reattach does.
> - The benefit is that a restart or a same-run retry can resume an
exited runner with the same provider pack, instead of failing.

## Linked Issues or Issue Description

**What happened?**
On a hosted staging instance, two OpenCode runs and one ACPX (Claude)
run were interrupted by a deploy (SIGTERM) or ended a turn without a
result. The server scheduled a same-run retry that reattached to the
existing remote runner. The runner had exited cleanly
(`runner_remote_process_exited lifecycle=suspended`). The retry then
failed within three seconds with
`runner_remote_provider_artifact_incompatible`.

**Steps to reproduce**
1. Prepare a remote OpenCode or ACPX run with a persisted runner
artifact (`controllerPersistence.runnerArtifact`).
2. Reattach with `restartRecovery.kind = "reattach_remote_runner"`,
where the runner process is not running and its state is `suspended` and
settled.
3. Observe that the transport options have no `adoptExistingRunner`, no
`providerPackAuthorityDigest` and no `opencodeCommand` /
`acpxSidecarSha256`.

**Expected behavior**
The relaunch gets the admitted provider pack's launch inputs, or it
fails with a clear pack-identity error.

## What Changed

- `createRunnerdBackend` (native session executor): when a
persisted-runner reattachment finds no live runner to adopt, select the
saved provider-pack authority (`reattach: true`), so
`expectedProviderPackManifest` is the admitted pack.
- `prepareRemoteRunnerAssets` no longer returns early for that relaunch.
It verifies the installed pack as an exact identity and sets the active
pack root, the same path as a reattach without a persisted artifact.
- A live adopted runner is unchanged: it still skips pack selection.
- The test `uses the bundled Linux image authority for remote
opencode|claude-x64` now also covers the settled-exit relaunch for
OpenCode and ACPX.

## Verification

- RED on `origin/master` (`c444da7d05`): the extended test fails for
`opencode` and `claude-x64` (the relaunch options have no provider-pack
inputs).
- GREEN with this change: `pnpm --filter @paperclipai/server exec vitest
run src/services/native-runtime/native-session-executor.test.ts -t "uses
the bundled Linux image authority"` (14 passed).
- `pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify`: see the result below.

## Risks

- Upgrade safety: the relaunch reads the saved authority, so a newer
controller does not select a newer pack for an older session. It
requires an exact installed identity. If the sandbox no longer has that
pack, the relaunch fails with a pack-identity error, and the existing
Retry path applies, as before.
- No change to a live adoption, to fresh runs or to the normal path
before `turn.start`.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only staging API reads).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:14:43 -06:00
DottaandClaude e33ede9a8a fix(native): settle a restart-orphaned write_document from its transactional receipt (#20674) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run calls Paperclip tools such as `write_document`. The
runner saves each tool input durably before the server runs the tool,
and the server saves a result receipt
> - A control-plane restart can cut a `write_document` call after the
runner saved its input and before any result was saved
> - Recovery then has no proof of the outcome. A call from an earlier
server process settles as "unknown outcome" (#20628). A call the current
server process admitted stays pending, close cannot prove that the
provider drained, and the session quarantines
(`native_session_cleanup_quarantined`). A person must click Retry
> - `write_document` saves its revision and its idempotency receipt in
one database transaction, under the run and issue row locks. The
revision is also compare-and-swap on `baseRevisionId`. So the receipt is
a durable outcome record: present means applied, absent means not
applied
> - This pull request makes recovery use that proof. Recovery runs the
same receipt-guarded write. It replays the saved result, or it applies
the write exactly once
> - The benefit is that an orphaned `write_document` settles with its
real result. The document is never written twice, and the run does not
need Retry for this call

## Linked Issues or Issue Description

**What happened?**
Daytona restart-chaos wave `w-20261110T174930-s9205`, task t01 (codex,
workload C1, fault RR2). A restart cut a `write_document` call. The
close settlement showed `semanticTools.pending: [{ operationId:
"write_document", callId: "exec-8b79ab78-…" }]` and `providerDrained:
false`. The session quarantined with
`native_session_cleanup_quarantined`, and the task needed Retry. Since
#20628, a call that an earlier server process admitted settles as
`tool_call_interrupted_by_restart` with an unknown outcome. The agent
then has to guess whether the write applied.

**Expected behavior**
When a durable record proves the outcome, recovery uses it. A write that
applied returns its saved result. A write that did not apply runs once.
When the record cannot prove it, the call stays as it is today.

**Steps to reproduce**
1. On master, run `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts -t
"orphaned"`.
2. `authority.recover()` returns `null` for a `write_document` that has
no receipt, so the call stays unsettled.

Refs #20628, #20639, #20507

## What Changed

- `PaperclipRunnerToolAuthority.recover()` now settles `write_document`
through its transactional receipt (`#recoverTransactionalWrite`). It
runs the same receipt-guarded mutation as an ordinary call. If the
receipt exists, it returns the saved result and writes nothing. If no
receipt exists, it applies the write once.
- A definite refusal that rolls the transaction back (a 4xx such as a
stale `baseRevisionId`, a reused idempotency key, invalid input) settles
as a tool error. Any other failure leaves the call unsettled, as before.
- The set of operations with this proof is explicit:
`TRANSACTIONAL_RECEIPT_OPERATIONS` contains only `write_document`. Other
tools keep their current recovery rules.
- New test: an orphaned `write_document` with no receipt applies once; a
second recovery and a late original call replay the receipt; an applied
write recovers its saved result; a stale base settles as a tool error
with no change.
- Updated test: a reused idempotency key with other input now settles as
an `paperclip_runner_tool_idempotency_conflict` tool error instead of
staying pending. The receipt proves that this call can never apply under
that key.
- Docs: `packages/paperclip-runner/docs/durable-recovery.md` (semantic
operations that outlive a turn) and a new design note,
`doc/plans/2026-11-10-durable-turn-start-and-tool-call-records.md`. The
note also covers the second change (an ACPX turn start cut off before
the prompt reached the provider), which comes in a separate pull
request.

## Verification

- Red on master, green here: `pnpm --filter @paperclipai/server exec
vitest run
src/services/native-runtime/paperclip-runner-tool-authority.test.ts` (43
passed).
- `pnpm --filter @paperclipai/server exec vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`:
178 passed. One unrelated ACPX timing test failed once under load and
passed when run alone.
- Server typecheck: `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 (a runner doc changed), so required CI supplies
the broad coverage.
- Live restart-chaos wave: see the comment on this pull request.

## Risks

- Recovery can now apply a write that never applied before the restart.
This is the call the agent made, applied once. The write is
compare-and-swap on `baseRevisionId`, so it cannot overwrite a revision
that someone wrote after the agent read the document. That case settles
as a conflict error.
- Upgrade safety: no saved shape changes. Receipts saved by the previous
release have the same shape. A server on the previous release keeps its
current recovery for these calls.
- Time to first token: no change. This code runs only during recovery.
- No runner source changes, so no new runner image is needed.

## Model Used

Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
file editing) in Claude Code. Extended reasoning was used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 19:05:15 -06:00
DottaandClaude cec903c6f0 fix(native): retire a finished remote owner whose gone Runner left a delivery unacknowledged (#20677) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native follow-up run on a remote sandbox first proves that the
previous per-turn owner finished. Then it retires that owner's
controller journal and starts a new Runner.
> - The proof requires a settled journal. A command that the Runner did
not acknowledge holds the owner.
> - On staging, a per-turn owner committed its result and its Daytona
sandbox was destroyed after the turn
(`paperclip_runner_destroy_after_turn`, cleanup `success`). Its journal
still held a `run.attach` or a `semantic_tool.result` that the Runner
never acknowledged.
> - The next run failed with `prior_owner_terminal_state_indeterminate;
retirement_refused=journal_commands_unsettled(run.attach.pending)` (or
`semantic_tool.result.pending`). A Retry or a new message gets the same
journal, so it fails the same way. The task stays blocked.
> - Retirement already excuses the Runner's own lifecycle commands when
the prior sandbox lease ended on another sandbox. This pull request
applies the same rule to deliveries that only that gone Runner could
receive.
> - The benefit is that these follow-ups start by themselves. Commands
that carry user intent still hold the owner.

## Linked Issues or Issue Description

**What happened?**
A census of staging failures (48 h, Nov 9–10) found 6 failed runs with
`prior_owner_terminal_state_indeterminate` and
`retirement_refused=journal_commands_unsettled(run.attach.pending |
semantic_tool.result.pending)`. The last one was at 18:56 UTC on Nov 10.
In two examples that I checked:
- The previous run succeeded. Its result was committed and its
coordinator lease was released.
- Its lease policy was `ephemeral`. The lease is `expired`, with
`failureReason=paperclip_runner_destroy_after_turn` and
`cleanupStatus=success`.
- The follow-up got a new sandbox (a different provider lease) and
refused in about 11 s. It started no Runner. The task became `blocked`,
and the next action was "Send a message to continue".

**Expected behavior**
When the record proves that the previous Runner is gone, an undelivered
attachment, tool answer or read cannot run. The follow-up retires the
finished owner and starts a new Runner. Retirement replays no command.

**Steps to reproduce**
1. Finish a per-turn remote native run whose controller journal keeps a
`run.attach` or `semantic_tool.result` command in `pending`.
2. Let the sandbox lease of that run end (`expired` or `released`) on
its own sandbox.
3. Start a follow-up run that gets a new sandbox.
4. Observe
`retirement_refused=journal_commands_unsettled(run.attach.pending)`.

**Paperclip version or commit**
Staging builds of Nov 10. The refusal still reproduces on `master` at
`48c6e996da` (the new tests fail there).

## What Changed

- `native-session-executor.ts` (`readSettledRunnerdProjection`): when
the prior run's sandbox leases ended on another sandbox
(`remoteRunnerGone`, the existing proof), unfinished `run.attach`,
`semantic_tool.result`, `session.snapshot` and `session.goal.get`
commands no longer hold the retirement. These join the lifecycle
commands that the rule already excused.
- In the same case, a journaled `semantic_tool.result` answers its
`semantic_tool.input`. The controller journals the answer after the tool
ran. Only its delivery was outstanding.
- No other proof changes. The prior run must still be terminal, with a
committed result, a released coordinator lease, no recovery state, a
stopped or fenced controller, a delivered `run.terminal`, and no live
work after it. Quarantine copies must pass the same checks.
- `doc/execution-semantics.md`: describe the rule.

## Verification

-
`server/src/services/native-runtime/native-recovered-owner-followup.test.ts`
(real Postgres):
- New: "retires a finished remote owner with an unacknowledged
pending-attach / pending-tool-result when its sandbox lease ended on
another sandbox". It fails on `master` with the exact staging errors
`journal_commands_unsettled(run.attach.pending)` and
`journal_commands_unsettled(semantic_tool.result.pending)`. It passes
with this change, and the log shows the retirement.
- Held cases stay held: a lease that is still retained, a lease on this
run's sandbox, no lease record, and a tool input with no journaled
answer (`journal_tool_input_unanswered`).
- New: "keeps a stopped local owner held by an unacknowledged
run.attach". A local Runner is not gone by record.
  - Whole file: 95 passed.
- Neighbouring suites pass: `explicit-native-continuation.test.ts` and
`native-controller-recovery.integration.test.ts` (460 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2: a global CI input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Live proof: not done. The redeploy chaos scenario (K15) is not on
`master` yet. Tests only.

## Risks

- The rule applies only when every sandbox lease of the prior run is
`released` or `expired`, has a release time, and names a different
provider lease than this run's. A reusable sandbox that this run resumed
keeps the existing hold.
- A command that carries user intent (`turn.start`, `turn.steer`,
`request.resolve`, `interaction.receipt`, `external_provider.operation`,
`session.goal.set`/`clear`, `session.close`, `run.cancel` and others)
still holds the owner.
- Retirement archives the old journal under `continuity-breaks` and
replays nothing, as before.
- No persisted shape changes. Old saved journals are read by the same
rule.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 18:49:26 -06:00
DottaandClaude Opus 5.5 ad0543ae90 test(recovery): re-arm a capture-policy save whose finished coordinator names a lapsed owner (#20672) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Workspace recovery re-arms a save that an older capture policy
exhausted (#20569), so the files are saved without a person
> - On staging, one save of this kind was not re-armed after the deploy.
Its export hold also carries a workspace finalization block, and it has
a resolved no-replay sibling for the same run
> - #20661 found that discovery skipped a hold when the run's finished
coordinator still named a lease owner whose lease had lapsed. The same
discovery clause gates the capture-policy re-arm
> - This pull request adds a regression test for that exact
capture-policy shape. It shows that #20661 alone makes the worker re-arm
and save it
> - The benefit is that the capture-policy path keeps this guarantee in
later changes

## Linked Issues or Issue Description

Refs #20661
Refs #20569

**What happened?**
An exhausted capture-policy save (`export_interrupted`, cause
`provider_error`, 6 of 6 attempts, phase `preserve_files`) was not
re-armed on the worker's first pass after #20569 was deployed.

**What did you expect to happen?**
The worker re-arms the save once under the current capture policy, saves
the files, and returns the hold to the system.

**Steps to reproduce**
1. Exhaust a save under capture policy 1 with `export_interrupted`. Add
a workspace finalization block and a resolved no-replay sibling for the
same run.
2. Leave a lapsed `native-finalizer:` lease owner on the run's
`terminal_failure` coordinator.
3. Run the workspace recovery worker. Before #20661,
`nextWorkspaceRecoveryAt` returns null and the hold is never selected.

## What Changed

- `workspace-recovery-capture-policy.test.ts`: add "re-arms a save whose
finished coordinator still names a lapsed finalizer owner". The test
uses the staging record: export hold with `historicalWorkspaceSource`
and `workspaceFinalizationBlock`, a resolved no-replay sibling, waiting
user input, and a lapsed finalizer owner. It drives the worker's own
discovery loop until the worker is idle.
- No production code changes.

## Verification

- Red: with the source files from before #20661 (`git checkout
686893636d^ -- <the four source files>`), the new test fails.
`nextWorkspaceRecoveryAt` returns null, so the worker is never woken.
- Green: on current master, `pnpm --filter @paperclipai/server exec
vitest run src/services/workspace-recovery-capture-policy.test.ts`
passes (5 tests). The file passed 4 runs in a row.
- The test asserts that the save completes under policy 2, the hold
returns to the system, `releasedLapsedOwner` is recorded, and the lapsed
owner is released.

## Risks

- Low risk. This change adds a test only.
- Previous-release story: the test uses rows in the shape an older
release saved, with no capture-policy marker. No migration is needed.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:47:29 -06:00
DottaandClaude Opus 5.5 e9db5e0d03 perf(github): read the inherited review waits of an issue list in one pass (#20673) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - An agent can review a GitHub pull request. After the agent publishes
the review, the task can wait in review for the next GitHub reply. A
later run of the same task can inherit this wait.
> - Issue lists and inbox attention show these waits. For each waiting
task, the server reads the inherited GitHub review wait: the source
comment, the latest review, and the review publication.
> - The server did these reads one time for each waiting task. A list
request with N waiting tasks sent N copies of the prepared review
statement and N copies of the publication read.
> - On a hosted database, the prepared review statement ran about
270,000 times in about 2.5 hours (up to 87 calls per second). The run
identity read before it ran the same number of times. The list read that
calls it ran only about 1,400 times, so each list read started about 190
review reads on average. No other caller reads the waits of many runs in
a loop.
> - This pull request reads the reviews and the publications of all
waits of one list request with one statement each. Each wait gets the
same result as before.
> - The benefit is about 200 times fewer review statements and fewer
pool connections for each issue list.

## Linked Issues or Issue Description

Refs #20400 (shares review-wait reads between the callers of one
request).
Refs #20551 (authorizes the waits of one issue page with one read; this
change does the same for the inherited GitHub review reads of that
page).

**What happened**

`listExternalReviewWaits` called `readInheritedGitHubReviewWait` one
time for each candidate wait, inside `Promise.all`. Each call sent a run
identity read, a source-message read, the prepared
`inherited_github_review_wait` statement, and a publication read. A list
with N waits sent each statement N times.

**Expected behavior**

One list request sends one review statement and one publication read for
each company. The result of each wait does not change.

**Steps to reproduce**

1. Put many tasks of one company in review with an inherited GitHub
review wait.
2. Open the issue list or the inbox.
3. Count the statements of the database. The review statement runs one
time for each waiting task.

## What Changed

- `server/src/services/chat-channels/github/review-wait.ts`:
- Split `readInheritedGitHubReviewWait` into shared steps: the source
comment read, the review checks, the publication key, and the published
result. The single-run read uses the same prepared statement and the
same checks as before.
- Add `readInheritedGitHubReviewWaits(db, runs)`. It reads the source
comment of each run with the same lineage check as before. Then, for
each company, one `DISTINCT ON` statement reads the latest review for
each (source comment, task, agent) key, with the same joins, filters and
order as the prepared statement. One read finds the processed
publications. Each run gets the same result as a separate call: `null`,
`revoked`, or `published`.
- `server/src/services/native-runtime/external-review-wait.ts`:
`listExternalReviewWaits` uses the batch read. The rest of the list and
its authorization do not change.
- `server/src/__tests__/github-review-wait.test.ts`: a new test reads 18
runs in one batch and compares each result with a separate single-run
read. The runs include each revocation case (connection disabled or
inactive, review state, review agent, repository, repository id, thread,
publication, identity, company), a newer unreviewed head, no identity,
no task, an endpoint of another agent, a run in another company and a
duplicate run. The test also makes sure that the batch sends one review
statement and one publication read for each company. The existing
revocation tests also check the batch read.
- `server/src/__tests__/helpers/github-review-wait.ts`: the seed helper
can add a wait to an existing company and agent.

## Verification

- `pnpm vitest run server/src/__tests__/github-review-wait.test.ts
server/src/__tests__/github-review-check-state.test.ts
server/src/__tests__/issue-list-review-wait-authorization.integration.test.ts
server/src/__tests__/blocked-inbox-review-wait-scope.test.ts`
- Focused local run of every test that imports the changed modules or
the seed helper (7 files, 335 tests passed), including
`external-chat-wait.integration.test.ts`, `external-review-wait.test.ts`
and `legacy-execution-recovery.test.ts`. The change to the shared seed
helper makes `pnpm verify` select the full server suite, so CI supplies
that coverage.
- Local measurement on real PostgreSQL (embedded PostgreSQL, `ANALYZE`
after seeding). One company and agent, each wait has 5 reviews and 10
outbound messages. "Before" is the previous list code: one
`readInheritedGitHubReviewWait` call for each run in `Promise.all`.
"After" is `readInheritedGitHubReviewWaits`. The runs alternate order in
each pair. Both return equal results in every sample.

| Waits | Version | Statements | Review statements | Median ms | Min–max
ms | Samples |
| --- | --- | --- | --- | --- | --- | --- |
| 200 | before | 800 | 200 | 106.8 | 98.0–117.5 | 10 |
| 200 | after | 402 | 1 | 44.3 | 41.6–50.9 | 10 |
| 20 | before | 80 | 20 | 9.8 | 8.6–14.7 | 12 |
| 20 | after | 42 | 1 | 6.3 | 5.3–7.0 | 12 |

- `EXPLAIN (ANALYZE, BUFFERS)` of the batch review statement for 200
waits: index scans on each join, 6.7 ms execution. The single statement:
0.05 ms execution and about 2.7 ms planning for each call.
- The run identity read and the source-message read still run one time
for each wait. They are cheap on the hosted database (about 0 ms each),
and they keep the lineage check for each run.

## Risks

- Low risk. The batch statement uses the same joins, filters and order
as the prepared statement, and `DISTINCT ON` keeps the first row for
each key. The array filters can also match a mix of two keys. The code
uses only rows of exact requested keys.
- The single-run read and its prepared statement do not change. The
finalizer, the status committer, and the attachment reuse paths still
use it.
- Errors do not change: a 403 from the lineage check gives `revoked`,
and other errors are thrown.
- No migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (code reading, editing, shell, tests).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1111.0-canary.1 		2026-11-10 18:35:39 -06:00
DottaandClaude Opus 5.5 c444da7d05 perf(run-events): stop blocking checkpoint FK checks during event appends (#20670) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A native run sends a stream of runner events. The server appends
each event to the run log. The runner controller then saves a checkpoint
for recovery.
> - Each checkpoint save inserts the new `event:N` row and updates the
`cursor` row in `native_runner_checkpoints`. The new row has a foreign
key to the run row in `heartbeat_runs`.
> - The event append locks the same run row `FOR UPDATE`. That lock
blocks the `FOR KEY SHARE` lock that the foreign key check takes. So
each checkpoint save waits for the event append that is in progress.
> - On a hosted database, the two-row checkpoint upsert had the largest
total SQL time. The one-row (cursor-only) upsert, which has no foreign
key check, was much cheaper.
> - This pull request changes the append lock to `FOR NO KEY UPDATE`.
Appenders still serialize. Child foreign key checks no longer wait.
> - The benefit is shorter checkpoint saves, shorter waits before each
runner ACK, and less lock wait on the database.

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The controller checkpoint save for native runs
(`nativeRunnerCheckpointWriter`), and the native run-event append
(`appendHeartbeatRunEvent`).

**Subsystem affected**
Server run-event log and native runtime checkpoint persistence.

**Current behavior**
On one hosted database, Neon query performance showed this statement as
the largest total: `insert into native_runner_checkpoints ... values
(...), (...) on conflict (run_id, part) do update`. It ran 6,582 times
at 62.6 ms (412 s) in the first 38 minutes after a compute restart, and
9,435 times at 31.8 ms (300 s) about two hours later. An earlier day
showed 100,650 calls at 102 ms. The one-row form of the same upsert ran
965 times at 13.4 ms.
The runner already writes only changed records (`captureCheckpoint`
compares plaintext), so unchanged material is not rewritten. The two-row
form is a new event row plus the cursor. Only the new row fires the
foreign key check. `appendHeartbeatRunEvent` locks the run row `FOR
UPDATE` for each native event, in a transaction with several statements.
`FOR UPDATE` conflicts with `FOR KEY SHARE`, so the check waits for that
transaction.

**Proposed behavior**
`appendHeartbeatRunEvent` locks the run row `FOR NO KEY UPDATE`. This
mode still conflicts with other appenders, `FOR SHARE`, `FOR UPDATE`,
deletes and updates. It does not conflict with `FOR KEY SHARE`. The
append never changes key columns: `runUpdate` is limited to process
fields, and the native and retry-exhaustion paths reject it. The fast
path already uses a plain `UPDATE`, which takes the same lock mode. The
execution-admission adapter already locks this row `FOR NO KEY UPDATE`.

**Reason and benefit**
The wait goes in both directions. A checkpoint save waits for an append.
An append also waits for an open checkpoint transaction that holds `FOR
KEY SHARE`. Both waits stop. The controller sends each ACK only after
its checkpoint commits, so shorter saves also shorten the event loop.

## What Changed

- `server/src/services/heartbeat-run-events.ts`: lock the run row `FOR
NO KEY UPDATE` instead of `FOR UPDATE` in the transactional append path.
Add a comment that names the contract.
- `heartbeat-run-event-atomic-append.test.ts`: add a regression test. An
append transaction stays open. A checkpoint row insert for the same run
must finish under a 2 s `lock_timeout`. A second appender must still
wait. After release, the sequence is 1, 2 with no gap and no duplicate.
The existing test with 40 concurrent appenders on two connections still
proves contiguous sequence numbers. Update the existing lock-statement
assertion to the new clause.
- Add `scripts/benchmark-run-event-checkpoint-lock.ts`. It replays a
native event stream with the real append and the real checkpoint writer
on separate connections. It times each checkpoint save for a baseline
append module and the current one.

## Verification

- Red: with the old `FOR UPDATE` clause, the new test fails with
`canceling statement due to lock timeout`. With the change it passes.
- `npx vitest run
src/__tests__/heartbeat-run-event-atomic-append.test.ts
src/__tests__/heartbeat-run-event-sequencing.test.ts
src/services/native-runtime/native-runner-checkpoint-authority.integration.test.ts
src/__tests__/native-recovery-notice-work.test.ts` (server): 4 files, 42
tests passed.
- `pnpm --filter @paperclipai/server typecheck`: passed.
- `pnpm verify` exits 2, because a new script under `scripts/` is a
global CI input. Required CI supplies the broad coverage.

Local benchmark. Embedded PostgreSQL 18.1 on a laptop. 400 events per
pass, 2 KiB event payload, 6 KiB header, real encryption. Order:
warm-up, baseline, current, baseline, current. `HOLD_MS` keeps each
append transaction open longer. It stands in for hosted round trips.

| Hold | Pass | Median save | p90 | Max | Saves > 2 ms | Total save time
|
| --- | --- | --- | --- | --- | --- | --- |
| 5 ms | Baseline (1) | 8.16 ms | 9.50 ms | 129.48 ms | 396 / 400 |
3,551 ms |
| 5 ms | Current (2) | 1.44 ms | 2.08 ms | 12.66 ms | 44 / 400 | 677 ms
|
| 5 ms | Baseline (3) | 8.12 ms | 9.64 ms | 22.08 ms | 398 / 400 | 3,453
ms |
| 5 ms | Current (4) | 1.50 ms | 2.37 ms | 10.54 ms | 80 / 400 | 700 ms
|
| 0 ms | Baseline (1) | 1.71 ms | 2.50 ms | 11.05 ms | 105 / 400 | 780
ms |
| 0 ms | Current (2) | 0.97 ms | 1.42 ms | 10.28 ms | 22 / 400 | 458 ms
|
| 0 ms | Baseline (3) | 1.49 ms | 2.02 ms | 6.45 ms | 41 / 400 | 667 ms
|
| 0 ms | Current (4) | 0.91 ms | 1.24 ms | 9.80 ms | 19 / 400 | 420 ms |

- Event order: the controller frame chain awaits each event's append and
its local commit, but not its checkpoint save. The ACK goes through
`#sendJson`, which queues the save and does not block the next frame.
Only semantic inputs and command results await the save. So the save for
event N overlaps the append of event N+1. Streaming events
(`item.delta`) also go through the same locked append, with no
checkpoint save, so a save can also overlap them. The benchmark uses
this order.
- Control without overlap (`SERIAL=1`, each save finishes before the
next append, 5 ms hold): baseline median 1.15 / 1.07 ms, current 1.03 /
1.00 ms. There is no material difference. The gain comes only from the
removed lock wait.
- Results are the same in every pass. The script asserts that all event
rows, the header and the cursor are stored, and that the run-log
sequence is contiguous.
- Command: see the script header. It needs a baseline copy of
`heartbeat-run-events.ts` from the base revision.

## Risks

- Low risk. The SQL change is one lock clause. Appenders still serialize
with each other, with `FOR UPDATE` and `FOR SHARE` lockers, and with
updates and deletes of the run row. Run deletion and key changes take
`FOR UPDATE`, so they still wait for an append, and an append still
waits for them.
- The only new concurrency is with `FOR KEY SHARE` lockers. In this code
base those are foreign key checks of child inserts. The one explicit
`FOR KEY SHARE` read is on `issues`, not on `heartbeat_runs`. A foreign
key check only needs the run row and its key to exist. The append does
not delete the row or change its key.
- The hosted cause is inferred. Neon shows calls and mean time only. We
did not read lock waits on the hosted database. The local proof shows
the mechanism and the size of the effect. A new query-performance window
after deployment must confirm the result.
- Persisted run state: no saved run, session, event or checkpoint shape
changes. State saved by the previous release carries forward unchanged.
The lock mode applies only inside one transaction, so a deploy during a
run needs no special case.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use (shell, file edits, local PostgreSQL).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:27:58 -06:00
DottaandClaude Opus 5.5 81c52e3666 perf(chat): find terminal GitHub responses by issue run indexes (#20671) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The GitHub chat channel posts a terminal response when an agent run
for a GitHub request stops. A background sweep finds acknowledged
response comments (`processed`, not final) whose issue is blocked or
whose latest run is terminal
> - For each candidate, the sweep reads the status of the latest run of
the assigned agent for the issue. It matches runs with
`coalesce(native_issue_id::text, context_snapshot->>'issueId')`
> - No index has that expression. So PostgreSQL reads every run of the
agent for each candidate, and it decompresses the large legacy
`context_snapshot` of each run
> - On a hosted test database, this statement ran 13 times at a mean of
11 s in a 38-minute window (149 s in total), with single runs of 16 s
and 41 s
> - This pull request splits the run match into its two disjoint
branches. Each branch uses an existing issue index
> - The benefit is that each candidate reads a few index entries instead
of the agent's full run history. The returned rows do not change, and no
migration is necessary

## Linked Issues or Issue Description

**What existing behavior does this improve?**
The discovery page query in `registerGitHubTerminalResponseWork` in
`server/src/services/chat-channels/github/terminal-response-work.ts`.
The query runs when the sweep reconstructs its candidates, and once or
twice for each candidate check.

**Subsystem affected**
GitHub chat channel terminal-response recovery.

**Current behavior**
The `WHERE` clause has `issues.status = 'blocked' or (select r.status
from heartbeat_runs r where r.company_id = ... and r.agent_id = ... and
coalesce(r.native_issue_id::text, r.context_snapshot->>'issueId') =
<issue id> order by r.created_at desc, r.id desc limit 1) in (<terminal
statuses>)`. The planner can only use
`heartbeat_runs_company_agent_started_idx`. It reads all runs of the
agent, then evaluates the `coalesce` on each row. That decompresses the
context of each legacy run. The cost is the number of candidates times
the number of agent runs.

**Proposed behavior**
The subquery takes the latest run from two branches and then the latest
of the two:

- runs with `native_issue_id = <issue id>`, through
`heartbeat_runs_company_native_issue_contract_id_uq`
- runs with `native_issue_id is null and context_snapshot->>'issueId' =
<issue id>`, through `heartbeat_runs_company_ctx_issue_created_idx`

The two branches are disjoint, and together they are exactly the rows
where the `coalesce` is equal to the issue ID. Both branches and the
outer query use the same order (`created_at desc, id desc`). So the
selected run is the same, including ties.

**Reason and benefit**
This statement was the largest single-statement cost in the hosted
window after the earlier database fixes. A local reproduction is about
490x faster on the default fixture, and the gain grows with the agent's
run history.

**Breaking changes**
None. The API, the candidate rules and the results do not change.

## What Changed

- Rewrite the latest-run status subquery in the GitHub terminal-response
discovery page as two index-backed branches (native binding, and legacy
context without a native binding).
- Add two integration tests for the sweep. In the first, newer runs of
the same agent that do not bind the issue (one with a native binding to
another issue, but a legacy context that names this issue) must not hide
the stopped run. In the second, a newer active run with a native binding
to the issue keeps the request waiting.
- Add `scripts/benchmark-github-terminal-response-sweep.ts`. It captures
the SQL that the real worker generates at a baseline commit and at
`HEAD`, compares all rows, and measures both.

## Verification

Cause and thesis. The cost comes from the `coalesce` run match, which
has no index and makes each candidate scan the agent's run history.
Thesis: if each precedence branch uses its own issue index, the cost of
the page no longer depends on the agent's run count, and the rows stay
identical. The benchmark confirms the thesis.

Local SQL benchmark:

```sh
pnpm --filter @paperclipai/server exec tsx ../scripts/benchmark-github-terminal-response-sweep.ts --baseline-ref=7b33384446
```

Fixture (synthetic): two companies, each with one GitHub endpoint and
one assigned agent. Each issue has one GitHub response comment. The
first N comments are acknowledged and not final; the others are final.
Every tenth issue is blocked. Runs alternate between native binding and
legacy context, and each run has a 4 KiB incompressible context. The
fixture also has these edge cases: newer active runs with a native
binding to another issue but a legacy context that names this issue,
active runs that tie on `created_at` with the latest run (between native
and legacy runs), and newer active runs of another agent. The benchmark
compares every column of every candidate row (the page limit is lifted
for this check). Embedded PostgreSQL 18.1 on an Apple laptop. The hosted
database is Neon with different compute, so absolute times are local SQL
measurements only. Each size has 2 warmups and then alternates the
baseline and the treatment.

| Size | Runs (both companies) | Candidate rows equal | Baseline median
(range) | Treatment median (range) | Speedup |
| --- | --- | --- | --- | --- | --- |
| Small: 200 issues, 3 runs each, 20 candidates, 7 samples | 1,372 | 26
of 26 | 18.7 ms (17.7–20.8) | 3.8 ms (3.3–4.3) | 4.9x |
| Default: 1,500 issues, 8 runs each, 150 candidates, 7 samples | 25,300
(156 MB) | 200 of 200 | 3,697 ms (2,875–4,032) | 7.6 ms (6.6–9.6) | 490x
|
| Large: 3,000 issues, 10 runs each, 400 candidates, 5 samples | 62,600
(382 MB) | 538 of 538 | 57,055 ms (48,343–66,079) | 31 ms (16–79) |
1,800x |

Sizes are set with `BENCH_ISSUES`, `BENCH_RUNS_PER_ISSUE`,
`BENCH_CANDIDATES` and `BENCH_ITERATIONS`. The timed query is the real
first page (limit 100).

Plan (default size): the baseline runs `SubPlan` 270 times. Each run is
a `Bitmap Heap Scan` on `heartbeat_runs_company_agent_started_idx` that
removes about 12,000 rows by filter, for about 8.2 million buffer reads
in total. The treatment runs an `Append` of two `Limit` index scans
(`heartbeat_runs_company_native_issue_contract_id_uq` and
`heartbeat_runs_company_ctx_issue_created_idx`), for about 3,800 buffer
reads in total.

Tests:

- `cd server && npx vitest run
src/__tests__/chat-channels.integration.test.ts -t "terminal GitHub"`:
13 passed, including the 2 new tests.
- Mutation check: when the legacy branch does not exclude runs with a
native binding, the first new test fails. It passes with the fix.
- `cd server && npx tsc --noEmit`: passed.

## Risks

- Low risk. The change is one SQL subquery. The match is the same set of
runs: `native_issue_id::text = <uuid text>` is equal to `native_issue_id
= <uuid>`, and a null `native_issue_id` makes the `coalesce` use the
legacy context.
- The candidate set itself is not bounded by this change. Acknowledged
responses that never become final stay in the candidate set. Each
candidate is now cheap, but the count still grows with that history.
- `readGitHubResponseRun` in `response-run-state.ts` uses the same
`coalesce` match. The sweep calls it once for each candidate row, and
other GitHub paths also call it. Each call is a separate, smaller
statement. This pull request does not change it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic. Used through Claude
Code with tool use (shell, file edits, and local PostgreSQL benchmarks).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:25:59 -06:00
DottaandClaude c14f2fe45d fix(native): read the scheduled-resume evidence the resume reconciler claimed (#20656) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20645 lets a local same-run scheduled resume send the one
same-conversation continuation for a turn lost on restore. It applies
only when durable records prove three facts: the previous runner
stopped, the attempt was admitted within the budget, and it is the same
checkpointed conversation.
> - It read the third proof from the coordinator's saved
`failure_detail` while the phase was `retryable_failure`.
> - In production, the resume reconciler
(`claimNativeSessionResumptions`) claims the run first. It moves the
coordinator to `observed` and clears `failure_detail`. So the executor
never saw that evidence, and #20645 refused every real scheduled resume.
> - The restart-chaos L7 wave showed this live. Codex B5 TM2 hit the
exact path (`runner.drain` timeout, scheduled resume,
`provider_turn_lost_on_restore`), sent no continuation, and needed
Retry.
> - This pull request makes the reconciler record that evidence durably
when it claims the run, and makes the executor read it from there.
> - The benefit is that #20645 works on the real path. The proof rule is
unchanged.

## Linked Issues or Issue Description

Refs #20645

**What happened?**
Wave `w-20261110T212307-s9107` (L7, seed 9107, local, master plus
#20645), task codex B5 TM2:
- `native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`;
- `retryable_failure` with `recoveryMode: "exact_checkpoint_resume"`,
`recoverable`, `providerSessionEstablished` and `checkpointExists` all
set;
- `native.local_process_stopped`;
- attempt 2 started;
- `turn.failed` with `provider_turn_lost_on_restore` for the same turn
ID that the checkpoint was running.

No continuation was sent. The run failed with
`native_provider_terminal_failed` and was preserved with
`uncertain_provider_action`.

**Expected behavior**
The proven scheduled resume sends the single continuation.

**Steps to reproduce**
The new integration test below. The live path is timing-dependent (the
L7 command in #20645).

## What Changed

- `native-finalization-reconciler.ts`: when it claims a same-run resume,
the reconciler appends one `same_run_resume` entry to
`recovery_history`. The append is bounded like other history writes, and
it runs in the same update that clears the failure. The entry copies
`recoveryMode`, `recoverable`, `providerSessionEstablished`,
`checkpointExists`, `successorRunId` and `replacementDenied` from the
saved failure, with the controller boot and provider attempt.
- `scheduled-resume-continuation.ts`: the evidence for the third proof
now comes from either of two durable records:
- the saved failure, when the executor claims a `retryable_failure`
coordinator directly (unchanged); or
- a reconciler claim: an `observed` coordinator whose last history entry
is a `same_run_resume` entry naming the executor's own controller boot.
The two other proofs are unchanged: the stop receipt, and an admitted
attempt from 1 to 3.
- `native-session-executor.ts`: passes the coordinator's last history
entry and the controller boot to the decision.
- `doc/execution-semantics.md`: describes the recorded evidence.

Readers of `recovery_history` that look at the last entry do so only for
restart claims, Stop recovery and shutdown suspension. Restart
recovery's previous-claim lookup skips non-claim dispositions, so a
`same_run_resume` entry does not change those paths.

**Still refused** (tested):
- every #20645 case;
- the last history entry is not the resume claim, or there is no
history;
- the resume claim was recorded by another boot;
- the claimed evidence is a bootstrap retry, has no checkpoint, or
records a replacement;
- a reconciler-claimed resume without a stop receipt.

## Verification

- `native-scheduled-resume-continuation.integration.test.ts` now also
runs through the real reconciler claim (`claimNativeSessionResumptions`)
before `executePaperclipNativeSession`. On master the reconciler-claimed
allowed case fails (`resumeInterruptedTurn` is false). With this change
it is true, and the claim admits attempt 2. A reconciler-claimed resume
without a stop receipt still passes false.
- `scheduled-resume-continuation.test.ts`: decision cases for the
reconciler-claim entry.
- `native-session-resumption.test.ts` passes. 54 tests pass across the
three files.
- `cd server && npx tsc --noEmit` passes.
- Live: not yet. The failing wave above is the reproduction. A rerun is
timing-dependent.

## Risks

- One more `recovery_history` entry per scheduled resume, inside the
existing 20-entry bound.
- The proof rule itself is unchanged. This only reads the same evidence
from the record where it now survives.
- No schema change. A run claimed by an earlier build has no
`same_run_resume` entry, so it keeps the existing failure: refused, with
Retry offered as today.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 18:21:25 -06:00
DottaandClaude Opus 5.5 48c6e996da fix(native): continue a task whose quarantined completed turn's holds settle (#20662) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, workspace preparation attaches the
verified saved work of earlier runs of the same task.
> - Preparation failed with `remote_workspace_recovery_receipt_invalid`
when any earlier run had a recovery receipt that it could not verify.
> - A refusal cannot change the saved state, so every Retry failed in
the same way, and the task stayed blocked. Recoveries must fall back
automatically.
> - This pull request skips an unverifiable source. It attaches nothing
unverified and starts from the workspace's newest verified save.
> - The benefit is that Retry and continuations start again, while
strict receipt checks still decide what is attached, and no saved work
is changed.

## Linked Issues or Issue Description

Refs #20631

**What happened?**
On staging, after the deploy that included #20631, a user pressed Retry
on a task whose previous run failed with
`native_controller_recovery_missing_evidence:runner_not_settled`. The
new run failed in setup in under a minute with `setup_failed` /
`remote_workspace_recovery_receipt_invalid` (`executionRecovery:
bootstrap`, `providerWorkStarted: false`). The task stayed blocked.
Another Retry would fail the same way. #20631 accepted one shape of this
refusal (a source that saved only the snapshot its own restore loaded).
This task had a different shape.

**Expected behavior**
A Retry starts. A source whose receipt cannot be verified is not
attached, and its saved state stays intact for workspace recovery.

**Steps to reproduce**
1. Let a remote native run fail before provider work and leave a
recovery receipt or restore hold that does not verify (for example a
receipt on an unsaved reference, a receipt from another epoch, or a
saved reference without a receipt).
2. Press Retry on the task.
3. See the new run fail in setup with
`remote_workspace_recovery_receipt_invalid`.

**Paperclip version or commit**
Staging deploy that includes `e3bc4ca798`. The gap remains on `master`
at `a3b34a9d88`.

## What Changed

- `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an
unverifiable source. Each of the five refusal shapes now skips the
source and logs `remote_workspace_recovery_receipt_invalid` with a
reason: `receipt_without_reference`, `reference_unreadable`,
`receipt_on_unsaved_reference`, `receipt_unverified`, and
`receipt_snapshot_missing` (was
`remote_workspace_recovery_snapshot_missing`).
- `retainRemoteRecoveryAttachments` no longer throws for a retained
recovered copy that it cannot verify. It does not publish that copy
again. The files stay in the workspace.
- The writer check (`remote_workspace_recovery_writer_changed`), the
workspace access check, and snapshot read and digest failures still
refuse.
- `doc/execution-semantics.md` describes the fallback rule.

## Verification

- Red on unfixed code:
`remote-workspace-recovery-receipt-fallback.test.ts` (moved out of
`workspace-recovery-preparation.test.ts` to keep that file under 1,000
lines) "starts a Retry from the newest verified save when a predecessor
has %s" (6 shapes: `receipt_without_reference`,
`unreadable_reference_with_hold`, `receipt_on_prepared_reference`,
`superseded_snapshot_with_hold`, `receipt_from_other_epoch`,
`receipt_for_other_run`). All 6 failed with
`remote_workspace_recovery_receipt_invalid`. The source is a native run
that failed with `runner_not_settled`, and the writer is a Retry
(`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry
query.
- Green: the same 6 tests pass. They also assert that nothing is
attached, and that the source run row, its workspace operations, and its
snapshot row are unchanged.
- Updated safety tests: the canonical-attachment fault matrix now
expects no attachment for receipt faults, and still expects a refusal
for a missing or corrupt archive and for a wrong company or task. The
retained-history test expects an unverifiable retained copy to be left
out of the published list, while verified copies stay published.
- `npx vitest run
src/services/remote-workspace-recovery-receipt-fallback.test.ts
src/services/workspace-recovery-preparation.test.ts`: 91 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Not verified live on Daytona. I could not read the staging rows for
this task: the browser had no free tab. The tests cover every refusal
shape of this function, so the exact staging shape is one of them.

## Risks

- Previous-release state: the rule reads the same rows as before (run
reference, result hold, workspace operations, recovery actions,
snapshots). Rows that earlier releases saved carry forward. No
migration. A blocked task recovers on its next Retry or wake.
- A Retry can now start without files from an unverifiable source. Those
files are not lost: the source's snapshot, restore hold, receipts and
recovery action stay unchanged, and workspace recovery still owns them.
The skip is logged with a named reason.
- A retained copy that fails verification is dropped from the published
list. Its directory stays in the workspace and is no longer retired
automatically.
- A missing or corrupt snapshot archive still refuses. A permanently
lost archive therefore still blocks attach. This PR does not change
that.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 18:06:41 -06:00
DottaandClaude Opus 5.5 8f6ca25976 fix(workspaces): skip unverifiable recovery sources instead of failing Retry (#20666) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Before a remote run starts, workspace preparation attaches the
verified saved work of earlier runs of the same task.
> - Preparation failed with `remote_workspace_recovery_receipt_invalid`
when any earlier run had a recovery receipt that it could not verify.
> - A refusal cannot change the saved state, so every Retry failed in
the same way, and the task stayed blocked. Recoveries must fall back
automatically.
> - This pull request skips an unverifiable source. It attaches nothing
unverified and starts from the workspace's newest verified save.
> - The benefit is that Retry and continuations start again, while
strict receipt checks still decide what is attached, and no saved work
is changed.

## Linked Issues or Issue Description

Refs #20631

**What happened?**
On staging, after the deploy that included #20631, a user pressed Retry
on a task whose previous run failed with
`native_controller_recovery_missing_evidence:runner_not_settled`. The
new run failed in setup in under a minute with `setup_failed` /
`remote_workspace_recovery_receipt_invalid` (`executionRecovery:
bootstrap`, `providerWorkStarted: false`). The task stayed blocked.
Another Retry would fail the same way. #20631 accepted one shape of this
refusal (a source that saved only the snapshot its own restore loaded).
This task had a different shape.

**Expected behavior**
A Retry starts. A source whose receipt cannot be verified is not
attached, and its saved state stays intact for workspace recovery.

**Steps to reproduce**
1. Let a remote native run fail before provider work and leave a
recovery receipt or restore hold that does not verify (for example a
receipt on an unsaved reference, a receipt from another epoch, or a
saved reference without a receipt).
2. Press Retry on the task.
3. See the new run fail in setup with
`remote_workspace_recovery_receipt_invalid`.

**Paperclip version or commit**
Staging deploy that includes `e3bc4ca798`. The gap remains on `master`
at `a3b34a9d88`.

## What Changed

- `attachPreservedRemoteWorkspaceSnapshots` no longer throws for an
unverifiable source. Each of the five refusal shapes now skips the
source and logs `remote_workspace_recovery_receipt_invalid` with a
reason: `receipt_without_reference`, `reference_unreadable`,
`receipt_on_unsaved_reference`, `receipt_unverified`, and
`receipt_snapshot_missing` (was
`remote_workspace_recovery_snapshot_missing`).
- `retainRemoteRecoveryAttachments` no longer throws for a retained
recovered copy that it cannot verify. It does not publish that copy
again. The files stay in the workspace.
- The writer check (`remote_workspace_recovery_writer_changed`), the
workspace access check, and snapshot read and digest failures still
refuse.
- `doc/execution-semantics.md` describes the fallback rule.

## Verification

- Red on unfixed code:
`remote-workspace-recovery-receipt-fallback.test.ts` (moved out of
`workspace-recovery-preparation.test.ts` to keep that file under 1,000
lines) "starts a Retry from the newest verified save when a predecessor
has %s" (6 shapes: `receipt_without_reference`,
`unreadable_reference_with_hold`, `receipt_on_prepared_reference`,
`superseded_snapshot_with_hold`, `receipt_from_other_epoch`,
`receipt_for_other_run`). All 6 failed with
`remote_workspace_recovery_receipt_invalid`. The source is a native run
that failed with `runner_not_settled`, and the writer is a Retry
(`retryOfRunId`, `taskRetryRequestId`) that follows the real ancestry
query.
- Green: the same 6 tests pass. They also assert that nothing is
attached, and that the source run row, its workspace operations, and its
snapshot row are unchanged.
- Updated safety tests: the canonical-attachment fault matrix now
expects no attachment for receipt faults, and still expects a refusal
for a missing or corrupt archive and for a wrong company or task. The
retained-history test expects an unverifiable retained copy to be left
out of the published list, while verified copies stay published.
- `npx vitest run
src/services/remote-workspace-recovery-receipt-fallback.test.ts
src/services/workspace-recovery-preparation.test.ts`: 91 passed.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.
- Not verified live on Daytona. I could not read the staging rows for
this task: the browser had no free tab. The tests cover every refusal
shape of this function, so the exact staging shape is one of them.

## Risks

- Previous-release state: the rule reads the same rows as before (run
reference, result hold, workspace operations, recovery actions,
snapshots). Rows that earlier releases saved carry forward. No
migration. A blocked task recovers on its next Retry or wake.
- A Retry can now start without files from an unverifiable source. Those
files are not lost: the source's snapshot, restore hold, receipts and
recovery action stay unchanged, and workspace recovery still owns them.
The skip is logged with a named reason.
- A retained copy that fails verification is dropped from the published
list. Its directory stays in the workspace and is no longer retired
automatically.
- A missing or corrupt snapshot archive still refuses. A permanently
lost archive therefore still blocks attach. This PR does not change
that.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits). Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 17:47:54 -06:00
DottaandClaude 9296c74e8d fix(native): let a restart claim take back its lapsed lease at budget admission (#20650) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart, restart recovery claims an
interrupted native run (`resume_dead_runner`) and grants a 20-minute
lease. Budget admission then re-checks that the claim still owns the run
before the resumed run may start provider work.
> - A host sleep, or the restart-chaos fault TM2, can make that stored
lease look expired while its owner is alive. #20625 lets the owner take
back such a lease when the row still shows exactly its own claim. That
rule did not cover budget admission.
> - So a claim whose lease lapsed before dispatch was refused with
`Native recovery no longer owns provider dispatch`. The run failed with
`adapter_failed`, recovery held it with
`native_continuation_requires_reconciliation`, and the user had to click
Retry.
> - This pull request applies the #20625 rule in budget admission: the
claim takes its lease back through the existing re-acquire condition.
> - The benefit is that a run resumed after a restart survives a time
jump before dispatch with no Retry. Every fence and every budget check
stays the same.

## Linked Issues or Issue Description

Refs #20625, #20474

**What happened?**
Restart-chaos wave L7 (seed 9107, local) on a branch with #20625, task
codex B5 TM2. A TM3 restart claimed the run (`resume_dead_runner`,
generation 2). TM2 then aged every database time by one hour before the
resume reached dispatch. Budget admission
(`lockRunBudgetRecoveryOwnerInTransaction`) required a live lease and
threw `Native recovery no longer owns provider dispatch`. The run failed
with `adapter_failed`, recovery stopped with
`native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`, and the task needed Retry.

**Expected behavior**
The claim's own lapsed lease is taken back, and the resume is admitted
when the budget allows it.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent).

## What Changed

- `native-execution-lease.ts`: two new helpers. Both use the #20625
own-claim rule. Neither takes a lock out of order.
- `readOwnNativeRecoveryClaimLease` (read-only) returns `live`,
`lapsed_own_claim` or `lost`. It compares the lease against the database
clock. A lapsed lease counts as `lapsed_own_claim` only when all of
these hold:
    - the same lease owner;
    - this process's controller boot, pid and process start;
    - a non-null `recovery_state`;
- `nativeExecutionLeaseHeldCondition({ remote: false,
controllerGeneration })`, which applies the #20625 own-claim rule and
every #20474 untouched check;
    - no non-local environment lease on the run.
- `extendOwnNativeRecoveryClaimInTransaction` extends such a lease by
the local execution TTL, with one conditional UPDATE on the same
predicate.
- `budget-reservations.ts`:
- `lockRunBudgetRecoveryOwnerInTransaction` still refuses another owner,
and refuses a lapsed lease unless it is the run's own untouched claim.
It no longer locks the run row, so the documented coordinator → issue →
run order holds.
- The lease is extended only at the end of an admitted reservation
(`reserveRunBudget`) or an admitted dispatch recheck
(`recheckRunBudgetReservationInTransaction`). At those points the caller
already holds the run row lock. A refused admission, for example one
over budget, leaves the lease lapsed.
- `heartbeat-native-restart-suspension.test.ts`: the test "still settles
a recovery whose claim merely expired" encoded "expired means not the
owner". #20474 and #20625 changed that rule for an untouched own claim.
The test now has two cases:
- an expired claim that another actor recorded on (`awaiting_evidence`)
still settles with no provider session;
  - an untouched expired claim is dispatched.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new user of the rule.

## Verification

- New
`server/src/__tests__/native-recovery-dispatch-lease-reacquire.test.ts`:
- `reserveRunBudget` takes back the claim's own lapsed lease and admits
the resume. This fails before the change.
  - An over-budget resume is still refused, and its lease stays lapsed.
- Fenced variants each throw `Native recovery no longer owns provider
dispatch` and leave the row unchanged: a graceful-shutdown suspension,
an appended failure settlement, `awaiting_evidence`, `blocked`, a claim
from an earlier generation, another boot, a remote sandbox lease, and
another lease owner.
- At the dispatch gate (`recheckRunBudgetReservationInTransaction`), an
admitted own claim gets its lease extended, and a fenced claim is
refused with the row unchanged.
- Related suites pass (308 tests):
`native-execution-lease-reacquire.test.ts`,
`closed-native-receipt-accounting.test.ts`,
`heartbeat-native-predispatch-retry.test.ts`,
`run-dispatch/adapters/postgres.test.ts`,
`heartbeat-native-restart-suspension.test.ts` and
`heartbeat-restart-claim-lapse.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: one L7 wave (seed 9107) ran on the first commit of this branch
(`w-20261110T212307-s9107`). It logged 0 `Native recovery no longer owns
provider dispatch` refusals, against 1 in the previous wave. Codex B5
TM2 got past dispatch, but it still needed Retry for another reason:
after a `runner.drain` timeout, the scheduled same-run resume did not
send the continuation of #20645. That case is tracked separately. The
other five tasks passed.

## Risks

- This widens when a restart claim can take back its own lapsed lease,
and only at budget admission. It is local only. It needs the same owner,
boot and generation, and the row must show exactly that claim. A claim
that any other actor touched still fails as before.
- Budget hard-stop behaviour is unchanged. The lease is extended only
after budget admission passes.
- No schema change and no migration. Rows saved by earlier releases
carry the same history shape.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 17:43:46 -06:00
DottaandPaperclip 7f72ea7534 fix(links): retain PR status across page reloads (#20654) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task messages contain GitHub pull request links.
> - A hover loads PR status into a shared memory cache.
> - A page reload clears that memory and returns each link to an unknown
state.
> - The browser already saves selected reads for each user and company.
> - This pull request adds learned link previews to that cache.
> - Reloads show the saved status. Only user interest can refresh stale
previews.

## Linked Issues or Issue Description

**What happened?**

After a user loaded PR status, a page reload removed the learned state.
The browser persistence rules excluded link-preview queries. Related
implementation: #20593. Searches of open and closed PRs found no
duplicate reload-cache fix.

**Expected behavior**

Keep learned PR status across reloads for the same account and company.
Do not load every PR on page render. Reuse fresh results and refresh
stale results on hover or keyboard focus.

**Steps to reproduce**

1. Open a task with repeated GitHub PR links.
2. Hover one link and wait for its status.
3. Reload the page.
4. Before this fix, both links return to unknown until another hover.

**Paperclip version or commit**

Reproduced against master source at
`30403781b9418355c78dc0ed978366af77036e76`.

## What Changed

- Add link-preview queries to the existing user and company persistence
rules.
- Keep at most 100 previews across accessible companies. Use the
existing 24-hour retention, build boundary, and sign-out cleanup.
- Keep the original read time and freshness when restoring previews.
Other query types keep their existing restore behavior.
- Add regressions for saved status, negative results, bounds, age,
account changes, membership loss, and sign-out.
- Add a production-component Storybook example and browser reload
checks. Update the cache documentation.
- Use one query-key definition for the component, persistence rules, and
cache tests.

The browser read-cache module owns persistence and account cleanup. This
change extends that owner. The link service and provider API contracts
stay the same.

## Verification

- Red: the new reload and retention tests failed because a fresh client
had no saved PR data.
- Green: 102 focused UI tests passed across the cache, sign-out,
account-query, and query-client suites.
- UI typecheck passed.
- Storybook catalog generation and the static Storybook build passed.
- All 21 browser checks passed with retries disabled. They prove status
and negative-cache restoration after real reloads, no eager requests, no
fresh-hover refetch, and one stale-interest refresh shared by duplicate
links. Existing light/dark and narrow PR states, keyboard access, exact
URLs, and card positioning also pass.
- The first browser attempt could not launch because this session lacked
Chromium. Chromium was installed before the complete passing run; that
first attempt supplies no test verdict.
- `pnpm verify --explain` returned exit 2. The browser test path
requires broad CI coverage. No full local suite is claimed.
- The supplied staging page was not accessible through the web tool. No
deployed acceptance is claimed.
- CI on the first head failed in a Slack long-publication timeout and a
font-loading-frame browser check. Both files are outside this change.
The two publication cases passed when run locally in isolation. The font
case also passed locally against the built UI. [Final-head
CI](https://github.com/paperclipai/paperclip/actions/runs/42909433167)
passes on `e19d5b8bb1a910b1cbb9bb9a62ab87cdf684737f`. Typecheck, build,
tests, E2E, and Rust checks pass. There are 58 successful check runs,
three expected skips, and one neutral Paperclip Review check. The legacy
Snyk status passes.

Both Greptile and the supplemental Paperclip review report 5/5 on the
final head. No actionable findings or unresolved review threads remain.
The Paperclip Review check itself has a neutral conclusion.

## Risks

- PR display data now remains in browser local storage until expiry,
build change, or sign-out. Each query requires the matching user and an
accessible company.
- A stale saved status can remain visible for up to 24 hours. Hover or
keyboard focus refreshes it after the provider freshness window.
- Storage restrictions retain the existing memory-only fallback. No
schema migration is required.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, repository tools, code
execution, and browser automation. The runtime does not expose the exact
serving model ID or context-window size. No subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 17:43:06 -06:00
Devin FoleydevinfoleyClaude Opus 5.5
 		7b33384446 docs(release): stable notes for the 2026.1110.0-beta.0 soak (v2026.1113.0) (#20610) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release process promotes a soaked beta to stable. The stable
GitHub Release body comes from `releases/beta/v<beta-version>.md` on
`master`
> - Beta `2026.1110.0-beta.0` published in workflow run 42763789753. Its
`draft_stable_notes` job stopped at the 10-minute job timeout while it
made the commit-log skeleton for 1,586 commits, so it did not push the
skeleton branch
> - Without a notes file on `master`, the stable promotion for this beta
has no notes to publish
> - This pull request adds the curated stable notes on the branch name
that the job would have used. Each claim traces to a pull request in
`v2026.1106.0..09e24dd`
> - The benefit is that the earliest stable promotion (v2026.1113.0)
finds finished notes on `master` on time

## Linked Issues or Issue Description

**Issue type**

Missing content

**Where is the issue?**

`releases/beta/v2026.1110.0-beta.0.md` is the stable-notes file for the
`2026.1110.0-beta.0` soak. It does not exist yet, because the
`draft_stable_notes` job in run 42763789753 timed out before it pushed
the skeleton.

**What's wrong?**

The stable promotion reads this file from `master` and publishes it as
the GitHub Release body. Until a notes file merges, the stable preflight
has no notes to resolve.

**Suggested fix**

Merge the curated notes so the notes invariant holds for the
v2026.1113.0 promotion. Correct the `> Released:` date in a follow-up if
the promotion date slips.

## What Changed

- Added `releases/beta/v2026.1110.0-beta.0.md` with curated stable notes
for v2026.1113.0 in the same layout as `releases/v2026.1106.0.md`:
overview, Breaking Changes, Highlights, Improvements, Fixes, Upgrade
Guide, Contributors
- The notes are delta-framed against `releases/v2026.1106.0.md`.
Features that stable already describes are not repeated
- Breaking Changes lists the Google Workspace connector gated off by
default (#19247), one execution workspace per subtask (#19860),
agent-created task trees limited to parent and child (#19312), native
agents reporting status through `paperclip_finish`/`paperclip_block`
(#18411, #18489), warm-by-default local Codex and OpenCode Runner agents
with migration `0443` (#19888), skill packages moving out of PostgreSQL
with this release as the rollback floor (#18082), Enter to submit
(#19272), and the narrower `pi_local` environment (#14503)
- Upgrade Guide enumerates migrations `0375` through `0451` with
one-phrase effects, and the new environment variables with defaults
- Release and CI internals, smoke specs, canary tooling, Paperclip Cloud
work, dependency bumps, and test-only changes are left out

## Verification

- Beta publish is complete: npm dist-tag `beta` is `2026.1110.0-beta.0`,
tag `beta/v2026.1110.0-beta.0` points at
`09e24dd12c36251dd12af2d73dbc715ce990f922`, `publish_beta` and both
`smoke_beta` jobs in run 42763789753 succeeded, and the beta Docker
build (run 42858086733) succeeded
- Stable `v2026.1106.0` is an ancestor of `09e24dd`; `git rev-list
--count v2026.1106.0..09e24dd12c36251dd12af2d73dbc715ce990f922` returns
1586, which matches the overview and Contributors section
- Contributors are the 37 unique GitHub logins of the commit authors in
the range, read from the GitHub API with `[bot]` accounts excluded.
External handles are thanked by name
- `./scripts/release.sh stable --date 2026-11-13 --print-version`
returns `2026.1113.0`
- Every `#NNNN` link in the file traces to a commit in
`v2026.1106.0..09e24dd`
- Migration numbers, setting keys, env var names, and defaults were
checked in the code at the source commit, not in commit subjects
- The file contains no internal ticket ids or instance-local links
- The compare view for this branch shows one added file and no source
changes

## Risks

- Low risk: a single markdown file, no source changes. If the promotion
date slips past 2026-11-13, the `> Released:` line and H1 need a
one-line update before the stable dispatch
- The `draft_stable_notes` job force-pushes this branch name. A rerun of
that job would replace this commit with the skeleton. Do not rerun it
for this beta
- The notes assume the next stable is promoted from this beta. If a
later beta is promoted instead, its own notes file supersedes this one

## Model Used

- Claude (Anthropic), model ID `claude-opus-5-5` (Claude Opus 5.5), tool
use via Claude Code

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: devinfoley <139239+devinfoley@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:41:13 -08:00
Devin FoleyandPaperclip bbf5cfeb62 fix(agents): return to the index after termination is accepted (#20667) 
Return to the Agents index when termination is accepted, including while asynchronous cleanup continues, so the profile does not remain on an Agent not found error.

Cover both action menus, roster refresh, failed requests, and unsaved changes during pending termination.

Verified with 53 focused tests, the full UI suite (11,949 passed; one skipped), UI typecheck and build, all applicable CI checks, and Greptile 5/5 with no unresolved comments.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
canary/v2026.1111.0-canary.0 		2026-11-10 15:38:00 -08:00
DottaandPaperclip de7ceac0de fix: preserve compatible historical Codex WEB evidence (#20665) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Native session migration preserves local evidence in encrypted
archives.
> - Each WEB receipt binds the exact provider directory and exclusion
policy.
> - The Codex credential refresh change added two excluded files and
changed that policy hash.
> - A new reader then rejected old receipts even when their facts,
files, skills, and authority still matched.
> - This pull request accepts the exact prior Codex policy while
checking archive paths against the current credential policy.
> - Operators can keep valid saved evidence across the upgrade without
copying it again.

## Linked Issues or Issue Description

Refs: #19814, #20646.

**What happened?**

A reader upgrade rejected saved Codex WEB evidence because the profile
hash changed when `auth-refresh.json` and `auth-refresh.json.tmp` became
excluded credentials. The other receipt bindings still matched. This
blocked the native harness migration gate.

**Expected behavior**

The exact known prior profile remains readable when the current archive
path checks accept every saved entry. An archive that includes either
newly excluded credential file must still fail. Unknown profiles and
changed authority must fail.

**Steps to reproduce**

1. Preserve WEB evidence with the frozen prior Codex profile and no
credential refresh files.
2. Read its unchanged receipt with the current profile.
3. Observe that the old reader returns false solely because the profile
hash changed.

**Paperclip version or commit**

Reproduced against `a3b34a9d88`; final branch is based on `e9343d8501`.

## What Changed

- Accept the exact prior Codex profile hash only when the complete
current profile has the reviewed successor hash and the provider is
Codex.
- Keep the current entry and excluded-path checks. Keep facts,
controller, archive, skill, and ownership checks.
- Add a frozen prior policy fixture with its literal hash. Test old
receipt reads, both credential files, unknown policies, changed
authority, and current writes.

## Verification

- Real PostgreSQL baseline: the frozen old-policy receipt test failed
because the reader returned false. The unchanged fixture passes with
this fix.
- All 18 selected PostgreSQL cases pass on final head after the master
rebase: six new cases and twelve existing WEB archive and authority
guards (14.93 seconds).
- Full layered server typecheck passes on final head after the rebase.
The module boundary check passes.
- A single read-only native-only candidate check on the deployed
dependency graph validated all 54 retained scopes. The 29 prior failures
passed, and the other 25 stayed valid. The check made no storage calls
or data writes. It did not replace final deployed enrollment acceptance.
- Final head `14112d0e3abb6e318a86fe9799d95bddd2dbf76b` passes all
[required CI
checks](https://github.com/paperclipai/paperclip/actions/runs/42919867108):
55 successful checks, four expected skips, and one neutral review. The
legacy security status also passes. Exact-head Greptile review is 5/5
with no open review threads.

## Risks

The compatibility branch accepts only the two reviewed policy hashes. A
future policy change can require another explicit compatibility
decision. The reader still rejects saved credential refresh files. It
does not rewrite receipts or change execution authority. No stored
schema or operator command changes.

## Model Used

OpenAI Codex, GPT-6 family. The exact serving model ID and
context-window capacity are not exposed in this session. Used reasoning,
source editing, shell execution, and tool calls.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 17:22:08 -06:00
DottaandPaperclip aa2a7be8b4 fix: recognize provider-owned Runner workspace storage (#20663) 
## Thinking Path

> - Paperclip manages agent work and preserves its execution state.
> - A server replacement must retain every workspace that depends on the
old server disk.
> - Runner task environments can own their workspace outside the Core
server.
> - These environments save an adapter-managed workspace with a
runnerTaskWorkspaceOwner binding.
> - Readiness and maintenance treated that provider path as a Core
filesystem dependency.
> - This change validates the saved owner and shares that classification
between both paths.
> - Local, malformed, and foreign workspace bindings still block
enrollment.

## Linked Issues or Issue Description

Related enrollment safeguards: #18835. A GitHub search found no
duplicate fix for this provider-owned class.

**What happened?**

Volume-free readiness returned workspaces_require_migration for an
adapter-managed Runner workspace owned by a plugin task environment.
Maintenance also selected it and failed with
workspace_provider_unsupported. The provider owned the path, which did
not exist on the Core server.

**Expected behavior**

A valid provider-owned workspace does not require migration from the
Core server disk. Ordinary local workspaces and invalid ownership
bindings remain blockers.

**Steps to reproduce**

1. Run a paperclip_runner agent in a plugin task environment.
2. Retain its adapter-managed workspace and complete
runnerTaskWorkspaceOwner binding.
3. Read startup eligibility for volume-free enrollment.
4. Observe the workspace blocker before this fix.
5. Run workspace maintenance and observe workspace_provider_unsupported
for the same row.

**Paperclip version or commit**

Reproduced on 353945c86d. The regression
also fails on the branch base a3b34a9d88.

**Deployment mode**

Docker with external PostgreSQL. Local regression coverage uses
disposable PostgreSQL.

## What Changed

- Share one metadata predicate between startup eligibility and offline
workspace migration selection.
- Validate a complete saved owner against a same-company
paperclip_runner agent and a plugin environment with matching nonempty
keys. Per-run environment overrides remain supported.
- Keep ordinary local workspaces and invalid ownership bindings in the
migration set.
- Add real PostgreSQL coverage for provider-owned readiness, zero-import
maintenance selection, unchanged retained rows, and
local/foreign/malformed ownership controls.

## Verification

- Readiness RED: the provider-owned fixture failed with
workspaces_require_migration on unchanged source.
- Maintenance RED: the provider-owned fixture failed with
workspace_provider_unsupported before the shared selector was applied.
- GREEN: 29 selected tests passed across three files. This covers 8
eligibility/configuration tests and 21 migration/offline-admission
cases, including eleven invalid/local owner controls inside the
eligibility test. The affected migration file has 207 cases; broad
coverage remains with required CI.
- Commands: `pnpm exec vitest run
server/src/__tests__/ephemeral-server-data.test.ts
server/src/__tests__/ephemeral-server-contract.test.ts` plus focused
selections in `server/src/services/remote-workspace-migration.test.ts`
for provider ownership, offline admission, local imports,
interrupted-batch resume, and unresolved workspace semantics.
- Server layered typecheck passed after the final source change.
- Final head `2fb98dbc664da14e6f60d812e1e10877b2b7f00a`: [required
CI](https://github.com/paperclipai/paperclip/actions/runs/42920262773)
completed with 55 successful checks, 1 neutral review, and 4 intended
skips. Greptile and Paperclip Review both rated this head 5/5. No
unresolved review threads remain.
- No hosted data migration or metadata rewrite was performed.

## Risks

- This changes a readiness classification. Incomplete, foreign,
non-Runner, or mismatched environment bindings remain conservative
blockers.
- The check validates saved ownership through current database bindings.
It does not inspect provider files or grant execution authority.
- Existing saved rows retain their shape. No migration or archive format
changes are required.

## Model Used

OpenAI Codex, GPT-6 family, with reasoning, repository tools, code
editing, and real PostgreSQL tests. The host does not expose the exact
model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 17:22:01 -06:00
Nicky Leach 0d4b1d8c89 Gate agent plugin tabs by execution environment (#20657) 2026-11-10 15:04:44 -08:00
Devin FoleyandPaperclip e9343d8501 fix(heartbeat): release task Retries held by a drain as soon as it ends (#20640) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a person clicks Retry on a stopped task, Paperclip saves the
request as a held wake and dispatches it right away, unless admission is
held. Admission is held during a task drain, for example while a
deployment restarts the server
> - A Retry held by a drain is released later by the deferred-receipt
worker. That worker shares one queue with every other held wake: comment
waits, execution waits, and interrupted queues
> - On a busy instance that queue holds hundreds of passive waits that
come due every 30 seconds. A Retry is checked in due-time order behind
all of them. Ending a drain does not wake the worker either
> - In a real case, a Retry clicked during a deployment drain was still
held 40 minutes later, after the drain expired and the server restarted.
It ran only when the person clicked Retry again
> - This pull request checks operator Retries ahead of passive waits,
and wakes the worker as soon as a drain stops or expires
> - The benefit is that a Retry held by a drain starts within one worker
pass after the drain ends

## Linked Issues or Issue Description

**What happened?**

A person clicked Retry on a blocked task 14 seconds after a deployment's
task drain started. The Retry was saved as `deferred_issue_execution`,
and the immediate dispatch returned early because of the drain. The
drain had a 20-minute TTL, and the deployment restarted the server about
6 minutes in. The held Retry still did not start, and its `updated_at`
never changed. Forty minutes later the person clicked Retry again, and
the new request started a run and merged with the held one.

That instance had 390 held wakes: 271 execution waits and 116 others,
against 6 task Retries. The execution waits come due every 30 seconds.
Measured on the instance, the deferred-receipt worker took about 7
minutes to get through that backlog once.

**Expected behavior**

A Retry that a drain held starts soon after the drain stops or expires,
or after the server restarts, whatever the size of the passive backlog.

**Steps to reproduce**

1. Start a non-idle task drain (`POST /instance/task-drain`).
2. Click Retry on a stopped task. The receipt is
`deferred_issue_execution`.
3. Stop the drain. On `master`, the deferred-receipt worker only notices
on its next 30-second re-check. It then checks the Retry in due-time
order behind every passive wait that came due before it.

**Paperclip version or commit**

`master` at 8148fc0b1e.

## What Changed

- `server/src/services/scoped-deadline-work.ts`:
- New optional `priority(entry)`. Due entries are sorted by priority
first, then by due time and id as before.
- New `refresh(hint)`. It clears a scope's reconstruction delay and
invalidates it. Failure backoff is kept.
- New optional `onStop` hook, called when the coordinator stops the
worker.
- `server/src/services/heartbeat/deferred-receipt-work.ts`:
- A `task_retry` receipt that a person requested is an operator Retry,
with priority 1. Passive receipts keep priority 0, so their order does
not change.
- Each deferred scan loads operator Retries first, with their own query,
then pages by id as before. A Retry is therefore in memory in the first
pass even beyond the first 100-row page. A receipt loaded twice merges
by id.
- New optional `owner.subscribeRefresh`. The work calls `refresh()` on
all three of its queues when it fires, and releases the subscription
from `onStop`, so the coordinator's own stop removes it.
- `server/src/services/heartbeat.ts`: passes `subscribeRefresh` that
listens to task-drain changes and expiry. Before this change, a deploy
drain left the worker waiting for its 30-second re-check, behind the
scan's reconstruction delay.
- `server/src/__tests__/deferred-execution-receipt-work.test.ts`, four
new tests:
- A real Retry, requested during a deploy drain, starts a run (to
`succeeded`) within 5 seconds after the drain stops.
- An operator Retry whose id sorts after 120 passive execution waits
(more than one load page) is resumed first, in the first worker pass.
- A held Retry inserted with no notification right after a full scan is
resumed as soon as the drain refresh fires.
  - Stopping only the coordinator removes the drain listener.

## Verification

- `npx vitest run src/__tests__/deferred-execution-receipt-work.test.ts
-t "held by a deploy drain|ahead of a backlog|ordinary committed task
Retry"`: 3 pass. The third is the existing ordinary Retry dispatch test.
- Red checks, removing one piece at a time:
- Without the drain subscription, the drain test fails: the Retry is
still `deferred_issue_execution` with no run after 5 seconds.
- Without the priority, or without the prioritized load, the backlog
test fails: the first resumed request is the passive `00000000-…` wake.
- With `invalidate()` instead of `refresh()`, the post-scan test fails
after 5 seconds.
  - Without the `onStop` release, the listener test fails.
- Full `deferred-execution-receipt-work.test.ts`: 50 pass.
- `chat-channels.integration.test.ts -t "still batches several replies
in one finding thread"`: it timed out once on CI and passes 2 of 2 times
locally with this change.
- `scoped-deadline-work`, `heartbeat-task-drain`,
`heartbeat-task-drain-admission-release`, `cloud-control-task-drain`,
`task-admission-deadline`, `onboarding-first-task-admission` and
`delivery-work-coordinator` suites: 164 pass.
- `pnpm run typecheck` in `server` (layered): clean.

## Risks

- Operator Retries are a small set; there were 6 on the instance above.
They now take the first slots of a worker pass, so passive waits can
move by at most that many places. Passive waits keep their own order.
- A Retry that keeps failing its check still gets the existing failure
backoff, so it cannot take every pass.
- A drain start also wakes the worker. That pass ends at once, because
the worker's run gate still reads the drain.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, vitest runs against embedded PostgreSQL, read-only
production log and database analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 15:00:50 -08:00
DottaandClaude Opus 5.5 686893636d fix(recovery): release a lapsed owner of a finished coordinator so stop-proof holds re-arm (#20661) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Workspace recovery saves the files of a run whose copy-back failed,
and it holds the shared workspace until the save is done
> - #20295 re-arms a save that was exhausted with
`agent_stop_unconfirmed`, so the run's stop is proven again and the
files are saved without a person
> - On staging, four holds of this shape were never re-armed. Their runs
ended six days ago, their sandbox leases are released with a provider
stop receipt, and the servers that ran them were replaced many times.
Their tasks stay blocked, and so do the parent tasks that wait for them
> - The finished coordinator of each run still names a lease owner whose
lease lapsed. Discovery treats that owner as "waiting for the
abandoned-owner settlement", but that settlement handles only observed
(unfinished) coordinators. Nothing ever releases the owner, so the
worker never selects the hold, the re-arm refuses it, and the attempt's
stop check refuses it too
> - This pull request makes a lapsed owner of a finished coordinator
count as stopped. The re-arm releases it in the same transaction, and
then the new attempt proves the sandbox stop and saves the files as
#20295 designed
> - The benefit is that these holds settle on the worker's next pass
after deploy, without a migration and without an administrator

## Linked Issues or Issue Description

Refs #20295

**What happened?**
An exhausted workspace save
(`native_workspace_sync_out_retry_exhausted`, last failure
`agent_stop_unconfirmed`) stayed blocked for six days. The task showed
"Paperclip tries again automatically", but no re-arm was ever recorded
(no `stopProofRecovery` marker, no `issue.workspace_recovery_rearmed`
activity), and the shared workspace stayed held by the failed run.

**What did you expect to happen?**
The recovery worker re-arms the save once the run is provably over,
saves the dirty files, and frees the workspace so the task and its
parent can continue.

**Steps to reproduce**
1. Exhaust a workspace save with `agent_stop_unconfirmed` (one attempt,
window spent).
2. Leave a lease owner on the run's `terminal_failure` coordinator with
an expired lease (or no expiry), as a finalizer pass of a replaced
server does.
3. Run the workspace recovery worker. `nextWorkspaceRecoveryAt` returns
null, discovery does not select the hold, and `rearmExhaustedStopProof`
refuses it.

## What Changed

- `workspace-recovery-stop-proof.ts`: add
`lapsedFinishedCoordinatorOwner` and `releaseLapsedCoordinatorOwner`. A
lease owner on a finished coordinator (committed, terminal failure,
applied) whose lease lapsed, or that has no expiry, does not block
`sourceRunProvablyOver`. The stop-proof re-arm clears that owner with a
compare-and-swap in its locked transaction, and records
`releasedLapsedOwner` in the marker and the activity.
- `workspace-recovery-capture-policy.ts`: the capture-policy re-arm uses
the same rule, because it shares the same check and discovery clause.
- `workspace-recovery-fallback-discovery.ts`: the discovery clause still
waits for the abandoned-owner settlement for a stale owner of an
unfinished coordinator, but it no longer excludes a hold whose finished
coordinator has a lapsed owner. A live lease still defers the re-arm to
its expiry.
- `system-recovery-state.ts`: add `FINISHED_COORDINATOR_PHASES`.
- `doc/execution-semantics.md`: document the rule.
- Tests: two red-then-green regressions in the staging shape (an expired
finalizer lease, and an owner without an expiry saved by an older
release), one strictness case (a live lease on a finished coordinator
keeps the hold and is not released), and the existing stale-owner case
now uses an observed coordinator, which still waits for the
abandoned-owner settlement.

## Verification

- Red before the fix: `pnpm --filter @paperclipai/server exec vitest run
src/services/workspace-recovery-stop-proof.test.ts` failed the two new
staging-shape cases (`nextWorkspaceRecoveryAt` returned null: the worker
was never woken for the hold).
- Green after the fix: `workspace-recovery-stop-proof.test.ts` and
`workspace-recovery-capture-policy.test.ts` pass (15 tests). The
staging-shape cases assert that the hold is re-armed, the save
completes, the dirty files are saved, the workspace hold is free, and
the lapsed owner is released.
- `pnpm verify`: exit 2 (broader verification required, because
`doc/execution-semantics.md` is a global CI input). Focused suites pass:
`workspace-recovery-stop-proof`, `workspace-recovery-capture-policy`,
`workspace-recovery-fallback`, `workspace-recovery-retry-copyback` (113
tests). Required CI supplies the broad coverage.
- Staging evidence (read-only GET requests): the run is `failed`,
finished six days ago, `controllerLeaseExpiresAt` is null, its only
lease is `released` with `cleanupStatus: success`, a `stopped`
termination receipt and a stop-and-retain receipt. The action's evidence
has no re-arm marker, and four holds of the same shape were not touched
after #20295 was deployed.

## Risks

- Previous-release story: rows saved by earlier releases carry the
lapsed owner and no marker. They are selected and re-armed on the
worker's first pass after deploy. No migration.
- Strictness stays where an owner could be live: a live lease on any
coordinator defers the re-arm to its expiry, and a stale owner of an
unfinished (observed) coordinator still waits for the abandoned-owner
settlement, which proves its controller stopped. The new attempt still
requires a provider stop receipt for the allocation before it reads
files.
- Clearing a lapsed owner matches the existing token-lease contract: an
expired finalizer or cleanup lease is free for the next claim, and a
holder's later write fails its token compare-and-swap.
- Low risk otherwise: the change is limited to the two exhausted-hold
re-arms and their discovery clause.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use (shell,
code editing, browser GET reads) in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 17:00:46 -06:00
DottaandPaperclip a3b34a9d88 fix(ui): allow queued message deletion before queue acknowledgement (#20553) 
## Thinking Path

Paperclip lets a user queue a message while an agent works. The user
must be able to remove that message before delivery starts. A new local
message can clear the displayed queue ID until the server accepts it.
This also disables trash for older saved messages. The page rejects the
action before it sends a request.

This change keeps trash available when the server says the user can
discard the message. When the queue ID is absent, the page uses the
existing comment cancellation route. The server finds the queue and
checks the author and dispatch state in the same transaction.

## Linked Issues or Issue Description

**What happened?**

Trash stays disabled for a saved queued message while another message
waits for queue acknowledgement. The page handler also throws before it
sends a cancellation request.

**Expected behavior**

Trash removes the selected queued message before dispatch. A message
that has started dispatch stays visible with an error. Other queued
messages stay queued.

**Steps to reproduce**

1. Send a follow-up while an agent works.
2. Send another follow-up before the queue read includes it.
3. Try to discard the first saved message.

**Paperclip version or commit**

Reproduced from source at `0e3fdebea825`.

**Deployment mode**

Source checkout on Linux. UI tests use jsdom. Route tests use isolated
embedded PostgreSQL.

Related work: #15865 changes mixed-queue interrupt handling. #15939
changes recovery after a queued message is discarded. #15465 changes
discard permissions for agent-authored entries. This change fixes trash
for a user's own saved messages while queue acknowledgement is pending.

## What Changed

- Keep trash enabled for a saved message with discard permission when
the displayed queue ID is absent.
- Use the existing author-only cancellation route when queue identity is
pending. Keep the revision-checked route when queue identity is present.
- Keep failed cancellations visible. Refresh the queue after
cancellation and remove successful deletions from comment caches.
- Cover deletion, dispatch conflicts, permission failures, network
failures, history reloads, final-message cleanup, and concurrent
dispatch.

## Verification

- Red: three regression cases failed on the original production code.
The trash button was disabled. Both page cases made no cancellation
request.
- Green: five complete UI suites pass, with 767 tests and one existing
skipped test. These cover the page, queue component, thread, queue
merge, and optimistic comments.
- Green: 11 focused PostgreSQL route cases pass. They cover both routes,
queue promotion, final-message cleanup, authorship, dispatch conflicts,
and a concurrent dispatch claim.
- `pnpm check:token-gates` and `git diff --check` pass.
- `pnpm --filter @paperclipai/ui typecheck` passes.
- The GitHub tree is `d4f0d61cbef2620c1fd62e6988ad80ccb1ba6173`. It is
identical to the tested local tree after merging master `902f3767960d`.
- `pnpm verify` selected more than 1,500 server test files. That broad
local run was stopped without a pass claim. CI will supply broad build
and test coverage. The preflight helper could not build Rust because
Cargo is absent.
- [CI
Build](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129897150252)
and [browser
E2E](https://github.com/paperclipai/paperclip/actions/runs/42835584447/job/129901676175)
pass for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`.
- [Greptile
review](https://github.com/paperclipai/paperclip/pull/20553#issuecomment-6770202777)
gives the current head 5/5 with zero comments and no unresolved review
threads.
- [Final CI
run](https://github.com/paperclipai/paperclip/actions/runs/42835584447)
passes for head `44dea22faea51a6664a6f45852a42f62f5c72bbd`: 55 check
runs passed and 4 were skipped. No failures or pending checks remain.
Typecheck, Build, all general test shards, serialized suites, Runner
checks, and browser E2E passed. Snyk also passed.

## Risks

The fallback has no client queue revision. The existing server
transaction finds the queue for the selected comment and checks
ownership and dispatch state. Tests cover rejection after dispatch and
when another user owns the message. No API contract or schema changes
are required. No visual tokens change. Existing user documentation
remains accurate.

## Model Used

OpenAI GPT-6 in Codex. The runtime does not expose a more specific
provider revision or context window size. Used reasoning, repository
tools, code edits, and test execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 16:38:58 -06:00
Nicky LeachandPaperclip f30ac226d6 Keep project environment mappings independent of shared defaults (#20651) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A project supplies work context and repository storage to task runs.
> - Creation selected a storage owner but did not retain the selected
environment on the project.
> - Configuration stored an environment inside checkout policy, but
policy reads and run selection dropped it.
> - Projects therefore continued to follow shared agent and instance
defaults.
> - This pull request saves placement on each project and uses it for
new runs.
> - The benefit is an independent Project to Environment association
with the existing storage, trust, and recovery safeguards.

## Linked Issues or Issue Description

Refs #20459 and #11644. Follow-up to #20587.

The reported journey is Project > Configuration: changing one project's
environment must not change another project's mapping or shared
defaults.

Related open PR #18176 repairs placement through the legacy
workspace-policy field. This follow-up introduces a separate project
association, saves creation defaults, and migrates saved legacy
references. It overlaps that PR's placement, permission, and recovery
safeguards. The changes should be reviewed together. #9588 concerns
environment-variable response privacy and does not address placement.
ROADMAP.md has no separate project-association work planned.

## What Changed

- Add nullable `projects.environment_id` and expose
`Project.environmentId`. Save the effective creation default once on the
project. Keep environment placement separate from checkout/isolation
policy.
- Read and write Environment directly in Project Configuration. Keep the
control available with isolated workspaces disabled. Each form reads its
project's value, even when environment choices share a query cache.
- Offer environments compatible with the current storage owner. Change
the target and storage together when the existing storage selector
attaches a core project to a plugin. Preserve retained-data restrictions
and validate the observed storage owner again on save.
- Resolve new task runs from their organizational project's target
before agent and instance defaults. File sources from another project do
not change placement. Explicit low-trust sandbox choices and enforced
sandbox policies retain precedence. External provider placement remains
separate.
- Restore admitted native runs from their owned environment lease. Later
project edits apply to new admissions. Require workspace-write
permission for environment changes and clears. Check company bindings
before runtime admission and during lease creation.
- Migrate valid saved policy references into the project column.
Normalize legacy API writes. Count and clear matching project references
when an environment is deleted. Update API and behavior documentation.
- Refresh the maintained SDK source registry for the two changed shared
files. Runtime pins are unchanged.

## Verification

- [Hosted
CI](https://github.com/paperclipai/paperclip/actions/runs/42908045146)
is green on `8b3b7d5b5aed51fc4c8e9c2e3506508bbaf9e00c`: **60 successful
checks**, three skipped gates, no failures, and `ci / verify` passed.
GitHub reports no merge conflicts. Standard Greptile scored the same
commit **5/5** with both review threads resolved.
- Source checkout on macOS arm64, Node 26.12.0, with fresh isolated
PostgreSQL. No production data or live provider calls.
- The two-project component test changes the first form with isolation
disabled and a shared query cache. The second form keeps its target.
Storage changes update only the selected project. A saved target remains
selected while the list loads, and an explicit clear remains possible.
- API integration creates two projects from the same default, patches
one, reads both, and checks that agent and instance defaults remain
unchanged. A later instance-default change does not rewrite either saved
project target. Tests cover unavailable/incompatible choices, anonymous
and cross-company denials, and storage preconditions.
- The migration test runs the actual SQL against an existing project row
and verifies its environment and disabled checkout policy survive.
Deletion accounting and clearing affect only matching project
references.
- The final guard set passes **53 tests across four files**:
`project-environments.test.ts`,
`environment-selection-route-guards.test.ts`,
`heartbeat/environment-selection.test.ts`, and
`ProjectProperties.environment.test.tsx`. All **64
environment-run-orchestrator tests** pass, including direct lease
acquisition without an extra caller argument. The broader affected set
passed **109 tests across eight files** before the additional admission
guards. These sets overlap.
- UI typecheck, migration numbering/safety, token gates, module
boundaries, query-error rendering, and diff checks pass. `pnpm verify
--explain` returns exit 2 and requires broad CI because shared
schema/runtime paths and behavior documentation changed. Broad coverage
is delegated to current-head CI.
- Early iterations failed due to a legacy parser that drops the
environment field, a fixture without real company membership, and
incorrect expectations for privacy-preserving 404 denials. The service
now reads legacy references before normalization, the API fixture has
real membership, and denial assertions match the existing privacy
contract. The orchestrator mock also needed the reused company-binding
guard in its service contract. The focused suites above pass after those
corrections.
- Local server typecheck remains blocked by installed ACPX, Slack, AI
SDK, and Runner type mismatches in unchanged dependency-facing code. It
reports no errors in the changed production paths. This is not a full
local typecheck or suite pass. Hosted CI supplies full current-head
coverage.
- SDK source-proof and privacy qualification passes **56 tests across
three files**, including the real built SDK. The maintained source
registry matches the committed revision; runtime pins are unchanged.
- Review reproduced two failures: an incoming policy without a storage
owner could be validated before the existing owner was restored, and
uppercase legacy UUIDs missed migration matching. Preserve the owner
before validation and compare normalized UUID text. Both regressions now
pass. The post-fix project/storage/orchestrator batch passes **107
tests**: `node node_modules/vitest/vitest.mjs run
server/src/__tests__/project-environments.test.ts
server/src/__tests__/project-workspace-provider.integration.test.ts
server/src/__tests__/environment-run-orchestrator.test.ts --silent`.
- The first hosted run failed build, typecheck, and canary at the same
`run-completion.test.ts` call because lease acquisition required a new
argument. The lease method now enforces binding internally. Its existing
caller compiles without a new field, and the orchestrator tests prove
the binding flag reaches the runtime. The full server check no longer
reports that error; the local dependency errors above remain.

## Risks

- Apply the normal database migration before serving the new revision.
The migration adds a nullable foreign key and copies valid legacy
project selections. It does not move repository data or resize provider
storage. The five newest schema snapshots are retained.
- New projects retain their creation target when shared defaults change.
Existing projects without a saved selection continue to inherit
defaults. A null target explicitly opts into inheritance.
- Existing saved project selections now affect placement. Missing,
archived, incompatible, and foreign-company targets fail validation;
they do not silently choose another storage owner. A running admitted
process keeps its original owned lease.
- Local dependency limitations are listed above. No live-provider,
deployed-browser, or production rollout qualification is claimed.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, tool use, and code
execution. The exact serving snapshot and context-window size are not
exposed in this session.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 14:31:48 -08:00
DottaandClaude Opus 5.5 358977454b test(native): cover unknown-outcome recovery and hold settlement without a recorded PID space (#20649) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - #20628 settles a host-tool callback that a server restart orphaned,
and settles the no-replay holds that earlier releases left for a
completed turn
> - Review of #20628 asked for two more cases. That PR merged before the
commit with those cases reached it
> - This pull request adds only those tests
> - The benefit is that the two edge rules of #20628 stay covered

## Linked Issues or Issue Description

Refs #20628

**What happened?**
The review of #20628 named two untested rules. First, a recovery
authority that throws `semantic_tool_outcome_unknown` must settle only
for a callback from an earlier controller, and must rethrow for this
controller's own callback. Second, a coordinator row saved before
`controller_host_hash` existed must still settle its holds through the
stopped controller boot.

**Expected behavior**
Both rules have regression tests.

**Steps to reproduce**
1. Read `recoverToolCallback` and `native_completed_turn_cleanup_hold`
on master.
2. Search the tests for a thrown `semantic_tool_outcome_unknown`, or for
a row with a null `controllerHostHash`. Neither exists.

**Paperclip version or commit**
master at 2150820bd8.

## What Changed

- `durable-prp-control-plane.test.ts`: an authority that throws
`semantic_tool_outcome_unknown` settles as
`tool_call_interrupted_by_restart` for `earlier_controller`, and
rethrows for `this_controller`. Any other rejection rethrows.
- `completed-turn-cleanup-hold-settle.test.ts`: a coordinator row
without `controllerHostHash`, whose server boot is recorded stopped in
`controller_instances`, settles the run's holds through the
controller-instance proof. No attestation is needed.

## Verification

- `npx vitest run src/control-plane/durable-prp-control-plane.test.ts -t
"orphaned by a server restart"` in `packages/paperclip-runner`: 7
passed.
- `npx vitest run
src/__tests__/completed-turn-cleanup-hold-settle.test.ts` in `server`: 5
passed.
- `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` and
`tsc --noEmit -p server/tsconfig.json` pass.

## Risks

- Low risk. Test-only change. No production code changes.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
canary/v2026.1110.0-canary.18 		2026-11-10 15:42:00 -06:00
DottaandClaude 30403781b9 fix(native): keep crediting proven unstarted restarts past the generation limit (#20647) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run survives a control-plane restart through restart
recovery, which resumes the interrupted provider turn
> - Restart recovery charges a provider attempt only when the provider
did work that could be lost, and it stops after three charged attempts
> - From controller generation 12 on, every restart charged an attempt
before that work rule was checked
> - In a busy restart wave, one long task sees 12 to 15 restarts, most
caused by other tasks. A restart that hit before the provider accepted a
turn was charged, the budget ran out, and a person had to click Retry
> - This pull request keeps the work rule past generation 12 for
restarts that a durable record proves did not start the provider, and
bounds a crash loop with a count of consecutive restarts without
provider progress
> - The benefit is that a long task survives many unrelated restarts
without a manual Retry, while a run that never makes progress still
stops

## Linked Issues or Issue Description

Refs #20625 (same restart recovery path). No public issue exists for
this case.

**What happened?**

`restartResumeKeepsAttempt` returned "charge" for every restart once the
run reached `NATIVE_PROGRESS_CREDIT_GENERATION_LIMIT` (12) controller
generations. It did this before it read the work-based outcome. In a
local restart chaos wave (L6, seed 9106, task t03, Claude B5 with RR1)
the run reached generation 13. The last restart interrupted a resume
whose runner started but whose provider had not accepted a turn. That
restart was charged at the last attempt, so the claim failed with
`execution_recovery_budget_exhausted` and
`native_restart_recovery_blocked`. The task needed a manual Retry.

**Expected behavior**

A restart that a durable record proves did not start provider work does
not spend an attempt, also past generation 12. A crash loop that never
makes provider progress still stops automatically. When it stops, the
task shows a Retry that works.

**Steps to reproduce**

Run a native task past controller generation 12. Make the last restart
hit after `native.process_start_requested` but before `turn.accepted`,
with the provider attempt at 3. The new integration test `does not
charge a restart past the generation limit whose resume never started
the provider` reproduces the failure before the fix.

**Paperclip version or commit**

`master` at ed5e7493bb.

## What Changed

- `restartResumeKeepsAttempt` now returns `{ keepsAttempt,
noProgressRestarts }`. A new helper judges the interrupted attempt as
`provider_not_started`, `provider_progressed` or `charged`, with the
same proofs as before.
- `noProgressRestarts` is 0 when the interrupted attempt is proven to
have made provider progress. Otherwise it is the previous claim's count
plus one. A clean-drain claim carries the previous count.
- Before generation 12 the decision does not change.
- From generation 12 on, a restart keeps the attempt only when the
interrupted attempt is proven `provider_not_started` and the count is
below the new `NATIVE_NO_PROGRESS_RESTART_LIMIT` (12). Provider work
(progress, or an accepted turn without progress) and unprovable cases
still charge.
- Each restart claim writes `noProgressRestarts` into its
`recovery_history` entry. No schema change: `recovery_history` is JSONB.
- Claims saved by earlier releases have no count. They are read as zero.
- Integration tests: the three generation-cap remote tests now check the
new bound. New tests cover the unstarted restart past the limit, the
consecutive-limit stop with a working Retry, the progress reset, an
accepted turn without progress, an unrecorded previous claim, and an
old-format claim.
- The durable-ownership plan (section F) records the rule.

**Why 12.** It equals the existing generation limit
(`PROVIDER_ATTEMPT_CREDIT_GENERATION_LIMIT`). The count includes
restarts before generation 12. Thus a run that never makes provider
progress is bounded exactly as before: the existing test `bounds a crash
loop that never starts the provider` passes unchanged. Only proven
progress resets the count. In the failing wave, the task had made
progress shortly before its last restart, so its count was 1.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/native-runner-restart-recovery.integration.test.ts
src/services/native-runtime/native-restart-recovery.test.ts
src/__tests__/heartbeat-native-restart-suspension.test.ts`
- Red before the fix: with only the new constant added to the old code,
the 9 new and changed tests fail. With the fix, they pass. The existing
crash-loop, progress and unstarted-resume tests pass without change.
- The consecutive-limit test checks that the stopped run is
`terminal_failure` / `native_restart_recovery_blocked` with no lease
owner and nothing scheduled, that `taskInterruptionPresentation` shows
`failed` with `canRetry: true`, and that `requestTaskRetry` accepts the
Retry.
- `cd server && npx tsc --noEmit -p .`
- Full local run of the three suites above: 322 passed, 14 skipped.
- Live: local restart chaos wave L6 (seed 9106, concurrency 3, `--env
local --adapters codex,claude --workloads A1,B5,C1 --faults RR1,RR2,RR3
--tasks 6`) at 7f7803ab0c: 6 of 6 tasks passed, and no task needed
Retry. Task t03 (Claude B5 RR1) reached controller generation 13 again.
The claim at generation 13 resumed the run (`resume_dead_runner`), and
the server log has no `execution_recovery_budget_exhausted`. The same
task on master was blocked at that claim and needed Retry.

## Risks

- A run that makes provider progress between restarts past generation 12
still charges each of those restarts. This is the same as before, by
design.
- An upgrade or a mid-run deploy reads old claims as count zero. A run
that was already in a no-progress loop can get up to 12 more
proven-unstarted restarts before the bound applies. It cannot be charged
where the old rule did not charge.
- A rollback ignores the new field and restores the old rule.
- Past generation 12 the claim now reads the event log, as it already
does before generation 12. This is not on the turn-start path.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, through Claude Code
with tool use (shell, file edits, local test runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:27:45 -06:00
DottaandClaude Opus 5.5 4f33bb3578 fix(workspaces): release the Blocked status of a sibling-settled no-replay hold (#20648) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a remote workspace save fails, recovery actions hold the task
until the run's files are saved
> - #20641 lets a resolved no-replay hold settle when a sibling action
of the same run saved the files
> - Review found that the settlement did not release the task's Blocked
status, and that a settled hold with a historical source stayed
selectable
> - A new test with the waiting user comment also showed that the worker
skipped the second hold of the same run in every pass, so it never
settled
> - #20641 was merged before these fixes were pushed
> - This pull request completes #20641 with those fixes
> - The benefit is that the task leaves Blocked and continues without a
person

## Linked Issues or Issue Description

Refs #20641, Refs #20569

**What happened?**
#20641 merged at its first head. The review follow-ups for it were
pushed later and are not on master:
- A settled no-replay hold that owned the task's Blocked status
(`nativeFailureBlock`) left the task `blocked`.
- A hold settled by a sibling stayed selectable through
`workspaceRecoverySourceEligible` when it named a
`historicalWorkspaceSource`.
- With saved user input waiting, the export hold stayed selectable and
came first in each worker pass. `reconcileRetainedWorkspaces` skipped
the other hold of the same run, so the no-replay hold never settled.

**Expected behavior**
After the save, the no-replay hold settles in the same or next pass. The
task returns to `todo` when that hold owns its Blocked status. The
settled hold is never selected again.

**Steps to reproduce**
1. Use the staging-shape test in
`workspace-recovery-capture-policy.test.ts`, with a waiting "try again"
comment.
2. Run the worker until nothing is due.
3. On master, the worker never goes idle and the no-replay hold stays
`resolved`/`blocked`.

## What Changed

- `workspace-recovery-fallback.ts`: new shared `holdOwnsBlockedStatus`,
used by `continueFromLastSavedWorkspace` and
`settleSiblingRecoveredHold`. A settled no-replay hold that owns the
Blocked status returns the task to `todo`. A newer decision's Blocked
status is kept. An export hold's status stays with its own result
finalization.
- `workspace-recovery-source.ts`: `workspaceRecoverySourceEligible`
excludes a resolved hold that carries `siblingWorkspaceRecovery`.
- `workspace-recovery.ts`: `reconcileRetainedWorkspaces` runs several
holds of one run one after another in that run's job, instead of
skipping the later ones.
- `workspace-recovery-capture-policy.test.ts`: the staging test now has
the waiting comment, and a variant with `nativeFailureBlock` and
`historicalWorkspaceSource`. It checks `todo`, no re-selection, and no
repeat handoff in a later pass.
- `doc/execution-semantics.md`: documents these rules.

## Verification

- Red, with each rule disabled: the status stayed `blocked`; the settled
hold was selected again; the worker never went idle (starvation).
- Green: `npx vitest run --root server` on 14 workspace recovery suites
(including `workspace-recovery-capture-policy`, `-fallback`,
`-admission`, `-source`, `execution-blocker-read`,
`preserved-native-response-wait.integration`): all passed.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.

## Risks

- Several holds of one run now run in sequence in one job. Concurrency
per run is unchanged (one job per run).
- A no-replay hold releases the Blocked status only when it still owns
it. Otherwise the status is kept.
- Previous-release story: no new persisted state. Holds saved by earlier
releases are handled on the first worker pass after deploy.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:24:04 -06:00
DottaandClaude 805531cc71 fix(native): never leave a cleanup-gated task without Retry or an owner (#20639) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs close their provider session at the end of each attempt.
When close cannot prove that the runner stopped, the runner keeps a
cleanup gate (`native_session_cleanup_quarantined`) in server memory
> - #20572 hid Retry for every gated run and said "Paperclip finishes
the cleanup automatically". Only a remote environment release settles
the gate. A local gate that needs an operator never settles, and a
server restart loses it
> - A restart-chaos wave on master found two tasks that rested blocked
with no Retry, no scheduled retry and no recovery owner. A person could
not continue them
> - This pull request makes the run's own recovery retire the gate when
the host proves the old processes stopped. It also offers Retry whenever
nothing settles the gate, and it makes that Retry work
> - The benefit is that a cleanup-gated task either continues by itself
or shows a Retry that starts. It never rests without both

## Linked Issues or Issue Description

Refs #20572, #20578

**What happened?**
Restart-chaos wave `w-20261110T185144-s9107` (local, claude, fault TM2:
the server is paused with SIGSTOP for about 33 s). Tasks t04 and t05
completed their provider turn. The runner checkpoint owner lapsed while
the server was paused (`native_runner_checkpoint_owner_lost`). The
session close then failed with `native_session_close_unrecoverable:
runner did not durably suspend before checkpoint`. The provider had
drained and no tool call was pending. The runner kept an
operator-required cleanup gate. Thirty seconds later the run's own
same-run recovery verified that the runner process and group had stopped
(`native.local_process_stopped`). The gate still refused the recovery,
and the run ended `failed` with `native_session_cleanup_quarantined`.
The task showed `canRetry: false` and a cleanup wait until the deadline.
Nothing owned that wait.

**Expected behavior**
When the old runner and provider processes are proven stopped, the run's
own recovery continues. When nothing settles the gate, the task offers a
Retry that starts.

**Steps to reproduce**
1. Run restart-chaos L7: `--env local --adapters codex,claude
--workloads A1,B5,C1 --faults TM1,TM2,TM3 --time-age-hours 1
--concurrency 3 --tasks 6 --seed 9107`.
2. Observe a claude TM2 task rest `blocked` with `canRetry=false`,
`retryWait=true` and no scheduled retry.

**Paperclip version or commit**
master at ed3ffc3bc7.

## What Changed

- `native-session-executor.ts`: add `provenStoppedOwnNativeSession`. It
returns proof only for a local run with a pid-bound
`native.local_process_stopped` record (no later launch) and with every
recorded provider process gone. An ACPX session open names no provider
PID; its sidecar counts only when it shares the runner's process group,
which the stop record proves empty.
- `native-session-executor.ts`: add
`retireStoppedOwnSessionCleanupGate`. With that proof, it retires the
run's own operator-required gate through the existing
`completeTerminatedLocalNativeSessionCleanup` boundary and records
`native.session_cleanup_gate_retired`.
- `heartbeat.ts`: the native recovery preamble calls it right after it
records the local process stop, before admission. The recovery decision
itself does not change.
- `explicit-native-continuation.ts`: Retry or a new message accepts the
run's own stop proof when the retained-state inventory proof does not
apply. It still starts a fresh session, and it never resumes the old
turn.
- `task-interruption-presentation.ts`: hide Retry for a gated run only
while a live owner settles the gate: an unreleased remote environment,
or a scheduled close in this process. Otherwise offer Retry. The same
rule applies to a Retry held on `cleanup_quarantined`.
- `doc/execution-semantics.md`: describe the new rule.

## Verification

- Red before the fix, green after (5 tests failed on master sources):
- `server/src/services/task-interruption-presentation.test.ts`: "offers
Retry when nothing settles the gate by itself", "holds a Retry on a
cleanup gate only while an owner settles it", and an invariant test: a
failed presentation without Retry must name a wait, and a cleanup wait
needs a settling owner.
- `server/src/__tests__/task-interruption-query.test.ts` (real
Postgres): an unreleased remote lease hides Retry; the released lease
offers it.
- `server/src/services/quarantined-task-retry.test.ts`: "starts a task
Retry for a run's own quarantine proven stopped by its own process
record" (stopped, and provider alive stays held).
- New: "retires a run's own failed-close gate once its runner and
providers are proven stopped" for codex PIDs, an ACPX sidecar in the
runner's group, a sidecar with its own group, a live provider, and no
stop record.
- `npx vitest run` of `quarantined-task-retry`,
`task-interruption-presentation`, `task-interruption-query` and
`explicit-native-continuation`: 575 passed.
- `npx tsc --noEmit -p server` passes.
- Live: LIVE_PROOF_PLACEHOLDER

## Risks

- The gate retirement removes process ownership only. It runs only for a
local run whose exact runner and group and every recorded provider
process are proven gone. A provider without a recorded process identity
keeps the gate.
- Retry on a gated run still runs every existing stop check. If the stop
cannot be proven, the Retry waits and the task keeps a Retry, instead of
a wait that nobody settles.
- No persisted shape changes. Runs saved by earlier releases use the
same fields.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:20:42 -06:00
DottaandClaude 37c50f70c2 fix(native): continue a turn lost on restore in a proven same-run resume (#20645) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run keeps its provider conversation when its local runner
dies. When the restored runner finds that the active turn was lost, it
reports `provider_turn_lost_on_restore` and does not invent a result.
> - After a control-plane restart, the admitted restart attempt may send
one continuation in that same conversation (#15297). The agent then
reconciles unfinished work. The task is not resent.
> - A runner can also be lost without a server restart, for example when
`runner.drain` times out. The run then resumes itself after the retry
delay from its exact checkpoint. That path never allowed the
continuation, because the executor passed `resumeInterruptedTurn` only
for a `resume_dead_runner` restart claim.
> - So the lost turn failed with `native_provider_terminal_failed`,
automatic replacement was refused (`uncertain_provider_action`), and the
user had to click Retry.
> - This pull request allows the same single continuation for that
resume, but only when durable records prove the same three facts that a
restart claim proves.
> - The benefit is that a runner lost mid-turn continues in the same
conversation with no Retry. No case that is refused today becomes
allowed without that proof.

## Linked Issues or Issue Description

Refs #15297, #20625

**What happened?**
Restart-chaos wave L7 (seed 9107, local), task codex B5 TM2, run on
master after #20625. A TM3 restart resumed the run (attempt 1). During
the TM2 freeze the TS runtime lost its runner:
`native_session_recovery_failed: runner_local_connect_failed: PRP
command runner.drain timed out`. The executor saved `retryable_failure`
with `recoveryMode: "exact_checkpoint_resume"`. After the delay,
recovery verified the runner process stopped
(`native.local_process_stopped`). The same run then resumed as attempt 2
(`native.process_start_requested`, `providerAttempt: 2`). The restored
runner reported `turn.failed` with `provider_turn_lost_on_restore`
(`recoverable: true`). No continuation was sent. The run failed with
`native_provider_terminal_failed`, recovery stopped with
`uncertain_provider_action` / `preserve_without_replay_v1`, and the task
needed Retry.

**Expected behavior**
The run sends the one same-conversation continuation, exactly as it does
after a restart claim, and finishes without Retry.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`
(timing-dependent: the drain timeout must land during an active turn).

## What Changed

- New
`server/src/services/native-runtime/scheduled-resume-continuation.ts`.
It decides whether a same-run scheduled resume may continue a lost turn.
Each of the three proofs comes from a durable record:
1. **The previous runner is stopped.** The latest process record of the
run in `heartbeat_run_events` (`native.process_start_requested`,
`native.process_identity_recorded`, `native.local_process_stopped`) must
be `native.local_process_stopped`. Its `processPid` must equal its
`processGroupId`, and neither the pid nor the group may be alive now.
This is the existing `readNativeLocalProcessStop` check, read in the
claim transaction before this attempt records its own
`native.process_start_requested`.
2. **The attempt was admitted under the bounded budget.** The executor's
claim moves `native_run_finalizations` from phase `retryable_failure` to
`observed`. It charges a provider attempt, and the claim refuses any
attempt above 3 (`native_session_retry_exhausted`). The decision also
requires `1 <= attempt <= NATIVE_PROVIDER_RECOVERY_ATTEMPT_LIMIT`.
3. **It is the same provider conversation.** The saved `failure_detail`
must show `recoveryMode: "exact_checkpoint_resume"`, `recoverable:
true`, `providerSessionEstablished: true` and `checkpointExists: true`,
with no `successorRunId` or `replacementDenied`. At run time the
existing runtime guards still apply: the backend must recover the
checkpointed session (`recovered`, not a continuity-break replacement),
and the lost turn must equal the checkpoint's `activeTurnId` and the
snapshot's interrupted turn. The provider kinds stay codex, acpx and
opencode, with no compaction, goal control, held result or pending
runtime request.
- `native-session-executor.ts`: computes that decision in the
coordinator claim. It passes `resumeInterruptedTurn` when either the
restart claim is `resume_dead_runner` (unchanged) or the decision allows
it.
- `doc/execution-semantics.md`: describes the rule and the refused
cases.

**Cases that stay refused** (each has a test in
`scheduled-resume-continuation.test.ts`):
- a restart-recovery claim (it keeps its own existing rule);
- a remote execution target;
- a coordinator that is not `retryable_failure`, including a fresh run;
- `recoveryMode` `bootstrap_retry` or `ambiguous_state`, or no saved
failure detail;
- a failure that is not recoverable;
- no established provider session;
- no checkpoint;
- a recorded replacement (`successorRunId` or `replacementDenied`);
- an attempt outside 1..3;
- no stop receipt, a stop receipt superseded by a newer launch, a
stopped process that is still alive, or a run that never had a process.

The runtime-side refusals are unchanged and already covered by
`native-session-runtime.test.ts`: real provider errors, historical
error-null failures, and no restart permission.

## Verification

- `cd server && npx vitest run
src/services/native-runtime/scheduled-resume-continuation.test.ts`: 22
tests, covering the decision table and the stop receipt in embedded
Postgres.
- `cd server && npx tsc --noEmit` passes.
- Live: not proven. An L7 wave on this branch (seed 9107) did not
reproduce the drain-timeout path, so this pull request has unit and
database tests only. In that wave the same cell (codex B5 TM2) needed
Retry for a different reason: TM2 aged its restart-recovery claim lease
before dispatch, and budget admission refused with `Native recovery no
longer owns provider dispatch`. This pull request does not change that
path. Four other tasks passed. Claude C1 TM2 rested blocked without a
Retry button, which is a separate cleanup-quarantine issue.

## Risks

- This allows one more path to send the same-conversation continuation.
It is limited to local runs. It needs the same proofs as a restart
claim, and it uses the attempt the resume already charged. It does not
change the generation cap or `restartResumeKeepsAttempt`.
- A rolling deploy reads `failure_detail` and stop events written by
earlier releases. Those fields and events already have this shape, so an
older saved failure is judged by the same rule.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 15:16:51 -06:00
DottaandClaude Opus 5.5 353945c86d fix(tool-gateway): let agents create gateway tokens without receiving the value (#20613) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Tool gateways let external MCP clients use a company's tools with a
gateway client token
> - Since delegated connection management (#17737), an agent can create
and revoke gateway client tokens for its responsible person
> - #20581 stopped agents from getting the one-time token value, but it
did this by making token creation person-only
> - The product decision is that agents keep their delegated authority
to create and revoke tokens
> - This pull request restores agent token creation, but the agent never
receives the token value: the person reveals the value once on the
gateway's Tokens page
> - The benefit is that an agent can do the rotation work for the
person, and the secret still never enters chat, run events, logs or tool
output

## Linked Issues or Issue Description

Refs #17737
Refs #20581

**What happened**
After #20581, `POST /api/tool-gateway/gateways/{gatewayId}/tokens`
returns 403 `gateway_token_requires_person` to every agent caller.
Agents lost the delegated authority to create gateway client tokens that
#17737 gave them. Before #20581, an agent that created a token received
the plaintext value in its tool result and could repeat it in chat.

**Expected behavior**
An agent with delegated connection management can create and revoke
gateway client tokens. The agent never receives a token value. The
responsible person gets the value once, on the gateway's Tokens page.
Person callers keep today's behavior.

**Steps to reproduce**
1. Give an agent delegated connection management for a responsible
person with `tools:admin`.
2. As the agent, call `POST
/api/tool-gateway/gateways/{gatewayId}/tokens` (for example through
`call_api`).
3. See 403 `gateway_token_requires_person`.

## What Changed

- Agent create: `POST /api/tool-gateway/gateways/:gatewayId/tokens` from
an agent with delegated authority returns 201 with token metadata only
(`id`, `tokenPrefix`, `expiresAt`, `pendingReveal: true`), plus
`tokensPath`, a `reveal_credential` `requiredUserAction` for the
responsible person, and a `nextStep` instruction. The saved row has no
usable value: its stored hash is of a value that is generated and then
discarded. Gateway authentication also rejects `pendingReveal` rows.
- Person reveal: new person-only `POST
/api/tool-gateway/gateway-tokens/:tokenId/reveal`. It generates the
value, stores its hash, clears `pendingReveal` and returns the value
once. A conditional update makes a second or concurrent reveal return
409. Only the token's responsible person can reveal it (403
`gateway_token_reveal_requires_responsible_person`). Agent callers get
403 `gateway_token_requires_person`. A token revoked before reveal
cannot be revealed.
- Why a reveal that generates the value: no stored plaintext existed to
reveal, and storing one (even encrypted) for later display adds a secret
at rest. Generating the value at reveal time keeps the existing one-time
display and stores only a hash.
- Schema: `tool_mcp_gateway_tokens.pending_reveal boolean not null
default false` (migration `0460_gateway_token_pending_reveal`). Shared
`ToolMcpGatewayToken.pendingReveal`, and `ConnectionRequiredUserAction`
gets `reveal_credential` / `gateway_token`.
- UI: the Tokens page shows a "Waiting for you to reveal" section for
pending tokens with a "Reveal token" button. The value appears in the
existing "New token — copy now" one-time block. Pending tokens show the
status "Not revealed".
- Agent gateway create: the 201 for agent callers also includes
`gatewayPath`, `tokensPath` and a `nextStep` that tells the agent to
link the saved gateway.
- The agent create-token `nextStep` and the catalog say not to revoke
the old token in the same turn: its client keeps working on it until the
person installs the new value.
- The reveal update and its `tool_gateway.token_revealed` activity entry
commit in one transaction. The Tokens page offers Reveal only for tokens
that act for the signed-in person.
- API reference: new `docs/api/tool-gateway-tokens.md` (create, reveal,
revoke) in the docs nav.
- Revoke stays allowed for agents with delegated authority (unchanged
route).
- API catalog: create-gateway, create-token, reveal and revoke
descriptions now say who acts, that agents never receive values, the
page paths, and that the old token is revoked only after the client
works with the new one.
- Agent access map: create-token policy note and a new reveal entry.
- Evals `gateways-profiles.create-gateway` and
`gateways-profiles.rotate-token`:
- Remove the keyword grader (`human|operator|admin|manage`) and the
handoff text checks. Grade saved state, rendered links and secret
absence. A plain reply or a question card is accepted.
- Remove the setup probes that called the create routes as the agent.
Under delegated authority, those probes created a real gateway
("Forbidden agent gateway"), which failed create-gateway before the
agent ran.
- create-gateway: exactly one gateway for the intended profile and
company, created by the agent (or by the person after a handoff); the
person opens it from a rendered link; attribution matches the creator;
any agent-created token is still pending reveal; no token value in chat,
cards or run events.
- rotate-token: after the agent's turn, both setup tokens still work and
any replacement the agent created is pending the person's reveal; the
person follows the rendered Tokens page link and reveals (or issues) the
value in a private browser; the new value works before the old token is
revoked; the unrelated client keeps access; no token value appears
anywhere, including a regex scan for any full `pcgw_<id>.<secret>`
value.
- The changed scenario context and outcomes are recorded in
`chat-full-catalog-testops-manifest.json` `changedFields` and pinned in
the manifest test. The inventory digest, descriptors (`oracleVersion`
v2, `sourceFiles`) and SDK public input pins are updated. The unused
`external-gateway-handoff.ts` module is removed.
- Fixture realism: the create-gateway profile now allows a real read
tool (an empty profile made the agent ask which apps to add instead of
finishing), and the rotation tokens get a neutral owner note instead of
"Disposable rotation fixture" (the agent read that note as an eval
hint).
- Shared private-browser helpers (`external-private-browser.ts`,
`external-private-browser-worker.ts`) get one optional `pendingTokenId`
input and one `revealedPending` receipt field.

## Verification

- Red: on `master` route code, the new integration tests `lets agents
create and revoke gateway client tokens without ever receiving a token
value` and `reveals an agent-created gateway token value only to its
responsible person` fail (403 `gateway_token_requires_person`).
- Green: `cd server && npx vitest run
src/__tests__/connection-management.integration.test.ts -t gateway` (25
passed). The test checks that the agent `call_api` result has no `token`
field and no token-shaped value, that the agent cannot reveal, that the
person reveals once (second reveal 409) and the value authenticates,
that agent revoke works and stops the value, that a token revoked before
reveal never gets a value, and that person create still returns the
value.
- `npx vitest run src/__tests__/tool-gateway.test.ts
src/__tests__/openapi-routes.test.ts
src/services/native-runtime/runner-api.test.ts` (2247 passed).
- `cd ui && npx vitest run
src/pages/apps/gateways/panels/TokensPanel.test.tsx
src/pages/apps/gateways/gateway-helpers.test.ts` (16 passed); `pnpm
check:token-gates` clean.
- Eval graders and catalog: `pnpm exec vitest run --config
tests/runner-e2e/vitest.config.ts` on the gateway oracle tests,
`external.test.ts`, catalog consistency/contract, TestOps
export/retained, phase/clock callers and SDK input/type tests (all
pass); `node --test` on the TestOps manifest, adapter, contract,
streaming and reconcile tests (all pass); `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`
(unchanged).
- `node scripts/check-maps.mjs` passes. `tsc --noEmit` for `server`,
`ui` and `tests/runner-e2e`: no errors in changed files.
- Paid end-to-end run of both eval cases, run
[42872921890](https://github.com/paperclipai/paperclip/actions/runs/42872921890)
on 39f38a7f6c (before the last three commits): both FAILED, for
agent-behavior reasons the later commits address. Not rerun yet.
- rotate-token: the agent created the replacement (pending reveal, no
value), then revoked the old token in the same turn, despite the create
response saying to wait. Its reasoning cited the eval-looking fixture
note. The grader rejected the premature revocation. The reply linked the
Tokens page correctly.
- create-gateway: the agent created the gateway and one pending token,
then asked a question card about which apps to add to the empty profile,
with no link to the gateway. The grader rejected the missing link.
- Token values: a scan of all evidence files of both cells found no full
`pcgw_<id>.<secret>` value and no `pcgw_<uuid>` form. Agent `call_api`
payloads in run events are redacted; the integration test proves the
agent tool result has no value.

## Risks

- Behavior change for agents: token create returns metadata and a Tokens
page path, not a value. Agents that expect `token` in the response must
send the person to the Tokens page.
- Migration adds one boolean column with a default. Existing tokens get
`false` and keep working.
- A pending token that is never revealed stays as an unusable row until
someone revokes it. It cannot authenticate.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 15:13:34 -06:00
DottaandPaperclip 2150820bd8 perf(native): parallelize durability eligibility checks (#20646) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A web server must prove that retained run state is durable before
volume-free enrollment.
> - The proof reads selected native sessions and validates their
existing recovery receipts.
> - Serial database checks made a valid inventory take 59.665 seconds.
> - The signed preparation request has a 55-second limit.
> - This pull request overlaps independent reads within fixed limits.
> - The benefit is a faster eligibility check with the same preservation
rules.

## Linked Issues or Issue Description

Refs #20599 and #20601. A search found no duplicate of this remaining
serial-read bottleneck.

**What happened?**

An actual full startup eligibility check returned eligible with no
blockers in 59.665 seconds. Controller checks took 44.134 seconds and
harness checks took 8.592 seconds. Signed preparation timed out at
55.002 seconds.

**Expected behavior**

A durable instance can complete signed preparation within its existing
deadline.

**Steps to reproduce**

1. Retain many native task sessions with database-backed controller
checkpoints and valid filesystem-independence receipts.
2. Run the full startup eligibility check.
3. Observe serial checkpoint and receipt reads consume the signed
request budget.

## What Changed

- Check the existing terminal-controller metadata and
checkpoint-existence predicate within the existing eight-reader source
batch.
- Validate harness independence receipts with at most four concurrent
readers per existing page.
- Stop admitting more harness reads after an invalid result or error.
Join all started reads before returning or throwing.
- Keep large controller, full-checkpoint and legacy archive reads
serial.
- Add real PostgreSQL regressions for overlap, concurrency limits and
failure settlement.

## Verification

- Genuine controller regression RED: all 16 checkpoint reads ran, but
peak overlap was one. GREEN: 10 focused PostgreSQL controls passed,
including bounded overlap, joined failures, active and warm handoffs,
and failed-source preservation. The other 45 cases were filtered, not
claimed as passes.
- Genuine harness regression RED: overlap was one instead of four.
GREEN: all three new PostgreSQL controls passed. Six existing authority,
replacement and refusal controls also passed.
- Final head `c059e031`: the same 10 controller and 9 harness PostgreSQL
controls passed after rebasing onto current master and refreshing Runner
artifacts. `node node_modules/typescript/bin/tsc --noEmit -p
server/tsconfig.json` also passed. The first `pnpm exec` attempt could
not start the compiler because package-manager bootstrap fetches failed;
it is not counted as a typecheck pass.
- `git diff --check` passed.
- Actual private-process acceptance with deployed `8e4c5e81`
dependencies and only the two changed owners: the complete check passed
in 37.596 seconds with no blockers. Nine runs were running before and
after; 1,750 sessions were selected. This is 17.4 seconds inside the
unchanged request limit. The earlier complete baseline was 59.665
seconds on 1,745 selected sessions, so these are observed timings on a
changing dataset, not a controlled exact speedup. No serving process was
patched.
- The first private candidate check stopped on a native-controller
blocker in 18.216 seconds. A targeted follow-up found no blocker without
any preservation or business write. Its cause was not identified. It is
retained as a nonpass and is not a speedup result.
- Current-head Greptile review is 5/5 with no actionable findings.
Required CI is running:
https://github.com/paperclipai/paperclip/actions/runs/42898013184.

## Risks

- The scans now use more simultaneous database reads. Limits remain
eight for controller source readers and four for harness proofs.
- Invalid or failed proofs still refuse eligibility. All started readers
settle before failure leaves the scan.
- Previous-release state carries forward unchanged. This changes no
stored shape, checkpoint identity, lease rule, schema, deadline or
provider protocol.
- Large legacy payload reads remain serial, so their existing memory
bound is unchanged.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. This is a measured
performance defect in the existing enrollment path.

## Model Used

OpenAI Codex, based on GPT-6, with reasoning, code execution and
parallel agents. The runtime does not expose the exact model variant or
context-window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes:` / `Closes:`
/ `Refs:` OR (b) described the issue in-PR following the relevant issue
template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change and contains no internal
ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 14:57:11 -06:00
DottaandClaude Opus 5.5 4398a723c0 fix(native): settle tool callbacks and holds that a server restart orphaned (#20628) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs send provider tool calls to the server as durable
semantic inputs. The server runs the tool and saves the result in the
controller journal
> - Close must prove that every admitted tool call has a saved result.
Otherwise it reports `native_session_settlement_incomplete` and the
runner holds a cleanup gate (`native_session_cleanup_quarantined`)
> - A server restart in the middle of a tool call leaves its input
without a result. The new controller asks the tool authority for a
receipt. Tools without a receipt (assigned app tools,
`connections_search`) return nothing, so the input stays pending forever
> - Runs that already failed this way kept no-replay holds for a turn
that had completed. The task stayed blocked with no Retry and no
continuation
> - Restarts are normal. Work that a restart interrupts must settle by
itself, not wait for a person
> - This pull request settles such a callback as interrupted, with an
unknown outcome, when its owner is provably gone, and settles the holds
that earlier releases left once their controller is gone. It never runs
the tool or the turn again
> - The benefit is that close drains after a restart, and tasks that
earlier releases blocked this way continue on their own

## Linked Issues or Issue Description

Refs #20572 (that PR fixed the presentation of this quarantine. Its
Risks section names this runner drain rule as a separate follow-up.)
Refs #20641 (it settles a resolved no-replay hold when a sibling action
saved the run's files. This PR covers a different case: the hold's own
recovery saved the files, and its run's turn completed. Both use the
same recovery receipt, `completedWorkspaceRecoveryCondition`.)

**What happened?**
A remote native run started an assigned app tool call. The server
restarted 3 seconds later. The new server adopted the runner, and the
provider turn completed about 9 minutes later. The tool call never got a
saved result. Close reported `native_session_settlement_incomplete` with
`runnerSuspended: true`, `providerDrained: false` and one pending
callback. The session quarantined, and the same-run recovery was refused
at admission. A second run showed the same shape with a
`connections_search` callback after a later restart.

The first run then had two resolved-but-blocked holds with
`preserve_without_replay_v1` and an unknown action outcome: the
cleanup-quarantine hold, and a continuation hold whose workspace
recovery saved the files. The task view said "Saved work recovered. Send
a message to continue." with no Retry and no continuation. The task
waits on a child task and stays blocked.

**Expected behavior**
After a restart, a tool call without a saved result settles
automatically. A durable receipt is replayed. Without a receipt, the
agent is told that the outcome is unknown. Close proves the drain, and
no cleanup gate is left. Holds that earlier releases left for a
completed turn settle without a person, and the task continues when its
own wait ends.

**Steps to reproduce**
1. Start a native run and call an assigned app tool.
2. Restart the server while the tool call runs.
3. Let the adopted turn complete, then close the session. Close reports
`native_session_settlement_incomplete` and quarantines. The task stays
blocked by the no-replay holds.

**Paperclip version or commit**
master at fd6e71a7ff.

## What Changed

- Root cause: `DurablePrpControlPlane.#recoverPendingSemanticInputsOnce`
(`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.ts`)
keeps an input pending when `recoverSemanticToolInput` returns null.
`PaperclipRunnerToolAuthority.recover` returns null for every tool
without a durable receipt. Nothing else could settle the input, so
`semanticToolResultsSettled()` stayed false and `#closeOnce` in
`runnerd-codex-transport.ts` threw
`NativeSessionCloseUnrecoverableError`.
- The control plane now passes `admission` to
`recoverSemanticToolInput`: `this_controller` when this instance
dispatched the callback live, `earlier_controller` when it restored the
input from an earlier controller's journal. The field is not written to
the result command or the journal. A live dispatch leaves this record
when its result is journaled.
- New
`packages/paperclip-runner/src/control-plane/orphaned-tool-callback.ts`:
- `dispatchToolCallback` records the callbacks that this process runs,
and the ones it saw fail.
- `recoverToolCallback` replays a durable receipt first. Without one, it
settles an `earlier_controller` callback that this process does not run
as `tool_call_interrupted_by_restart` (error result, `outcome:
"unknown"`, `retryable: false`). A `semantic_tool_outcome_unknown`
rejection from the authority settles the same way, only under the same
owner-gone proof.
- `native-session-executor.ts`: `SessionAuthorityEpoch.execute` runs
through `dispatchToolCallback`. `SessionAuthorityEpoch.recover` runs
through `recoverToolCallback`. The three recovery call sites pass
`admission`.
- New ownership kind `native_completed_turn_cleanup_hold`
(`server/src/services/native-runtime/completed-turn-cleanup-hold-settlement.ts`,
registered in `recovery/ownership-registry.ts`). It lists a native run
that failed with `native_session_cleanup_quarantined` after its saved
checkpoint recorded a completed turn, and that has a system-recorded
no-replay hold. After `proveOwnerGone` proves the closing controller
gone (its in-process cleanup gate went with it), it settles each such
hold as `restored`, removes its replay block, and logs
`issue.recovery_action_resolved`. A hold that still protects files not
verified as saved stays for workspace recovery. Operator decisions are
never reopened. It starts no turn. `heartbeat.ts` registers the ordinary
resume path (`resumeRemoteStopComments`) for saved user input.
- Docs: `doc/execution-semantics.md` and
`packages/paperclip-runner/docs/durable-recovery.md`.

## Verification

- Red before the fix, green after:
-
`packages/paperclip-runner/src/control-plane/durable-prp-control-plane.test.ts`
"host-tool callback orphaned by a server restart". It uses the real
control plane and an authenticated runner. A callback starts 3 seconds
before the restart, the old controller retires with no result, the new
controller adopts the runner, the turn completes, and then recovery
runs. `paperclip_call_assigned_tool` and `connections_search` settle as
`tool_call_interrupted_by_restart`, the runner accepts the late result,
`semanticToolResultsSettled()` becomes true (the condition close
requires), and the tool is not dispatched again. Before the fix, 4 of 5
cases failed: recovery received no admission and the callback stayed
pending.
- `server/src/__tests__/completed-turn-cleanup-hold-settle.test.ts` uses
the staging shape of both holds (real Postgres, rows as earlier releases
saved them). Before the fix, the ownership reconciler did not list them
(4 of 4 failed), and the execution blocker was the files hold with
`canRetry: false` and `canContinue: false`. After the fix, both holds
settle as `restored`, the execution blocker is null, no run is created,
the task keeps `blocked`, and the next pass lists nothing.
- Safety cases:
- A callback that this process still runs stays pending. After it
finishes, the next recovery pass settles it.
- A callback that this controller dispatched and saw rejected stays
pending, as before.
  - A durable receipt is replayed and not replaced.
- Holds stay while the closing controller is alive (its gate may still
exist).
- A hold whose files are not saved stays; the run's other hold settles.
  - Holds of a run whose turn did not complete are never listed.
- Focused suites pass: `durable-prp-control-plane.test.ts` and
`runnerd-codex-transport.test.ts` (506 tests); runner adoption,
release-resume, semantic readiness, ACPX reconciliation and external
driver tests; `native-session-executor.test.ts` and
`native-runner-restart-recovery.integration.test.ts` (1496 tests);
`ownership-reconciliation-report.test.ts` and
`interrupted-preparation-hold-settle.test.ts`.
- `tsc --noEmit -p packages/paperclip-runner/tsconfig.test.json` passes.
`pnpm --filter @paperclipai/server typecheck` passes.
- `pnpm verify` returns exit 2 (a global doc input changed), so required
CI supplies the broad coverage.

## Risks

- Previous-release story: no persisted shape changes. `admission` is
computed at recovery and never saved. A journal that an earlier release
saved with a pending callback restores as `earlier_controller`, so that
session settles at its next adoption. Holds that earlier releases saved
settle on the reconciler's first pass after deploy. No migration.
- No double execution: recovery never calls the tool again. It replays a
durable receipt or reports an unknown outcome. The agent is told to
check the current state before it repeats the call. Hold settlement
starts no turn and replays nothing.
- The owner-gone proof for a callback: the new controller holds the
run's checkpoint fence, and the earlier controller retired its journal
writer. If the earlier process is still alive during an overlapping
deploy, it can still finish the tool, but it can never record a result
in this session. The unknown outcome is accurate in that case.
- Hold settlement requires `proveOwnerGone` on the closing controller's
stamp, because the cleanup gate lives in that process. On a server that
has not restarted since the quarantine, the holds stay.
- The in-process failure memory is bounded to 4,096 entries. Eviction
only lets a much older failed callback settle as interrupted later. It
never replays it.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:39:45 -06:00
DottaandClaude Opus 5.5 d06f43b14f fix(workspaces): settle a resolved no-replay hold once a sibling saved its files (#20641) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - When a remote workspace save fails, recovery actions hold the task
until the files are saved
> - One failed run can have two holds: its export hold, and a no-replay
hold that a later "try again" opened for the same run
> - The no-replay hold is recorded resolved-but-blocked, and it is the
task's execution blocker
> - #20569 lets the export hold retry under the new symlink capture
policy and save the files
> - The sibling settlement that should then release the no-replay hold
only looked at active holds, so the task stayed blocked with no working
Retry
> - This pull request lets the sibling settlement and its discovery
include resolved no-replay holds
> - The benefit is that the task continues without a person once the
files are saved

## Linked Issues or Issue Description

Refs #20569

**What happened?**
On staging, a native run on a Daytona sandbox failed to save its
workspace (`remote_workspace_archive_unsafe_symlink`). Its export hold
(`native_workspace_sync_out_retry_exhausted`) was exhausted after 6
attempts and handed to the board. A user comment then opened a second
hold for the same run (`native_continuation_requires_reconciliation`,
`preserve_without_replay_v1`). That hold also failed 6 times and was
recorded `resolved` with outcome `blocked`. The task's execution blocker
is this second hold.

**Expected behavior**
After #20569 is deployed, the worker re-arms the export hold and saves
the files. The no-replay hold of the same run then settles, and the task
is no longer blocked.

**Steps to reproduce**
1. Create both holds with the exact staging evidence (see the new test
`stagingIncident`).
2. Run the recovery worker until nothing is due.
3. Before this change, the export hold completes, but the no-replay hold
stays `resolved`/`blocked` and `getExecutionBlocker` still returns it
with `canRetry: false`.

## What Changed

- `workspace-recovery-fallback.ts` (`settleSiblingRecoveredHold`): a
resolved no-replay hold (`isRetainedFileHold`) can now settle when a
verified receipt of a sibling action saved its run's files. Settling
removes `automaticRecovery`, so the hold stops blocking the task. It
starts no turn.
- `workspace-recovery-fallback-discovery.ts`: the worker discovers such
a resolved hold when a sibling of the same run is recovered.
- New test in `workspace-recovery-capture-policy.test.ts` with the exact
staging shape of both holds: board owner, `active_run_watchdog`, system
recovery policy version 1 without a capture policy version, stage
`restoring_files`, `export_interrupted` with cause `provider_error`, 6/6
attempts, `historicalWorkspaceSource` and `workspaceRestoreRecovery`
lease ids.
- `doc/execution-semantics.md`: states that a resolved no-replay hold
also settles from a sibling save.

## Verification

- Red before the fix: the new test failed. The no-replay hold stayed
`resolved`/`blocked`, and it stayed the execution blocker.
- Green after the fix: `npx vitest run --root server
src/services/workspace-recovery-capture-policy.test.ts` (3 passed). The
test drives the worker's own discovery (`nextWorkspaceRecoveryAt` +
`reconcileRetainedWorkspaces`) until nothing is due. It checks: the
export hold is re-armed once and saves the files; the no-replay hold is
`resolved`/`restored` by the sibling; `getExecutionBlocker` returns
null; the continuation hook ran when the task was free; no new run was
created; the worker goes idle.
- Related suites: `workspace-recovery-fallback`, `-admission`,
`-stop-proof`, `-source`, `-preparation`, `-preparation-gates`,
`execution-blocker-read`, `preserved-native-response-wait.integration`:
409 passed.
- `npx tsc --noEmit -p server/tsconfig.json`: clean.
- `pnpm verify` exits 2 (global doc input), so CI supplies the broad
coverage.

## Risks

- A resolved no-replay hold is now released when another action's
verified receipt saved the same run's files. The receipt check is
unchanged, so an unverified sibling still does not release it.
- Settling starts no turn. Waiting input and the run's own result
continue the task through normal admission, which still checks every
other hold.
- Previous-release story: holds saved by earlier releases need no
migration. The worker discovers them on its first pass after deploy.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, used through
Claude Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:39:24 -06:00
DottaandClaude Opus 5.5 e3bc4ca798 fix(native): settle warm follow-ups that a restart left behind (#20631) 
<!-- Write all pull request text in Simplified Technical English
(ASD-STE100): short sentences, one instruction per sentence, simple
approved vocabulary, and the active voice. -->

## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native warm runs on a remote sandbox continue the saved session of
the previous run. Admission first proves what happened to the previous
Runner.
> - A deploy restarted the staging server while warm Runners were idle.
Three follow-up runs then failed before provider work, each with a
different refusal, and their tasks stayed blocked.
> - The refusals asked for proof that a restart had made impossible. A
restart is a product failure, so recovery must settle it automatically
and keep the strict checks for real uncertainty.
> - This pull request accepts the proof that the restart leaves behind,
in three places: the warm owner on a replaced sandbox, the suspended
Runner with an undelivered suspend receipt, and the restore hold of a
run that saved nothing new.
> - The benefit is that these follow-up runs start, and state saved by
older releases carries forward without a migration.

## Linked Issues or Issue Description

Refs #20567, #20346

**What happened?**
On staging, a deploy replaced the webserver process at about 18:04 UTC
on 2026-11-10 (lease controller boot `348ec0bb` on one deployment, then
boot `3e84c195` on a new deployment that started at 18:04:16). Three
runs of one Codex agent on Daytona failed between 18:08 and 18:15:

1. A warm follow-up got a newly created sandbox. The previous warm
Runner stayed `ready` in its own retained sandbox. Live adoption denied
it with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`
(`remote-warm-session-recovery.ts:127`), because only a resumed sandbox
may be adopted. The per-turn retirement path refused warm owners
(`lifecycle_not_per_turn`).
2. A warm follow-up resumed the same sandbox and directory. The previous
Runner had been suspended by the shutdown, but the controller stopped
before it acknowledged the suspend result. The Runner kept its terminal
fence (`pendingTerminalDelivery` for `runner.suspend`). The read refused
it with `native_controller_recovery_missing_evidence:runner_not_settled`
(`native-session-executor.ts:7032`).
3. A continuation into a fresh workspace failed with
`remote_workspace_recovery_receipt_invalid`
(`remote-workspace-recovery-attachments.ts:60`). Its predecessor had
restored the committed save, failed before provider work, and kept a
restore hold with no recovery receipt. Its saved reference named the
snapshot it restored.

**Expected behavior**
After a restart, the next run continues. It retires a finished owner
that nobody can adopt, reads a finished suspension, and skips a source
that left no unsaved files. Real uncertainty still refuses with a named
reason.

**Steps to reproduce**
1. Run a warm native Codex turn on a reusable Daytona sandbox and let it
commit.
2. Restart the server (deploy) while the Runner is idle.
3. Start the next turn. Get a new sandbox (case 1), or resume the same
sandbox after the shutdown suspended the Runner (case 2).
4. For case 3, continue a task whose previous run restored the last save
and failed before provider work, into a fresh workspace.

**Paperclip version or commit**
Staging before `2e3cbe57` (the runs predate that deploy). The gaps
remain on `master` at `9b21dcbd`.

## What Changed

- `inspectRemoteWarmSessionRecovery` returns `null` when this run's
sandbox is `created` or `replacement`. Such a sandbox holds no Runner to
adopt. The caller then uses ordinary admission.
- `inspectFinishedPerTurnRunnerdOwner` also retires a **warm** owner on
a remote sandbox runner when this run's sandbox is `created` or
`replacement`. It uses all the existing proofs: terminal run, committed
result, released coordinator lease, stopped or fenced controller,
settled journal, and quarantine copies. Local warm Runners and resumed
sandboxes keep the live path.
- `restoreRemoteSuspendedRunnerCheckpoint` accepts a suspended Runner
whose only open fence is its own undelivered `runner.suspend` receipt.
The fence must name `runner.suspend` with lifecycle `suspended`, it must
be the last controller command, and the restored journal must not name
that command differently. A provider cleanup, unsent events, a turn's
terminal fence or a warm transition still refuse with
`runner_not_settled`. A later authorized Runner process reconciles the
receipt, as the Runner already does.
- `attachPreservedRemoteWorkspaceSnapshots` skips a source with a
receipt-less restore hold when its saved reference names the exact
committed snapshot that its own succeeded `remote_workspace_restore`
loaded at the same epoch. Any other saved snapshot still needs the
canonical receipt.
- `doc/execution-semantics.md` describes the three rules.

## Verification

- Red (unfixed code), each in the exact staging shape:
- `native-recovered-owner-followup.test.ts` "retires a finished warm
owner left behind by a restart when this run's sandbox is
created/replacement" failed with `runner_state_identity_mismatch:
prior_owner_terminal_state_indeterminate`.
- `native-controller-recovery.integration.test.ts` "defers a remote warm
continuation…" failed: live adoption rejected a created/replacement
sandbox with the same error.
- `native-controller-recovery.integration.test.ts` "reads a resumed
sandbox's suspended Runner with an undelivered suspend receipt" failed
with `native_controller_recovery_missing_evidence:runner_not_settled`.
It reads through a sandbox runner that refuses paths outside the
workspace base.
- `workspace-recovery-preparation.test.ts` "decides a receipt-less
restore hold of a source that restored_unchanged" failed with
`remote_workspace_recovery_receipt_invalid`.
- Green: the same tests pass. Focused files pass:
`native-recovered-owner-followup.test.ts` and
`native-controller-recovery.integration.test.ts` (104 tests),
`workspace-recovery-preparation.test.ts` (85 tests).
- Safety cases (strict checks stay strict):
  - A warm owner on a resumed sandbox is not retired.
- A warm owner with an unsettled journal (pending command, unanswered
tool input) stays held and names `retirement_refused=journal_…`.
  - A warm owner whose finalization lease is still held stays held.
- A turn terminal fence, a provider cleanup, unsent events, a fence
behind a later command, or a journal that names the command differently
still fail with `runner_not_settled` and cache nothing.
- A restore-hold source that saved another snapshot, whose restore did
not succeed, or whose restore was at another epoch still fails with
`remote_workspace_recovery_receipt_invalid`.
- Neighbouring suites pass: `native-session-executor.test.ts`,
`plugin-runner-release-resume.integration.test.ts`,
`explicit-native-continuation.test.ts` and
`native-harness-filesystem-independence-continuation.integration.test.ts`
(1786 tests). The last one asserted the old warm refusal on a created
sandbox. It now shows that the warm follow-up resumes the same provider
session for the new run.
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` returns exit 2, because a global doc input changed
(`doc/execution-semantics.md`). Required CI supplies the broad coverage.

## Risks

- Previous-release state: all three rules read state that earlier
releases already save (lease outcome, coordinator rows, Runner state in
the sandbox, journal commands, run results and workspace operations).
Rows saved by older releases carry forward. No migration.
- Case 1 starts a new Runner for a warm session whose old Runner may
still exist, idle, in its retained sandbox. Its controller is gone and
its result is committed, so it can accept no new commands. The provider
session follows the existing resume-or-replace policy.
- Case 2 trusts a suspend fence only in the exact shape that the Runner
writes before it sends the result. Other fences still refuse.
- After case 1, a warm follow-up resumes from its own warm checkpoint.
That checkpoint carries no replacement policy, so a failed provider
resume is still reported as `native_session_recovery_failed` and is not
replaced. This is the existing warm behavior.
- Case 3 skips a source only when it provably saved nothing beyond a
committed snapshot. That snapshot stays in its own workspace.
- The three tasks that already failed stay blocked until their next wake
or a user message. This PR does not add an automatic continuation for a
completed no-replay hold. A follow-up can add that.
- `native-runner-restart-recovery.integration.test.ts` (runs only with a
runner container) no longer probes a replacement sandbox, because that
probe now retires the owner by design.
- Not verified live on Daytona.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, read-only browser access to staging).
Extended reasoning enabled.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:38:30 -06:00
DottaandClaude 24661683db fix(recovery): bootstrap a remote follow-up that a restart cut before its Runner launched (#20642) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - After a control-plane restart, restart recovery classifies each
claimed native run: reattach a live Runner, resume from a checkpoint, or
bootstrap again when durable records prove no provider work
> - A follow-up run on a continued session gets a rebound admission
checkpoint. It names the new run, but it keeps the previous turn's
provider session ID
> - Recovery read that copied provider session as this run's provider
work. A follow-up killed after `native.process_start_requested` but
before its remote Runner launched was classified
`reattach_remote_runner`
> - The reattach then had no Runner to find, failed with
`native_continuation_requires_reconciliation`, and the task needed Retry
> - This pull request stops the copied session from counting as this
run's work, and keeps the existing durable proofs as the authority
> - The benefit is that these restarts bootstrap again by themselves and
do not repeat provider work

## Linked Issues or Issue Description

Refs #20561, #20555

**What happened?**
Two Daytona restart-chaos tasks needed a Retry click:
- Wave `w-20261110T173243-s9202` t02 (claude C1, K2): the follow-up run
was killed at `stage.asset.home`, after
`native.process_start_requested`. Recovery chose
`reattach_remote_runner`. Verification failed with
`native.remote_runner_recovery_failed {stage: runner_state, reason:
state_read_failed}` (`runner_remote_state_unavailable: exit=1`):
`runner-state.json` did not exist because the Runner never launched. The
storage fault window (ST7) had ended 30 s before, so it was not the
cause.
- Wave `w-20261110T174930-s9205` t04 (codex C1, RR1): the scheduled
retry of a `process_lost` run was killed at
`session.checkpoint.restore`, after `native.process_start_requested`.
Recovery chose `reattach_remote_runner`. Verification failed with
`native_remote_recovery_lease_mismatch: stage=runner_verification
field=identity_run_id`: the local Runner identity belonged to the
previous run, because this run's Runner never started.

**Expected behavior**
A run whose durable records prove that no Runner received a provider
turn bootstraps again without a person.

**Steps to reproduce**
1. Start a remote follow-up run on a continued session (for example, a
comment after a completed turn).
2. Kill the server after `native.process_start_requested` and before the
remote Runner launches.
3. Observe `reattach_remote_runner`, a failed verification and a Retry
banner.

**Paperclip version or commit**
master close to `6d6b2d0e` (waves); still reproduced by the new tests on
master `90dd864bff`.

## What Changed

- `native-restart-recovery.ts`, no controller checkpoint for the run
(`remote_runner_never_received_commands`): a follow-up with
`sessionIdBefore` qualifies when it carries only its continued-session
checkpoint, and a predecessor's settled Runner identity
(`carriesInheritedRunnerIdentity`) no longer disqualifies it. A recorded
process identity, a Runner-sent event, a provider event or a pending
request still keeps the run held.
- `native-restart-recovery.ts`, controller checkpoint exists: the
existing Postgres proof (`nativeRunnerCheckpointRemoteBootstrapProof`)
now also runs when the run carries only a rebound checkpoint whose
provider session a settled earlier run of the same session saved
(`carriesSettledPredecessorSession`). The copied provider session then
does not count as this run's provider evidence. The controller
checkpoint stays the authority: any turn command or turn event keeps the
run held.
- `doc/execution-semantics.md`: describe the rule.

## Verification

-
`server/src/services/native-runtime/native-runner-restart-recovery.integration.test.ts`
(real Postgres):
- "restarts a follow-up whose controller never saved a checkpoint (s9205
t04)" and the variant "with its predecessor's remote runner identity":
red on master (`reattach_remote_runner`), green now.
- "restarts a follow-up whose saved controller checkpoint proves no
provider turn (s9202 t02)": red on master, green now.
- Held cases stay held: own recorded Runner identity, a carried active
turn, a provider session that no settled earlier run saved, a
`turn.start` command, a projected `turn.started` event.
- Whole file: 171 passed.
- `npx tsc --noEmit -p server` passes.
- The executor's existing `bootstrap_incomplete` path handles both
shapes: a scoped root that belongs to the previous run goes through the
existing prior-owner verification, and a root of this run restored from
the controller checkpoint goes through
`runnerdStateProvesIncompleteBootstrap` /
`runnerdStateProvesNoRemoteProviderTurn`.

## Risks

- A run with any recorded Runner process, Runner-sent event, provider
event, turn command or pending request keeps today's reattach or hold.
- A bootstrap claim charges one provider attempt, as today. The bounded
limit (3) still applies.
- No persisted shape changes.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:37:56 -06:00
DottaandClaude 6892ed3a98 test(restart-chaos): save a screenshot and banner HTML when the Retry button wait fails (#20643) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart-chaos harness drives the board UI and clicks Retry the
way a person would
> - When no visible Retry appears, the harness grades
`ui_retry_button_missing` and keeps only the banner text
> - That is not enough to tell a product bug (no button, a hidden
button, a wrong name) from a harness locator bug after the wave has
ended
> - This pull request saves a full-page screenshot and the banner and
Retry-button DOM when the wait fails
> - The benefit is that every `ui_retry_button_missing` can be diagnosed
from the wave folder

## Linked Issues or Issue Description

Refs #20616

**What happened?**
Wave `w-20261110T174930-s9205` t01 graded `ui_retry_button_missing`, but
the captured banner text contained "Retry". The harness saved no
screenshot or DOM, so nobody could prove whether the page had a hidden
duplicate or no actionable button.

**Expected behavior**
A failed Retry-button wait leaves enough evidence to diagnose it later.

**Steps to reproduce**
1. Run a restart-chaos wave with the board UI.
2. Get a task whose Retry wait times out.
3. Observe that the wave folder holds only the banner text.

## What Changed

- `tests/restart-chaos/board-ui.ts`: add `saveRetryMissingDiagnostics`.
On a failed Retry wait it saves `ui-retry-missing-<issue>-<time>.png`
(full page) and a `.json` file with the page URL, each
`[role=status]`/`[role=alert]` banner's `outerHTML`, and every
Retry-named button's visibility, bounding box, banner membership and
HTML. Diagnostics never change the click result.
- `tests/restart-chaos/main.ts`: the wave's board helper writes to
`<wave>/ui-retry-diagnostics`.
- `tests/restart-chaos/observer.ts`, `types.ts`: the task's retry
attempt records the saved file paths.
- `tests/restart-chaos/board-ui.test.ts`: a test for the saved files.

## Verification

- `npx vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/board-ui.test.ts`: 9 passed (new: "saves a
screenshot and the banner's HTML when no visible Retry appears").
- `npx tsc --noEmit -p tests/restart-chaos/tsconfig.json` passes.

## Risks

- Harness only; no product code changes. A failed screenshot or file
write is ignored and does not change the grade.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:37:19 -06:00
DottaandClaude ed5e7493bb fix(native): take back a lapsed lease held under the run's own restart claim (#20625) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A native run holds an execution lease on its
`native_run_finalizations` row. The owner renews it every 5 minutes.
After a host sleep (or the restart-chaos time fault TM2), the stored
lease looks lapsed although the owner is alive.
> - #20474 lets the same owner take back a lapsed lease when no other
actor recorded anything on the row. One of the checks is
"`recovery_state` is null".
> - Restart recovery writes `recovery_state` (`resuming_session`,
`awaiting_runner_reattach` or `bootstrap_incomplete`) in the same update
that grants the lease. The executor keeps that value until a result is
accepted.
> - So a run that a restart resumed can never take back its own lapsed
lease. Renewal interrupts the turn, and failure settlement is refused
with `native_session_lease_lost`. The run fails with `adapter_failed`,
recovery holds it with `preserve_without_replay_v1`, and the user must
click Retry.
> - This pull request counts the claim's own record as untouched when
the row still shows exactly that claim and nothing recorded since.
> - The benefit is that a resumed run survives a later host sleep with
no Retry. Every real takeover still fences the late owner out.

## Linked Issues or Issue Description

Refs #20474

**What happened?**
The zero-retry restart-chaos matrix (local wave L7, seed 9107: TM1, TM2,
TM3 with 1 h aging) had 3 of 6 tasks that needed Retry. All three were
Codex runs. In each run, a TM3 kill made restart recovery resume the run
(`resume_dead_runner`, generation 2). The runner restored the
conversation, reported `provider_turn_lost_on_restore`, and the
same-conversation restart continuation started as designed. Then the TM2
fault aged every database time by one hour. The execution lease lapsed
while its owner was alive. The run then failed with
`native_session_lease_lost` (`adapter_failed`), and recovery stopped
with `native_continuation_requires_reconciliation` /
`preserve_without_replay_v1`. One run had already proposed its result
and completed its turn. The Retry click worked every time.

**Expected behavior**
A run whose lease lapsed only because time jumped, and that no other
actor claimed, continues with no Retry. This must also be true after a
restart resumed that run.

**Steps to reproduce**
`node cli/node_modules/tsx/dist/cli.mjs tests/restart-chaos/main.ts
--env local --adapters codex,claude --workloads A1,B5,C1 --faults
TM1,TM2,TM3 --time-age-hours 1 --concurrency 3 --tasks 6 --seed 9107`

## What Changed

- `server/src/services/native-runtime/native-execution-lease.ts`: the
"untouched" part of `nativeExecutionLeaseHeldCondition` now also accepts
the run's own local restart-recovery claim. All of these must be true:
- the last `recovery_history` entry is that claim: same controller boot
and generation as the row;
- its disposition matches the current `recovery_state`
(`resume_dead_runner` → `resuming_session`, `reattach_existing_runner` →
`awaiting_runner_reattach`, `bootstrap_incomplete` →
`bootstrap_incomplete`);
  - `recovery_request_id` equals the request that claim recorded.
Any later actor appends a history entry (shutdown suspension, failure
settlement), moves the state (`awaiting_evidence`, `blocked`), or
replaces the owner, boot or generation. Each of these still fences the
late owner out. All other untouched checks are unchanged.
- `native-execution-lease-reacquire.test.ts`: new cases for each claim
kind, a requested claim, failure settlement under the claim, and eight
fenced variants.
- `doc/plans/2026-11-09-explicit-native-lease-claim-tokens.md`: Step 1
lists the new rule.

## Verification

- Red/green: `cd server && npx vitest run
src/services/native-runtime/native-execution-lease-reacquire.test.ts`.
Without the source change, 5 new cases fail (the three claim kinds, the
requested claim, and failure settlement). With it, all 37 pass.
- Related suites pass: `native-execution-lease.test.ts`,
`heartbeat-restart-claim-lapse.test.ts`,
`heartbeat-native-orphan-sweep-lease-race.test.ts`,
`native-runner-restart-lease-renewal.integration.test.ts`,
`native-runner-restart-preparation.integration.test.ts`.
- `cd server && npx tsc --noEmit` passes.
- Live: the same L7 wave (seed 9107, local) on this branch. The server
log has 0 `native_session_lease_lost` (the baseline had 3). The two
tasks whose cells failed this way in the baseline (codex C1 TM1, codex
C1 TM3) now pass with no Retry. Two other tasks needed Retry for
different causes, which this pull request does not change: codex B5 TM2
(a `runner.drain` timeout, then a same-run retry that may not continue a
turn lost on restore) and claude C1 TM2 (`ACPX session is not open` on
`turn.start` until the retry budget ran out). The L6 wave (seed 9106)
also ran on this branch: 5 of 6 passed. Its one failure is
`execution_recovery_budget_exhausted` and is not related to leases.

## Risks

- The rule widens when an owner may take back its own lapsed lease. It
applies only to local runs, before a result, with the same owner token,
attempt, boot, pid, process start and generation. A claim that any other
actor changed afterwards does not match.
- Rows saved by earlier releases carry the same history entry shape, so
a run claimed before a deploy is handled the same way. A deploy changes
the boot, so the new process never takes back the old process's lease.
- No schema change and no migration.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, via Claude Code with
tool use (shell, file edits, test runs). Extended reasoning.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:31:15 -06:00
DottaandClaude Opus 5.5 2e6850e745 perf(task-chat): open tasks from a preloaded cache and cold tasks in one request (#20592) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - People open tasks from task lists, the Inbox and the sidebar many
times each hour
> - A task's chat shows its first frame only after about seven separate
reads finish
> - #20355 added one batched `chat-bootstrap` read that returns all of
these reads. #20450 used it only for the sidebar's Recents
> - The hosted database is CPU-bound, so the UI must open tasks faster
and must also send fewer requests
> - This pull request uses the bootstrap for task lists, for open intent
on any task link, for cold opens and for a hard load of a task URL. It
shares one coordination module between all of these paths
> - The benefit is that a task opens from cache more often, a cold open
sends one chat request instead of about seven, and the first content
paint comes sooner

## Linked Issues or Issue Description

Refs #20355 (server read that this pull request consumes). Refs #20450
(Recents preload, now on the shared path).

**Is your feature request related to a problem?**
A click on a task in a task list waited for about seven separate chat
reads. The default task list (Coordinator) had no preload at all. A
press or a resting hover on a task link sent the same seven reads, and
the page then sent a second set. A hard load of a task URL started its
chat reads only after the JavaScript bundle and the company list loaded.
Sidebar project task links had no open-intent prefetch.

**Describe the solution you'd like**
Use the single `chat-bootstrap` read everywhere a task is about to open.
Let the page's gating reads share a bootstrap that is in flight instead
of sending their own requests. Keep the core issue read separate so the
header does not wait for the larger read.

**Describe alternatives you've considered**
Preload 30 rows per list in three requests (the first prototype). Each
task costs the server about fifty queries, and a resting hover warms any
other row before the click, so this change preloads 10 rows in one
request.

## What Changed

- `ui/src/lib/issue-chat-bootstrap.ts`: one module decides when to send
a bootstrap and lets page reads share it (`readFromIssueChatBootstrap`).
Seeding uses the existing `issueChatBootstrapCache` (replace-style reads
are refreshed when newer, merge-style reads are filled only when empty,
other companies are ignored). It skips tasks that are loading, loaded in
the last 30 seconds, or already fresh in cache, under the id or the
identifier. It never aborts a request in flight, so a click that
unmounts the list still uses the answer. Each request is bound to the
account that sent it; a request sent before the first session read is
bound to the first account the session reports, and a sign-out revokes
it. A page read shares a bootstrap only while its key holds no data (a
refresh after a write always reads the server), and waits at most 5
seconds for it. Every ref of a task (id and identifier) is tracked at
request start. Entries expire after 30 seconds.
- Task lists preload the chat of their top 10 rows in one request, after
the list's own reads settle and the browser is idle
(`usePreloadTaskChats`, `when-page-settles.ts`): the Coordinator task
list (`TaskList` with `preloadCompanyId`), the streamlined and legacy
issues lists and the streamlined Inbox. Recents now uses the same path,
so a task in both places loads once. A list that stays open preloads
each task at most once in five minutes: live events (#20563's lanes)
keep a preloaded task's cache current or mark it stale, and a stale key
is read from the server, never from an older bootstrap. All seeded keys
are per-task keys on the fast live lane, not the ambient lane.
- Open intent (press, click, resting hover or focus) on a task with no
chat history in cache warms it with one bootstrap request instead of six
separate reads (`prefetchIssueDetailForNavigation`). A task seen before
still refreshes its cached reads one by one.
- A cold open starts one bootstrap. The gating reads (recent comments,
activity, runs, input facts, interactions, attachments, live runs,
active run) wait for it instead of sending their own requests. The core
issue read stays independent.
- A hard load of a task URL starts `GET /api/issues/:ref/chat-bootstrap`
from the inline preload script, in parallel with the JavaScript bundle.
The page adopts that answer.
- Sidebar project task links (Coordinator sidebar) get the same hover,
press and focus warming as Recents and `IssueLinkQuicklook`.
- Tests: unit tests for the coordination module, Recents, `TaskList`
preload, sidebar warming and the cold-intent path. A new layout journey
checks that a task the Inbox preloaded reveals once at its latest
message without reading its chat again. Journeys that hold one separate
read now also hold or fail the bootstrap. The Inbox reveal journey held
a read that the chat no longer sends on open (it was flaky on master
too), so it now holds the recent comment window.
- The perf harness gains `PAPERCLIP_ISSUE_PERF_LIST_DWELL_MS` and the
README documents it.

## Verification

Measurement setup: production UI builds of master (`e01fe7f84a` and the
earlier base `8d2bf78323`) and of this branch, served by one local
server with `PAPERCLIP_UI_DEV_MIDDLEWARE=false` and embedded PostgreSQL.
Only `index.html` changed between variants (both asset sets were
present), so the server, database and seeded task were the same. Four
rounds of the issue-detail harness, with master and branch alternated in
each round: 20 pairs per S1 cell, 40 per S2 cell. The host was shared
with other agents (load average 16 to 43), so the spread is large.

Content paint (the task is readable), median paired delta, branch minus
master:

| Scenario | Profile | Pairs | Content paint | Branch faster in | API
requests before content (master / branch) |
|---|---|---:|---:|---:|---:|
| S1 preloaded click (dwell 1500 ms) | unthrottled | 20 | -45 ms | 15/20
| 25 / 18 |
| S1 preloaded click (dwell 1500 ms) | Fast 4G, 4x CPU | 20 | -23 ms |
13/20 | 26 / 18 |
| S1 click, no dwell | unthrottled | 20 | -51 ms | 15/20 | 25 / 18 |
| S1 click, no dwell | Fast 4G, 4x CPU | 20 | -21 ms | 12/20 | 26 / 18 |
| S2 cold deep link | unthrottled | 40 | -61 ms | 28/40 | 54 / 40 |
| S2 cold deep link | Fast 4G, 4x CPU | 40 | -129 ms | 29/40 | 47 / 39 |

Medians (p25 to p75), content paint in ms: S1 dwell unthrottled 295
(283-315) → 248 (217-276). S1 dwell Fast 4G 1133 (1114-1226) → 1093
(853-1176). S1 no dwell unthrottled 295 (277-328) → 260 (233-332). S1 no
dwell Fast 4G 984 (842-1196) → 923 (816-1110). S2 unthrottled 1002
(677-1235) → 896 (619-1280). S2 Fast 4G 4643 (4155-5061) → 4516
(4298-4894).

Header paint is neutral to slightly later in two cells: S1 dwell
unthrottled +39 ms (3/20 faster) and S2 Fast 4G +56 ms (17/40). In those
cells the header now paints in the same frame as the content. In the
first two rounds, the median gap from header to content fell from
105-157 ms to 29-74 ms unthrottled and from 557 ms to 298 ms for S2 on
Fast 4G. Other header cells are -27 to +1 ms.

The local server answers each read in a few milliseconds, so the network
wait that the preload removes is small here. The request count is the
stable result: a cold open sends one chat request instead of about
seven.

List page: the preload request starts after the first task row is in the
DOM (64 ms after it unthrottled, 200-280 ms after it on Fast 4G, page
clock). Time to the first visible row, 30-task list: unthrottled master
1863 ms vs branch 1563 ms (6 pairs, branch faster in 5). Fast 4G master
3626 ms vs branch 3727 ms (14 pairs, branch faster in 6, mixed signs).
The preload starts after the row paints, so it cannot delay it; the Fast
4G difference is noise. One 10-task bootstrap took 78-162 ms on the
server.

Checks run:
- `pnpm exec vitest run` for the bootstrap, cache, prefetch, Recents,
sidebar, quicklook, `TaskList`, issues list, Inbox, issue detail and
Coordinator tests: 18 files, 646 tests pass.
- `pnpm exec tsc -p ui/tsconfig.json --noEmit`: clean, except missing
generated Storybook catalog files that this change does not touch.
- `pnpm check:token-gates`: clean.
- `tests/e2e/task-loading-frame.spec.ts`,
`disconnect-resilience.spec.ts` and `connection-reviews.spec.ts`
(production build): 21 passed, 1 skipped.
- `tests/perf/issue-detail/layout-stability.spec.ts` and
`plan-startup.spec.ts` against the production build: all pass. The Inbox
reveal journey and the new preloaded journey passed 4 of 4 repeats each.
- `pnpm verify` asks for broader CI coverage because an e2e spec
changed.

## Risks

- More server work for list views: one bootstrap of up to 10 tasks per
list visit (about fifty queries per task), skipped for tasks already
loaded in the last 30 seconds. It runs only after the page's own reads
settle and the browser is idle, and only while the tab is visible.
- A resting hover on a never-opened task now sends one bootstrap instead
of about six reads. The server work per task is about the same; the
request count is lower.
- If a bootstrap fails, each page read falls back to its own request.
The fallback is slower than before for that one open (the reads start
when the page mounts, not at the press).
- The header can paint a few tens of milliseconds later in the same
frame as the content when everything is cached.
- An answer that arrives after a sign-out or account change is dropped.
An answer for another company is ignored.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use (shell, file edits, Playwright runs).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:26:27 -06:00
DottaandClaude 6505538fc5 fix(realtime): limit a shared live-event read set to 100 events and 250 ms (#20632) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - The server sends the output of a running agent to the browser on the
company WebSocket.
> - #20624 lets the events that are already in the queue share one set
of permission and redaction reads.
> - That set could serve a full queue of 1,000 events for one second.
> - If access is revoked during that time, the principal still receives
the queued output. The security scan on #20624 reported this.
> - This pull request limits one set to 100 events and 250 ms.
> - The benefit is a small, stated limit on output that a revoked
principal can receive, with the same delivery speed.

## Linked Issues or Issue Description

Refs #20624

**What happened?**

After #20624, one set of permission reads can authorize each event that
was in the queue when the set opened. The queue holds at most 1,000
events or 1 MiB. The set stays in use for one second. If a membership,
an API key or a task grant is revoked in that time, the server still
sends the remaining queued events.

**Expected behavior**

A revocation stops delivery after a small, fixed quantity of queued
output. The documentation states that quantity.

**Steps to reproduce**

1. Open a company WebSocket as a user who can read a private task.
2. Publish 1,000 transcript events for a run of that task in one burst.
3. Revoke the task grant while the server delivers the burst.
4. The server delivers the remaining queued events.

**Paperclip version or commit**

`master` at 90dd864bff.

## What Changed

- `live-event-delivery.ts` replaces a shared read set after 100 events
or 250 ms. Before, the limit was one second with no event limit.
- `live-event-delivery.test.ts` pins the exact limits. One set serves
events 2 to 101 and event 102 opens a new set. Access is revoked during
that set, and event 102 is not delivered. A set is kept at 249 ms and
replaced at 250 ms.
- `doc/live-transcript-delivery.md` states the limit and that queued
output from before a revocation can still be sent up to that limit.

## Verification

- `npx vitest run src/realtime
src/__tests__/live-events-run-read.test.ts` in `server/`. 66 tests pass.
- The boundary tests fail when either limit changes by one in either
direction (99, 101 events; 249, 251 ms).
- The 200-event burst test against PostgreSQL issues 22 statements with
this limit. Separate reads issued 1,401.
- Not run locally: a clean server typecheck. The local worktree reports
type errors in `server/src/services/chat-channels/slack/adapter.ts`,
which this change does not touch. CI supplies the typecheck.
- Not verified: delivery delay on a hosted instance after deployment.

## Risks

- More permission reads for large bursts: one set for each 100 queued
events. A burst of 1,000 events uses 10 sets, not 1,000.
- Output produced before a revocation and already in the queue can still
be sent, up to 100 events or 250 ms. Before #20624 the limit was the one
event in flight.
- No schema, API or protocol change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 14:18:20 -06:00
DottaandClaude Opus 5.5 fd6e71a7ff fix(native): restore a warm backup when its source sandbox is idle (#20627) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Native runs on remote sandboxes keep a warm sandbox and a verified
harness backup so the next run continues the same provider session
> - A run that starts while the previous run's workspace export is still
saving gets a new (created) sandbox lease, because reuse skips an
exporting sandbox
> - When that run prepares the harness, it finds a backup but no
recorded reusable lease, and it fails closed with
`runner_harness_state_mismatch: backup_without_reusable_lease`
> - #20278 accepted a destroyed source and a settled restore hold as
retired backup sources, but not an idle warm source that is still
`retained` for reuse
> - This pull request accepts an idle warm source under the same proofs,
claims it so it is never resumed again, and covers the resumed lease
that the failed run leaves behind
> - The benefit is that these recoveries continue the same session
automatically instead of blocking the task

## Linked Issues or Issue Description

Refs #20278

**What happened?**
A monitor wake started about 90 seconds after the previous run on the
same task finished. The previous run's warm sandbox lease was `retained`
with policy `reuse_by_environment`, but its workspace export was still
saving. Reuse skipped it and created a new sandbox. Workspace
realization waited for the export (about 136 s) and restored the saved
files. Then harness preparation found the previous run's backup (its
lease stamp names this exact manifest, and no provider work followed
it). The source lease was not destroyed and was not a restore hold, so
the run failed with `native_session_recovery_failed: Error:
runner_harness_state_mismatch: backup_without_reusable_lease` and the
task was blocked.

**What did you expect?**
The run continues the same provider session. The idle source sandbox
holds exactly the backed-up state, so restoring the backup into the new
sandbox is safe.

**Steps to reproduce**
1. Finish a warm native run on a remote sandbox. Its workspace export is
still saving.
2. Start the next run on the same task before the export settles.
3. Reuse skips the exporting sandbox and creates a new lease. Harness
preparation fails with `backup_without_reusable_lease`.
4. A retry resumes that new lease, which has no harness state, and fails
with `resumed_unavailable`.

## What Changed

- `native-retired-backup-source.ts`: an idle warm source is a third
retired backup source. It must be `released` or `retained` with policy
`reuse_by_environment`, its own run must be terminal and not the current
run, and it must have no cleanup in flight, no restore hold or pending
shutdown settlement, and no pending native workspace export. The backup
stamp, single-owner chain, and later-provider-work checks still apply.
- Before the restore, the idle source is claimed with a compare-and-swap
on the checked row. The claim moves its `reusableSandboxLease` scope
into a `reusableSandboxLeaseSuperseded` receipt. Reuse never resumes
that older state again, and the next acquisition treats it as obsolete
and destroys it. If a run resumed or changed the source first, the claim
matches no row and the backup is not restored.
- `environments.ts`: a cross-run reuse handoff now requires that the
source row has no superseded receipt. The supersede claim and the
handoff compete on the same row, so an acquisition that selected the
source before the claim cannot resume it afterward.
- The proof now rejects a backup whose source is the target allocation
itself (`targetProviderLeaseId`).
- `native-session-executor.ts`: a resumed lease whose sandbox has no
harness state at all (`unavailable`) restores the backup under the same
proof. This recovers the lease that the failed run left behind, so rows
saved by older releases recover after deploy without a migration.
- `doc/execution-semantics.md`: documents the idle-source rule, the
supersede claim, and the resumed stateless lease.

## Verification

- Red before the fix, green after:
- `native-session-executor.test.ts` "restores a warm backup into a
created lease only when an idle warm source finished its run and its
export" failed with `backup_without_reusable_lease`.
- "... a resumed lease never held the session and the warm source is
idle" failed with `resumed_unavailable`.
- `native-retired-backup-source.integration.test.ts` (real Postgres)
"authorizes an idle retained/released warm source and claims it as
superseded" failed (returned false).
- Integration test "refuses a reuse handoff of a source superseded while
its provider resume was pending" fails without the handoff guard and
passes with it.
- Safety rows stay refused: source run still running, workspace export
pending, cleanup in flight, restore into its own allocation, stamp names
another checkpoint, provider work after the backup, two host leases for
one allocation, claim lost to another run, and a resumed lease that lost
its own state. Existing destroyed-source and restore-hold rows are
unchanged.
- `cd server && npx vitest run
src/services/native-runtime/native-session-executor.test.ts
src/services/native-runtime/native-retired-backup-source.integration.test.ts
src/services/native-runtime/remote-warm-session-recovery.test.ts` passes
(1408 tests).
- `npx tsc --noEmit -p server/tsconfig.json` passes.
- `pnpm verify` exits 2 because a global CI input
(`doc/execution-semantics.md`) changed. Required CI supplies the broad
coverage.

## Risks

- Previous-release story: no persisted shape changes. Lease rows saved
by earlier releases carry forward. The new
`reusableSandboxLeaseSuperseded` metadata key is additive.
- The idle source is destroyed at the next acquisition for its
workspace. Its state is the verified backup, which the restored lease
now holds, so no work is lost.
- Not covered: when the source cannot be proven retired or idle (for
example a live owner or later provider work), the run still fails closed
with the same named code. An automatic fresh-session fallback for that
case is not in this PR.
- The root trigger (reuse skips a sandbox while its export saves, then
waits for that export) is unchanged. This PR does not change the reuse
lookup.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (Anthropic), model ID `claude-opus-5-5`, with tool use
and code execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 14:11:10 -06:00
DottaandPaperclip 76c787fffc fix(runner): report monitor notes length limits (#20604) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Agents use task monitors to resume a future check.
> - The monitor schema limits notes to 500 characters.
> - The field description did not state the limit. Rejection showed only
the schema path.
> - This pull request adds the limit to the description and to safe
length errors.
> - Agents can correct notes before they retry a rejected call.

## Linked Issues or Issue Description

Related prior fix: #16868.

**What happened?**

A native `set_task_monitor` call with 501 characters in `monitor.notes`
failed with a path that ended in `notes/maxLength`. The error did not
state 500. The schema already had `maxLength: 500`, but the field
description did not state it.

**Expected behavior**

State the limit in the field description and in the rejection. Accept
500 characters. Reject 501 characters before dispatch. Preserve a saved
monitor when a replacement is invalid.

**Steps to reproduce**

1. Supply a future `monitor.nextCheckAt` and 501 characters in
`monitor.notes`.
2. Read the native provider error.
3. Before this fix, it shows the constraint path without the maximum.

**Paperclip version or commit**

Reproduced on `7368a0b15c2f02107778a961144d86e2eb59a952` in a Linux
source checkout, with Node 25.9.0 and Rust 1.97.1.

## What Changed

- Use one local value for the published notes limit and its description.
The server test checks agreement with the persisted shared validator.
Keep the standalone Runner package boundary.
- Read the numeric maximum from the authorized JSON Schema validation
error. Keep submitted values private and preserve the 512-byte feedback
bound.
- Test the published schema and real bridge dispatch path at 500 and 501
characters. Test saved contents and rejection without monitor changes
with real PostgreSQL.
- Refresh generated contracts and document the limit.
- Sync two SDK proof lock identities after the reviewed master lock
refresh in #20615. This fixes the remaining CI proof mismatch. Keep
compiler, parser, Zod, and all rejection checks unchanged.

## Verification

- Final head: `bac60234ab3a8fc7043a90c7f9c81b02e250428b`.
- Red: the server regression failed because the field description
omitted `500`. The Rust regression failed because the error omitted
`maximum 500 characters`.
- Local passes after the final base update: 44 PostgreSQL
monitor/scheduler tests, 14 real-source Rust bridge tests, 11
cache/Docker guard tests, and 56 SDK proof tests. Runner TypeScript
build and generated contract checks pass.
- The focused Rust check compiles the actual source in a small Cargo
workspace. The full Rust CI suites also pass on Linux and Windows.
- Final-head [PR CI
run](https://github.com/paperclipai/paperclip/actions/runs/42881633978)
passes. This includes typecheck, build, Runner support, Runner Rust and
Vitest, server tests, serialized tests, and all eight e2e shards.
- Final-head [Windows Rust
run](https://github.com/paperclipai/paperclip/actions/runs/42881633419)
passes.
- Greptile: 5/5 on the final head, with no open findings. There are 61
successful checks and five conditional skips. The separate Paperclip
Review remains queued; its comment says work is paused. No CI or test
check is pending or failed.
- `pnpm verify` returned exit 2 and selected broader CI. The full local
suite was not run. Final-head CI supplies that coverage. Earlier failed
attempts were repaired and are not counted as passes.

## Risks

- Low risk. The existing 500-character bound, scheduling semantics, and
permissions stay in place.
- Length feedback uses only a schema path and numeric bound. It does not
print submitted text.
- The Runner binary and tool contracts must reach an instance for the
new guidance to apply. This PR does not deploy or merge.

## Model Used

OpenAI Codex, GPT-6, with reasoning, tool use, and code execution. The
runtime does not expose a more specific model ID or context window size.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
canary/v2026.1110.0-canary.17 		2026-11-10 13:59:58 -06:00
DottaandClaude Opus 5.5 38944cff1a fix(environments): retry and safely replace a reusable sandbox that will not resume (#20626) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - A task that runs on a reusable (warm) cloud sandbox resumes the same
sandbox on the next wake
> - When a follow-up arrives while the provider is auto-stopping or
archiving the idle sandbox, the resume can fail
> - Each failed attempt waited the provider's whole five-minute start
deadline and then failed with `reusable_sandbox_resume_failed`, even
when the workspace was saved and nothing lived only on the sandbox
> - Two such runs in a row made a user wait for many minutes and
recorded two failed runs
> - This pull request bounds the resume, retries a state change inside
the run, and replaces the sandbox from the saved snapshot when that is
safe
> - The benefit is that recovery works automatically and the first turn
is not delayed by minutes

## Linked Issues or Issue Description

**What happened?**
Two consecutive runs failed before provider dispatch with: `Reusable
sandbox lease "<id>" could not be resumed (cause=provider_error); the
lease was preserved and no replacement was created.` Each run took about
5.5 minutes (320s and 330s) in preparation. The third run resumed the
same sandbox from `stopped` in 4.7 seconds after the provider settled.
The previous run had ended with a clean final save and a verified
harness backup.

**Steps to reproduce**
1. Let a task's reusable sandbox go idle so that the provider auto-stops
or archives it.
2. Send a follow-up while the provider still reports the sandbox as
changing state.
3. The Daytona worker waits in `ensureSandboxStarted` until its 300s
start deadline and throws "is still <state> at its start deadline". The
host does not classify that as transient and it has no safe fallback, so
the run fails.

**Expected behavior**
The run retries briefly. If the sandbox still cannot start and nothing
lives only on it, the run continues on a replacement sandbox that
restores the saved workspace.

**Root cause**
`server/src/services/environment-runtime.ts` (reusable resume block,
about line 1820): the resume deadline was the provider's full start
deadline (300s plus RPC margin), the transient classifier did not
recognize a state transition, and the catch always threw
`ReusableSandboxResumeError`. #20212 waits out a state change inside the
same 300s start deadline, so it covers the start but not the time budget
or the fallback.

## What Changed

- Add `server/src/services/reusable-sandbox-resume-replacement.ts`:
- `REUSABLE_SANDBOX_RESUME_BUDGET_MS` (90s): the total resume time for
one run, retries included.
- `sandboxStateTransitionFailure`: "state change in progress" and "is
still <state> at its start deadline" are transient.
- `retireUnresumableReusableSandboxLease`: replace only when the lease
is `released`/`retained` with no live owner, the task workspace is clean
with a committed saved snapshot and no other writer, and a
`paperclip_runner` session has a verified lease-bound harness backup.
Access refusals (401/403) and an unavailable worker stay strict. After
the replacement lease (which carries the harness backup) is recorded,
the old lease moves to `pending_cleanup` (reason `resume_failed`) in one
guarded write, so the existing cleanup sweep destroys the old sandbox
and no provider delete runs before the first turn. A failed creation or
a restart before that write leaves the old lease `retained` for the next
attempt.
- `environment-runtime.ts`: cap the resume deadline at the budget, treat
a state transition as transient, and call the replacement policy before
failing. A replacement records `sandboxLeaseAcquisition: { outcome:
"replacement", reason: "resume_failed" }`; workspace preparation then
restores the saved snapshot.
- `remote-workspace-transport.ts`: add the closed failure label
`sandbox_state_transition`, so a strict failure names the cause instead
of `provider_error`.
- `doc/execution-semantics.md`: document the budget, the in-run retry,
the replacement conditions, and the strict cases.
- Update the two slow-start tests: the resume RPC timeout is now 90s
instead of 330s/150s. A 61s start still resumes the same sandbox.

## Verification

- Red before the fix (new file
`server/src/__tests__/environment-runtime-reusable-resume-replacement.test.ts`,
staging shape: lease created by the real acquisition path, then retained
after a clean save with a backup stamp, 300s provider deadline, worker
refusal with no status): 3 tests failed with `could not be resumed
(cause=provider_error); the lease was preserved and no replacement was
created.`
- Green after the fix: 10/10 pass, including a case where the
replacement creation fails and the next run replaces the sandbox with
the backup carried forward. The safe case replaces the sandbox, keeps
every resume RPC at or below 90s, makes no synchronous destroy call, and
moves the old lease to `pending_cleanup`. A single state-change refusal
resumes the same sandbox inside the run. Six strict cases keep the lease
and fail with `cause=sandbox_state_transition`: unsaved work, no saved
version, an uncommitted saved version, a live owner, no verified backup,
and a 403.
- `pnpm exec vitest run src/__tests__/environment-runtime.test.ts` (197
pass), `heartbeat-retry-scheduling`, `legacy-execution-recovery`,
`remote-workspace-transport` (251 pass).
- `tsc -p server/tsconfig.json --noEmit` passes.
- `pnpm verify` exits 2 (a doc change is a global CI input). Required CI
supplies the broad coverage.
- Not run: a live Daytona run on staging. The provider message is kept
off the run record, so the exact transition state on staging (stopping
or archiving) is inferred from timing, not observed.

## Risks

- A strict resume now gives up after 90s instead of 300s. The provider
start continues in the background, and the bounded bootstrap retry
resumes the same sandbox. A stopped sandbox resumes in seconds and an
archived one in about 25s on Daytona, so 90s leaves headroom.
- A replacement leaves the old sandbox to the cleanup sweep. If the old
start finishes later, the sweep still destroys it. The guarded write
only claims a `released`/`retained` lease with no live owner.
- Upgrade safety: no new persisted fields are required. Leases saved by
older releases are read with the same rule; one without a verified
backup stays strict. The replacement reason `resume_failed` already
exists in the acquisition outcome.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:51:10 -06:00
Devin FoleyandPaperclip 8148fc0b1e fix(release): fetch draft-notes PR summaries in batches (#20602) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The release workflow drafts the next stable's release notes when a
beta publishes (`draft_stable_notes`)
> - The draft nests each referenced PR's summary under its commit
subject, and it fetches each summary with its own `gh pr view` call
> - A beta range now references more than 1,500 PRs, so these serial
calls do not finish inside the job's 10-minute timeout
> - This pull request fetches the summaries in batches of 50 per GraphQL
query
> - The benefit is that the job finishes in under a minute and pushes
the notes branch again

## Linked Issues or Issue Description

**What happened?**

The `draft_stable_notes` job of [run
42763789753](https://github.com/paperclipai/paperclip/actions/runs/42763789753)
(beta `2026.1110.0-beta.0`) reached its 10-minute timeout. GitHub
cancelled it during "Draft stable notes from the published beta", while
a `gh` process was still running. The beta itself published and its
smoke tests passed. The `release-notes/v2026.1110.0-beta.0` branch was
not pushed.

**Expected behavior**

The job writes the notes draft and pushes the branch for every beta,
whatever the size of the commit range.

**Steps to reproduce**

1. Publish a beta whose range from the last stable has many
PR-referencing commits. This range has 1,586 commits, and 1,583
different PRs.
2. `draft-stable-notes.sh` calls `gh pr view <n>` once for each commit,
one after another.
3. The job reaches `timeout-minutes: 10` before the loop ends.

**Paperclip version or commit**

`09e24dd12c36251dd12af2d73dbc715ce990f922` (beta `2026.1110.0-beta.0`).
This was the first beta to run the job after #12124 added the PR
summaries; earlier betas skipped it.

## What Changed

- `scripts/draft-stable-notes.sh`:
- Collects the unique PR numbers first. Fetches their bodies 50 per `gh
api graphql` query (aliased `pullRequest(number:)` fields), into a
temporary cache that `enrich_pr` reads.
- Parses the raw response with `jq`, not with `gh --jq`. When any number
in a batch is not a PR (for example, an issue reference), gh exits
non-zero and does not apply its `--jq` filter. The other PRs in the
batch still have usable data.
- The output format does not change. A missing `gh` or `jq`, a failed
batch, a cache directory that `mktemp` cannot create, or
`DRAFT_NOTES_SKIP_PR_ENRICHMENT=1` still gives bare subjects.
- No code comments are added (per `CONTRIBUTING.md`); this description
holds the explanation.
- `scripts/draft-stable-notes.test.mjs`:
- The enrichment tests now use a fake `gh` that answers GraphQL like
GitHub.
- New tests: batching (120 PRs make 3 queries, and a repeated PR is
fetched once); a batch with a non-PR number; a failed batch; a failed
`mktemp`; prose-only and empty bodies; the opt-out.
- Range tests now run with enrichment disabled, so an authenticated `gh`
on the test machine cannot reach GitHub.
- `package.json`: adds the test file to `test:release-registry`. These
tests did not run in CI before.

## Verification

- `node --test scripts/draft-stable-notes.test.mjs`: 17 pass. They also
pass with macOS bash 3.2 first on `PATH`.
- `node scripts/ci-profile/node-test.mjs
scripts/draft-stable-notes.test.mjs scripts/release-lib.test.mjs`: 31
pass.
- Red check: the new tests against the old script fail 5 cases (all
batching and enrichment cases). The `mktemp` test fails against an
unguarded `mktemp` call.
- Live run on the real range: `./scripts/draft-stable-notes.sh
2026.1110.0-beta.0` finished in 47 seconds. It wrote 4,596 excerpt lines
for 1,583 PRs.
- Parity: for 40 randomly sampled PRs from that range, the new excerpts
are byte-identical to the old per-PR `gh pr view` output.
- `shellcheck`: no new findings. The two remaining notices were there
before this change.

## Risks

- Low. The change touches only the post-publish draft job, and every
failure still gives the bare-subject draft.
- `jq` is now needed for the PR summaries. GitHub's Ubuntu runners
include it. Without `jq`, the draft has no summaries, the same as
without `gh`.
- I did not change `release.yml`. The job already checks out `master`,
so after this merges, re-running the failed job of run 42763789753
should push the missing notes branch with the fixed script.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, test runs, read-only GitHub API calls).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 11:48:55 -08:00
Devin FoleyandPaperclip d2832a5a0b fix(runner): keep redacted display events inside their schema bounds (#20607) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. Its daemon, runnerd,
stores provider events durably before the server receives them
> - Before storing rich display events, such as tool output and provider
notices, runnerd redacts credential-shaped values. It then checks the
result against the event's protocol schema
> - Redaction can make a string longer. A field already at its schema
limit then goes over the limit, runnerd rejects the event, and the
daemon exits
> - The event is replayed after each restart, so every replacement
runner exits the same way. The run loops until something outside it
stops the run
> - This pull request trims a field that redaction pushed past its
schema limit, in both places that redact rich display events, so the
event is accepted and the runner keeps working
> - The benefit is that long tool output that contains a secret no
longer crashes the runner

## Linked Issues or Issue Description

**What happened?**

An agent run read GitHub Actions output and, after a context compaction,
entered a loop. About every 14 seconds runnerd started, restored its
durable PRP session, and exited with `paperclip-runnerd: redacted
display activity exceeds its schema bounds`. This continued for 15
minutes (62 launches) until a server restart ended the run. Every
reattach after that failed the same way, so the task was blocked.

**Expected behavior**

Redaction never makes a valid display event invalid. runnerd stores the
redacted event and continues.

**Steps to reproduce**

1. Produce a tool output longer than 64 KiB that contains a
credential-shaped pair, for example a GitHub Actions log line
`GITHUB_TOKEN: ***`. The ACPX sidecar keeps its last 64 KiB, which is
exactly the `toolExecution.output` limit of 65,536 characters.
2. runnerd redacts the pair to `GITHUB_TOKEN: [REDACTED]`, which is
longer, so `output` now has more than 65,536 characters.
3. `preserve_bounded_display_content` returns `redacted display activity
exceeds its schema bounds`, and runnerd exits.

A probe test with 64 KiB of output that contains `GITHUB_TOKEN: ***`,
`token=abc`, `Authorization: Bearer x`, or `password: hunter2` gets
exactly this error on `master`.

**Paperclip version or commit**

`master` at 491dfa1c6d.

## What Changed

- `acpx_event_payload.rs`: new `fit_redacted_display_to_length_bounds`.
- It asks the rich-event validator which fields fail `maxLength`, and
trims each one to its limit.
- `output` keeps its tail, like the sidecar's own truncation, and sets
`outputTruncated: true` on tool execution events. Other fields keep
their start.
- Fields within their limits do not change. A payload that still fails
its schema for another reason (type, pattern, or enum) is rejected as
before.
- Both places that redact rich display events call it before their
post-redaction schema check:
  - `decode_rich_event`, for ACPX `runtime.rich_event` payloads;
- `preserve_bounded_display_content` in `durable/state.rs`, before
durable storage.
- A new `tests/acpx_rich_events.rs` test decodes a 60,000-character
`plan.proposed` (Claude ExitPlanMode) with `token=abc`, and a
65,536-character tool output, through `decode_acpx_event`.
- Three new tests in `durable/tests/redaction_display_bounds.rs`,
included by the `durable/state.rs` test module:
- A full-length tool output stays within its bound for each of the four
credential shapes. It is the tail of the redacted text, no secret
remains, `outputTruncated` is set, and the stored event passes its
schema.
- A provider notice `summary` at its 4,000-character limit keeps its
start and gets no `outputTruncated` field.
- A short tool output that redaction makes longer, but still within its
limit, is not trimmed.

## Verification

- `cargo fmt --all -- --check`: clean.
- `cargo check --locked --workspace`: passes, with no new warnings.
- `cargo test -p paperclip-runner-core --lib -- redaction_
durable_display_content keeps_a_long_plan governance_display`: 16 pass,
including the three new tests.
- `cargo test -p paperclip-runner-core --test acpx_rich_events --test
acpx_event_payload`: 6 and 9 pass. Without the decoder call, the new
decoder test fails with `ACPX rich event redaction exceeded its schema
bounds`.
- Red check: before the fix, the probe described above failed with
`redacted display activity exceeds its schema bounds` for all four
credential shapes.
- Full `cargo test -p paperclip-runner-core --lib`: 620 pass and 2 fail.
The 2 failures are
`codex_provider::tests::fenced_receipt_output_reports_capacity_without_dropping_known_frames`
and `fenced_tool_rejection_keeps_identity_and_real_delivery_strict`.
They fail only under the full parallel run. In isolation they pass on
`master` and pass 3 of 3 times with this change.

## Risks

- Low. Only payloads that runnerd rejects today change, and those
payloads stop the runner. A trimmed display field loses some characters
at the limit. The tail of tool output is kept and `outputTruncated` is
set.
- Redaction still runs before trimming. Trimming only removes
characters, so it cannot expose a secret.
- Companion pull request #20608 bounds the transport's restore loop, so
any other event that runnerd rejects fails the run quickly instead of
looping.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, cargo builds and tests, read-only log analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 11:48:27 -08:00
Devin FoleyandPaperclip 4f8dd0e25a fix(runner): stop relaunching a runner that keeps exiting after restore (#20608) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Agent runs execute through Paperclip Runner. The server-side
transport supervises the runnerd process and restarts it when it exits
unexpectedly
> - That recovery keeps launching replacements until one restores the
durable PRP session, within a grace window
> - Each successful restore starts a fresh grace window. So a runner
that restores and then exits again a few seconds later loops with no
limit
> - In a real run, a deterministic runnerd crash looped this way 62
times in 15 minutes, with no error recorded, until a server restart
ended it
> - This pull request fails the transport after 3 restored runners each
exit within a minute of their restore. The error carries the runner's
last diagnostic
> - The benefit is that a crashing runner fails the run in under a
minute, with the reason recorded, instead of looping silently

## Linked Issues or Issue Description

**What happened?**

After runnerd exited unexpectedly, `#recoverRunnerProcess` launched a
replacement. The replacement authenticated ("runner process restored its
durable PRP session") and then exited about 5 seconds later with the
same diagnostic. `#watchRunner` then started a new recovery with a new
420-second grace window. This repeated every ~14 seconds for 15 minutes,
until the server shut down. The run produced no output and recorded no
failure during that time.

**Expected behavior**

A runner that keeps exiting right after each restore fails the transport
quickly. The failure names the runner's own diagnostic.

**Steps to reproduce**

1. Start a transport with `runnerReconnectGraceMs` set.
2. Kill runnerd. Recovery launches a replacement, which restores the
session.
3. Kill each restored replacement within a few seconds. On `master`,
recovery relaunches every time; the new test shows no failure after 3
restored runners exit.

**Paperclip version or commit**

`master` at 491dfa1c6d.

## What Changed

- `packages/paperclip-runner/src/live/runnerd-codex-transport.ts`:
  - Records the time of each successful restore.
- Both runner-exit paths in `#watchRunner` (exit and completion
rejection) now go through `#recoverOrFailRunnerProcess`. It counts an
exit that comes within `RUNNER_RESTORE_STABLE_MS` (60 s) of the last
restore. A runner that stays up longer than that resets the count.
- On the third such exit (`MAX_UNSTABLE_RUNNER_RESTORES`), it fails the
transport with `native_runner_process_exited: runnerd exited 3 times
within 60000ms of being restored: <last diagnostic>`, and does not
launch again.
- The counting rule is a pure function, `unstableRunnerRestoreCount`,
exposed through `runnerdRecoveryInternals` for tests.
- `runnerd-codex-transport.test.ts`:
- An integration test with the real runnerd: it kills each restored
runner and expects the failure after 3 restores, with exactly 4 launches
and no further relaunch.
- A unit test for the counting rule, including the reset after a stable
runner.

## Verification

- `npx vitest run src/live/runnerd-codex-transport.test.ts -t "keeps
exiting right after|stable window|replacement whose staging"`: 3 pass (2
new, and the existing replacement-restore test).
- Red check: with `master`'s transport and the same tests, the loop test
fails. The transport never reports the failure and keeps relaunching.
- Full `npx vitest run src/live/runnerd-codex-transport.test.ts`: 296
pass.
- `npx tsc -p tsconfig.json --noEmit` in `packages/paperclip-runner`
(after `pnpm run ensure:eval-build-deps`): clean.

## Risks

- A runner that crashes repeatedly right after restoring now fails the
run after 3 restores instead of retrying indefinitely. The server's own
recovery then handles the failed run as it does any other
`native_runner_process_exited` failure.
- A healthy runner that restores and stays up longer than 60 seconds is
unaffected, and a later exit starts a new count.
- A companion pull request, #20607, removes the runnerd crash that
caused the observed loop.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`) via Claude Code, with tool use
(shell, file editing, vitest and cargo runs, read-only log analysis).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [x] All Paperclip CI gates are green
- [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 11:48:21 -08:00
DottaandClaude 90dd864bff test(restart-chaos): wait for any visible Retry, not the first match (#20616) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The restart chaos harness checks that a task never needs a human
Retry after a control-plane restart, and that every Retry the board
shows works
> - When a task rests with a Retry offer, the harness opens the task
page and clicks the Retry button like a user
> - The harness waited for the first `Retry` button in document order to
become visible
> - A page can hold an earlier `Retry` button that is in the
accessibility tree but has no box. The harness then waited on that
button for the full timeout and reported `ui_retry_button_missing`,
although the user saw a Retry in the status banner
> - This pull request makes the harness wait for any visible Retry
button, banner first
> - The benefit is that a chaos grade reports a missing Retry only when
the user cannot see one

## Linked Issues or Issue Description

**What happened?**
A Daytona chaos wave graded one task `ui_retry_button_missing`. The
captured status banner text was "Couldn't finish this attempt. Your
message is saved. Retry", so the page showed a Retry button. The harness
waited on `getByRole("button", { name: "Retry", exact: true }).first()`.
That locator selects the first match in document order. When that match
is not visible, the wait times out even when another Retry is visible.
The click loop after the wait already used only visible buttons, but the
wait itself did not.

**Expected behavior**
The harness reports `ui_retry_button_missing` only when no visible Retry
button exists before the timeout. When a visible Retry exists, the
harness clicks it, banner first.

**Steps to reproduce**
1. Serve a page with a zero-size `Retry` button before a visible status
banner that holds a second `Retry` button.
2. Call `BoardUi.clickRetry` for that page.
3. Before this change, the result is `ui_retry_button_missing`. After
it, the harness clicks the banner Retry and sees the retry request.

Separate finding, not changed here: in that wave the Retry was also a
dead end on the server side. The run had failed with
`native_session_cleanup_quarantined`. The server build that ran the wave
still projected `canRetry: true` for that run, and the retry route
refuses that code with 409. Current master already removes that Retry
offer (#20572).

## What Changed

- `tests/restart-chaos/board-ui.ts`: wait on the `Retry` role locator
filtered to visible elements (`filter({ visible: true })`), instead of
the first match in document order. After the wait, re-query the visible
buttons and click them banner first, as before.
- `tests/restart-chaos/board-ui.test.ts`: add a page with an earlier
zero-size `Retry` and a visible banner `Retry`. The test fails before
the change (`ui_retry_button_missing`) and passes after it.

## Verification

- `pnpm exec vitest run --config tests/restart-chaos/vitest.config.ts
tests/restart-chaos/board-ui.test.ts
tests/restart-chaos/triggers.test.ts`: 43 passed. With the old
`board-ui.ts`, the new test fails with `ui_retry_button_missing`.
- `pnpm exec tsc --noEmit -p tests/restart-chaos`: passes.

## Risks

- Low risk. The change is test-harness code only. No product code
changes.
- The timeout and the "missing" result do not change. The locator
`visible` filter needs Playwright 1.51 or later; the repo uses 1.62.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, used through Claude
Code with tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 13:40:31 -06:00
DottaandClaude Opus 5.5 28063474d2 fix(ui): show why a queued message edit was not saved (#20583) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - In task chat, a person can edit a queued message before the agent
receives it
> - The server rejects an edit with 409 when the queue changed in
another session
> - The composer kept the text, but it showed no reason, so the edit
looked stuck
> - This pull request shows the reason under the editor
> - The benefit is that the person knows the edit was not saved and what
to do next

## Linked Issues or Issue Description

**What happened**
Two sessions edited the same queued message. The older save got `409
queued_comment_revision_conflict`. The composer still said "Editing
queued message" and showed no error.

**Expected behavior**
The composer tells the person that the queue changed in another session,
keeps the text, and lets them save again.

**Steps to reproduce**
1. Queue a message on a task with a running agent.
2. Open the task in two tabs and start editing the queued message in
both.
3. Save in the second tab, then save in the first tab.
4. See no error in the first tab.

## What Changed

- `TaskChatComposer` shows a save failure for a queued message edit in
its existing `role="alert"` line
(`data-testid="task-chat-queued-edit-error"`). The error clears when the
edit ends or a new save starts.
- `api/errors.ts` has copy for `queued_comment_revision_conflict`,
`queued_comment_stale_queue`, `queued_comment_stale_target` and
`queued_comment_not_pending`.
- The existing queue refresh after a 409 updates the revision, so a
second save works.

## Verification

- Red then green: `cd ui && npx vitest run
src/components/task-chat/TaskChatComposer.test.tsx` (new test `shows why
a concurrent queue change rejected the save and keeps the edit` fails on
`master`; all 146 pass with the change).
- `npx vitest run src/components/TaskChatThread.test.tsx
src/api/errors.test.ts` (479 passed).
- `pnpm check:token-gates`: all gates clean.
- `tsc --noEmit -p ui`: no errors outside unrelated Storybook files.

## Risks

- Low risk. UI copy only, no new API. A stale edit now shows a short
error line in addition to the "Queued message changed" label.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:37:31 -06:00
DottaandClaude 070d6f83fd perf(realtime): share permission reads among queued live events (#20624) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - A task page shows the output of a running agent as it is produced.
> - The server sends each output event to the browser on the company
WebSocket.
> - Before each send, the server checks permissions and redacts secrets.
Each event did its own sequential database reads.
> - When output arrives faster than those reads complete, the queue
grows. The text reaches the browser later and later, and the server then
closes the connection.
> - This pull request lets the events that are already in the queue
share one set of reads.
> - The benefit is that live output keeps pace with the run, and a
revoked permission still applies to the next event.

## Linked Issues or Issue Description

**What happened?**

On a hosted instance, the live output of a running task sometimes
appeared very slowly. A probe WebSocket on the company event stream
received events 0.5 seconds after their creation time at the start.
After 30 seconds the delay was 23 seconds. The server delivered about 6
events each second. Then it closed the socket with code 1013.

The delivery loop for a connection is serial. For each transcript event
it ran a membership read, a scope decision, a run lookup and a task read
decision. Native run events added a settings read and a secret registry
read. A burst of 200 events issued 1,401 SQL statements.

**Expected behavior**

Live output reaches the browser at the rate the run produces it. The
cost of the permission checks does not grow with the number of events in
a burst. A revoked permission stops delivery immediately.

**Steps to reproduce**

1. Use a deployment that authenticates users, with a database that has a
network round trip.
2. Start several agent runs that produce output at the same time.
3. Open the task page of one run and read the live output.
4. Compare the creation time of each `heartbeat.run.log` or
`heartbeat.run.event` message with its arrival time.
5. The delay grows continuously until the server closes the socket with
code 1013.

**Paperclip version or commit**

`master` at 79e28dc0d0.

## What Changed

- `live-event-delivery.ts` gives each queued event a sequence number. It
passes a shared read set to `authorize`. A set serves only the events
that were in the queue when the set opened. The delivery replaces the
set after one second and discards it when the queue is empty.
- `sharedDeliveryRead` returns the pending read for a key, or starts it.
It removes a read that fails, so the next event reads again.
- `live-events-ws.ts` uses the shared set for the membership or key
check, the transcript scope decision, the run binding and the task read
decision.
- `live-run-event-redaction.ts` uses the shared set for the username
setting and the registered secret values of the run.
`run-secret-redaction.ts` adds `redactorForRun`, which reads the values
one time and returns a redactor.
- `doc/live-transcript-delivery.md` describes the shared reads and their
limits.

## Verification

- `npx vitest run src/realtime/live-event-delivery.test.ts
src/realtime/live-event-delivery.cost.test.ts
src/__tests__/live-events-run-read.test.ts
src/__tests__/live-events-ws.test.ts
src/__tests__/live-events-toast.test.ts` in `server/`. All tests pass.
- New integration test against PostgreSQL: 200 transcript events in one
burst. All events arrive in order. The burst issues at most 40
statements. Without the shared reads the same test fails with 1,401
statements. After the grant is revoked, the next burst delivers nothing.
- New unit tests: 39 events that wait behind one read share the next
read. An event that arrives during a read starts a new read. A set is
replaced after one second. A failed read is not shared.
- The existing tests for revocation, deleted tasks, restricted agent
keys and live redaction pass without changes.
- Assumption tested: an event never uses a read that began before the
event entered the queue. The unit test holds a read open and adds an
event to prove it.
- Not verified: the delay on a hosted instance after deployment. The
measurement above is from before this change.

## Risks

- Permission freshness. Before, each event had its own check. Now a
check can serve events that were already in the queue, for at most one
second. A revocation applies to the first event published after it.
Events already in the queue at the time of the check can still be sent.
That window existed before between a check and its send.
- Secret values. A read set holds decrypted registry values while a
burst drains. The delivery discards the set when the queue is empty and
when the connection closes.
- Plugin run streams use the same redactor and get the same shared
reads.
- No schema, API or protocol change.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude, model ID `claude-fable-5-2`, in Claude Code with
tool use and code execution.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude <noreply@anthropic.com>
 2026-11-10 13:37:25 -06:00
DottaandClaude Opus 5.5 5f34f777ec fix(evals): grade 16 chat cases by saved state instead of reply keywords (#20600) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The chat-feature eval catalog checks that a chat agent does the work
a person asks for
> - In the latest full-cohort run on master, 16 cases failed although
the saved work was correct
> - Their graders matched words, one literal value, or one API route in
free-form model output
> - A grader that fails correct work hides real failures in the noise,
and it can also pass wrong work that uses the expected word
> - This pull request grades those cases by saved state, audit records,
setup literals and rendered page structure, and keeps only narrow checks
for false claims
> - The benefit is that these cases fail only when the product or agent
does the wrong thing

## Linked Issues or Issue Description

Refs #18971 (related: company-secret question handoffs)

**What happened**

The full-cohort chat-eval run on master failed 16 cases where the
retained evidence shows correct behaviour. Examples:
`companies.rename-company` accepted only `company.updated`, but the
supported branding route logs `company.branding_updated`.
`recovery.post-stop-message` required a signup button label that matched
a regex, and rejected a labelled email form with the button "Start
writing free". `secrets.secret-proposal` flagged "the value is delivered
when Morgan next runs" as a false delivery claim. Other graders required
words such as "uncommitted", "telemetry", "administrator" or the app
name that the question did not ask for.

**Expected behavior**

A grader accepts every supported route and wording that achieves the
outcome. It fails when saved state is wrong, when a requested fact is
wrong, or when the reply makes a claim that saved state contradicts.

**Steps to reproduce**

1. Load the retained observation for one of the cases below from the
full-cohort evidence.
2. Grade it with the old oracle. The keyword or route check fails
although the saved state is correct.
3. Grade it with the new oracle. It passes. A broken variant (wrong
saved state or a contradicting claim) still fails.

## What Changed

- `activity.audit-change`: attribution is graded by structure. The reply
names the recorded editor, names no other company agent (new setup
literal `otherActors`), and states only real transitions. Sentence
grammar is no longer matched.
- `app-permissions.ask-before-action`: each continuation reply must
exist and must link no app record other than the reviewed connection. It
no longer must name the app.
- `budgets-costs.spend-breakdown`: the reply must point at the record
with unresolved usage (task link, identifier or run id) and must not
give it a zero or free amount. The words "subscription" and "incomplete"
are no longer required.
- `cases.browse-cases`: an excluded case may be mentioned in prose. It
must not be linked or listed as a result row.
- `cases.handoff-same-task`: handoff instructions carry findings forward
when they point at the saved `findings` document, reproduce it, or
restate it. The premise (a primary Morgan run performs the handoff) is
unchanged.
- `cases.update-case-facts`: a saved browser value may add clauses that
only rule out other platforms ("desktop Safari unaffected").
- `companies.agent-observability`: the reply no longer must name
Telemetry and the run log. It must not claim that they are exported.
- `companies.rename-company`: both supported name routes
(`company.updated`, `company.branding_updated`) are accepted.
- `connection-setup.choose-provider`: the answer must exist and must not
claim a connected account. The unavailable method no longer must be
named; the setup path stays graded by clicking the card or link.
- `goals.align-existing-work`: a question card from the answering run
counts as the phase-two response. Refusal words and "unchanged" words
are no longer required; false change claims still fail.
- `recovery.post-stop-message` (and `steering.interrupt-with-direction`,
which shares the helper): a visible form with a labelled field and a
submit control is a signup action, whatever the button says.
- `routines.pause-owned`: the "paused"/"running" words are no longer
required. The saved checkpoint proves both; a cancellation claim still
fails.
- `secrets.secret-proposal` (and `secrets.configure-company-secret`,
which shares the helper): a sentence that says when delivery will happen
is not a delivery claim.
- `skills.explicit-restriction`: the reply no longer must say
"administrator". Success claims and invented tool limitations still
fail.
- `workspaces.inspect-git`: the reply no longer must say "uncommitted".
Unchanged HEAD and status prove it; commit or push claims still fail.
- Every affected `oracleVersion` is bumped in code and in
`chat-catalog-descriptors.json`. No scenario YAML changed.

## Verification

- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
--testTimeout=180000 tests/runner-e2e/chat-catalog-domains
tests/runner-e2e/chat-catalog-contract.test.ts
tests/runner-e2e/chat-catalog-consistency.test.ts`: all pass.
- `PAPERCLIP_CHAT_EVAL_RENDERED_ARTIFACT_CANARY=1` rendered-artifact
canary with Chromium: pass, including new form cases.
- `node --test
tests/runner-e2e/chat-full-catalog-testops-manifest.test.mjs`: pass.
`node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --check`:
unchanged pins.
- `npx tsc --noEmit -p tests/runner-e2e`: no errors in changed files.
- Offline calibration on retained evidence, for each case: the retained
correct evidence passes, earlier passing evidence passes where it
exists, and a broken variant fails. `spend-breakdown` retained evidence
still fails, because its reply shows unresolved usage as `$0.00`.
`handoff-same-task` retained evidence still fails on the kept
primary-run premise.
- Live paid runs: see the comment below.

## Risks

- Low risk. The change is limited to eval graders and their unit tests.
Product code does not change.
- Each grader is more accepting of wording. Saved-state checks and the
narrow false-claim checks remain, so a wrong outcome still fails.
- `secrets.configure-company-secret` still fails its retained evidence.
Its agent asked a two-question card, and the act supports only
one-question cards. This pull request changes it only by the shared
delivery-claim helper.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), tool use and code
execution in Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:37:16 -06:00
DottaandPaperclip 89ec4e7614 feat(links): add cached GitHub pull request previews (#20593) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work.
> - Task conversations contain links to pull requests and other work.
> - A URL alone cannot tell the operator whether a PR is open, merged,
or blocked by conflicts.
> - Fetching every link at render time wastes API calls and can expose
private data through a shared cache.
> - This pull request adds a demand-driven links service and a GitHub PR
adapter.
> - Each viewer and company has a separate cache. Repeated links share
one request and result.
> - Operators get context when they hover or focus a link. Normal links
stay useful when a provider is unavailable.

## Linked Issues or Issue Description

**Subsystem affected**

Task Markdown links, shared link metadata, and the server GitHub API
boundary.

**Problem or motivation**

PR links have compact labels and neutral icons. They do not provide a
rich hover card or a common status source across repeated mentions. A
background resolver cannot borrow a person's GitHub connection because
it has no viewer identity.

**Proposed solution**

Recognize supported links locally. Request metadata after a delayed
hover or keyboard focus. Show a card at the first line of the link,
aligned with its icon. Share the result across all mentions in the
current viewer and company. Keep the exact original navigation URL.

The first adapter supports GitHub pull requests. Open is green, merged
is purple, and merge conflicts are orange. Draft, closed, unknown,
loading and unavailable states remain quiet.

**Alternatives considered**

Eager page scans would spend API quota on links the operator does not
inspect. Company-wide caching of personal GitHub results would cross the
credential boundary. This change uses neither approach.

**Roadmap alignment**

ROADMAP.md was checked. This extends existing task links and the GitHub
resolver. It adds no deployment adapter or new background worker.
Searches of open and closed PRs for link previews and PR hover cards
found no matching implementation.

## What Changed

- Add host-owned link recognition and a normalized preview contract. The
service selects an adapter by provider.
- Add a board-only company read endpoint. It checks company access
before cache reads and sends a private, no-store response.
- Use the current viewer's eligible GitHub connection. A configured
connection that cannot supply credentials fails closed. With no managed
connection, reuse existing company credentials or public access.
Instance credentials retain the public-repository check.
- Reuse the GitHub external-object resolver for API transport and
response parsing. Add title and mergeability to its normalized snapshot
without changing its existing state contract.
- Coalesce requests by company, viewer and canonical PR URL. Bound
cached entries and concurrent work. Cache success for one minute and
failures for a short period or the provider retry window.
- Add an abortable request deadline. Timed-out work stays in the
concurrency budget until it settles. No periodic worker or idle polling
is added.
- Add a shared browser query and anchored preview card. Repeated
Markdown links observe the same result. There are no automatic retries,
window-focus refreshes, toasts or connection prompts.
- Add ten production-component Storybook examples and real
pointer/keyboard browser checks. Document the API and verification
route.

## Verification

- Linux source checkout: 233 focused tests passed in nine suites. They
cover recognition, fixed API destinations, provider states, access
failures, credential selection, viewer/company isolation, request
coalescing, expiry, cache bounds, concurrency bounds and mounted OpenAPI
coverage.
- A live public GitHub smoke returned this PR's correct title and open
state. Two equivalent URLs shared one real network request. Personal
connection access was not part of that smoke.
- A controlled HTTP server stalls a real response body. The test proves
cancellation while another lookup completes.
- UI typecheck and the server layered typecheck passed. The UI
production build passed.
- Token gates and query policy gates passed. All 30 map/catalog support
checks passed. The three new server test files also passed a focused
TypeScript check.
- All 27 focused browser checks passed on the reviewed source tree. They
cover light/dark themes, narrow layouts,
open/merged/conflict/draft/closed/loading/unavailable/failed states,
fixed positioning during pointer movement, exact navigation and keyboard
access. A real browser HTTP interception counts one deferred lookup for
two equivalent PR links.
- The first browser attempt lacked Chromium. A later combined build/test
attempt exhausted the sandbox's 8 GiB memory limit and killed Storybook.
Those attempts supply no pass evidence. Chromium is installed, and the
final browser checks run with the build stopped. One initial Storybook
dependency reload interrupted a streaming check. All eight existing
inline-link cases passed on the warmed server, followed by a complete
27-case pass.
- Synthetic Storybook pointer events did not exercise the native hover
boundary. The assertions now run through real Playwright pointer and
keyboard actions. The fixture stories retain the production components.
- Review fixes preserve the Draft label when GitHub reports both draft
and conflicts. They also preserve neutral link colors in dark mode. The
25 affected service checks pass, including the added combined-state
regression. Browser checks compare unknown, draft and closed links with
the muted theme color.
- The first CI policy run rejected a literal retry opt-out. The code now
uses the existing explicit zero-retry helper. A new test-spy type error
was also corrected. These failures are fixed; their earlier runs remain
failed.
- `pnpm verify --explain` returned exit 2 because the browser test path
selects broad CI. No full local suite is claimed. The final CI run
supplies that coverage.
- The previous head `437913c797963aa113bb82e2ae8b836969317ef9` passed CI
and received a 5/5 Greptile score. All four review threads are resolved.
Master then advanced. The current head
`ae2a1c2adbd61a621cca9bdd160561952613df6e` resolves the source-manifest
conflict against master `9b21dcbdda01296349f3cfe0a3a6891e79df6130`.
Fresh CI and review are running.
- The previous CI run found a lost class on decorated PR links and a
stale SDK source manifest. Both are fixed. The existing CommentThread
integration test and 26 focused link service tests pass. All 45 SDK
source/privacy tests pass. All eight source-manifest helper tests pass.
The manifest helper changed only source identities and added the two new
shared files. Runtime, compiler, parser and library pins stay unchanged.
- A later review found that expired timeouts could reuse a settled
promise. The service now removes settled requests from its
shared-request map and counts unfinished provider work separately. The
new regression failed before the fix and passes after it. The
concurrency-bound and real stalled-HTTP tests also pass.
- UI typecheck and token gates pass after these fixes. The browser rerun
needed Chromium in the new session. That first attempt did not exercise
the UI. The complete 27-case browser rerun passed before the master
merge with retries disabled. All 27 browser cases also passed on the
merge revision with retries disabled.

- After the master merge, all 84 focused link, OpenAPI and integration
tests pass. All 55 SDK source/privacy/manifest tests pass. UI typecheck
and token gates pass again. The source manifest retains master's
reviewed runtime pins and includes both sets of source changes. GitHub
confirms the PR can merge cleanly. The preflight helper rebuilt the
shared package and plugin SDK; Cargo is unavailable locally. Final CI
supplies Rust coverage.

- The latest master update changed the reviewed report-lock policy but
left the SDK lock hash at its previous value. The SDK source/evidence
checks reproduced the mismatch. This change sets only the SDK lock hash
to the authenticated output of the existing master policy and patch.
Compiler, parser and Zod pins stay unchanged. All 56 SDK
source/evidence/manifest tests now pass. All 84 focused
link/API/integration tests, UI typecheck and token gates also pass on
the updated source.

## Risks

- This cache is process-local. Separate server replicas can make
separate GitHub requests.
- A cached result can remain visible for its short lifetime after GitHub
changes. A later hover refreshes expired data.
- GitHub can report unknown mergeability. The UI shows conflicts only
when GitHub explicitly reports them.
- Personal preview data stays in viewer/company caches. It is not
written to company external-object records or persisted browser query
storage.
- Stories simulate the API. They do not prove a live GitHub login or
private-repository access.
- No schema migration is required.

## Model Used

OpenAI Codex, based on GPT-6. The runtime does not expose the exact
serving model ID or context-window limit. Used reasoning, repository
inspection, source editing, shell execution and browser tools. No
subagents were used.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

---------

Co-authored-by: Paperclip <noreply@paperclip.ing>
 2026-11-10 19:32:51 +00:00
DottaandClaude Opus 5.5 161ee3b9a2 fix(runner-e2e): refresh SDK public input pins after lockfile refresh (#20618) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - The Runner support suite checks the SDK public source pins in
`tests/runner-e2e/source-sdk-public-inputs.json`
> - The automated lockfile refresh #20615 changed `pnpm-lock.yaml` and
`tests/runner-e2e/report-lock.json`
> - It did not update the two lockfile values in the SDK pin registry
> - So `ci / Runner support suite` and `ci / verify` fail on every PR
and on master
> - This pull request refreshes those two values
> - The benefit is that PR CI is green again without a maintainer
override

## Linked Issues or Issue Description

Refs #20615. Refs #20550 (the same kind of resync).

**What happened?**
After #20615, the Runner support suite fails in
`source-sdk-public-inputs.test.mjs` ("source-sdk-public-inputs.json is
stale for 1 pinned input(s): pnpm-lock.yaml (changed)"). It also fails
in `source-sdk-evidence.test.ts` and `source-sdk-public-type.test.ts`
("Reviewed SDK reference policy does not match public proof").

**What did you expect?**
The pin registry matches the committed lockfile and the reviewed
report-lock policy, and the suite passes.

**Steps to reproduce**
On master `3bf7af6c77`, run `node
tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse HEAD)
--check`. It exits 1 with `"changed": true`.

## What Changed

- `inputs["pnpm-lock.yaml"]` is now the git blob of master's lockfile
(`6d825f97…`). I generated it with `node
tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD> --write`.
- `lockSha256` is now `b3208306…`. This is the `outputSha256` in
`tests/runner-e2e/report-lock.json` after #20615, and the sha256 of the
committed lockfile. `source-sdk-reviewed-fixture.ts` requires these two
values to be equal.
- No other values change.

## Verification

- `node tests/runner-e2e/source-sdk-public-inputs.mjs $(git rev-parse
HEAD) --check` exits 0 with `"changed": false`.
- `shasum -a 256 pnpm-lock.yaml` equals `report-lock.json`
`outputSha256` and the new `lockSha256`.
- CI `ci / Runner support suite` on this PR runs the three failing test
files.

## Risks

Low risk. The change is test pin data only. It does not change product
code, the lockfile, or the report-lock policy.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), Anthropic, with tool use in
Claude Code.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:20:48 -06:00
DottaandClaude Opus 5.5 42d4499489 fix(daytona): force a stuck stop before starting a retained sandbox (#20609) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - On Daytona, a follow-up task resumes the agent's retained sandbox
before the agent starts
> - The host stops an idle sandbox gracefully. A process that ignores
SIGTERM can leave Daytona in `stopping` for many minutes
> - Resume waited for that state change until its 300 s start deadline.
It never escalated, so each attempt failed after five minutes, and the
bounded retries did the same
> - On staging, a user comment waited about 21 minutes before the agent
started. Two resume attempts each used 299.7 s in the plugin dispatch
> - This pull request forces a stuck stop once, then starts the sandbox
within the same deadline. It also traces the wait with the observed
state
> - The benefit is that a stuck stop costs about 30 s instead of failing
the run, and the next case is diagnosable from the trace

## Linked Issues or Issue Description

**What happened?**
A user commented on a task that used a retained Daytona sandbox. The
first run and its first retry each failed after about 5 min 20 s with
`reusable_sandbox_resume_failed (cause=provider_error)`. The Honeycomb
trace shows `execution.plugin_dispatch` at 299.7 s in both. Inside it
there is only `lease.lookup` (44 ms) and `lease.channel_cleanup`, and no
`lease.start`. So the whole budget went to `waitOutSandboxTransition`,
which polled a sandbox that stayed in a transition state. The later
attempt, after a control-plane restart, found the sandbox `stopped`. Its
`lease.start` took 1.3 s. A transition that ends in `stopped` is a stop.

**Expected behavior**
A sandbox that stays `stopping` for longer than the host's own
graceful-stop budget is forced to stop, then starts. The run does not
fail on the start deadline.

**Steps to reproduce**
1. Retain a Daytona sandbox whose graceful stop does not finish (Daytona
reports `stopping`).
2. Resume or realize the lease.
3. Before this change, the call polls until the 300 s deadline and fails
with "is still stopping at its start deadline".

## What Changed

- `packages/plugins/sandbox-providers/daytona/src/plugin.ts`:
`waitOutSandboxTransition` tracks how long the sandbox stays `stopping`.
After `STUCK_STOPPING_FORCE_AFTER_MS` (30 s) it calls
`sandbox.stop(seconds, true)` once, then keeps polling. A refused force
stop keeps the old wait within the same deadline.
- The wait is a `lease.wait_transition` span. The force stop is a
`lease.force_stop` child span.
- `lease-tracing.ts`: the closed `paperclip.sandbox.startup.state` label
also records `starting`, `stopping`, `restoring`, `archiving` and
`resizing`. Before, these were all `other`.
- `server/src/services/plugin-host-services.ts`: the host admits the two
new span names and the new closed state values.
- `doc/observability.md`: documents the new spans and states.

## Verification

- Red/green: with the plugin change stashed, the three new
stuck-`stopping` tests (realize, refused force stop, resume) fail by
hanging to the deadline. With the change they pass. The "settles on its
own" test confirms that no force stop happens for a normal stop.
- `vitest run` in `packages/plugins/sandbox-providers/daytona`: 14
files, 618 tests passed (8 skipped). `tsc --noEmit` passed.
- `server/src/__tests__/plugin-host-services-span.test.ts`: red before
the allowlist change, 25/25 after.
- `pnpm verify` exited 2 (the doc is a global CI input), so CI supplies
the broad coverage.
- Not verified on a live Daytona sandbox. Hosted acceptance needs the
change deployed to staging.

## Risks

- The force stop sends SIGKILL to processes in a sandbox that is already
stopping and is about to start. The disk persists across a stop, so the
retained workspace is kept. A graceful stop that is still running past
30 s is killed. The host's own graceful stop budget is 30 s, so the host
had already given up on it.
- Daytona can refuse a stop while a state change is in progress. Then
the code waits as before, so the result is no worse than today.
- The root cause is inferred from the trace (a transition that ends in
`stopped`). The new state label confirms or refutes it the next time it
happens.

## Model Used

- Claude Opus 5.5 (`claude-opus-5-5`), via Claude Code with tool use
(shell, file edits, browser, test execution).

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass
- [x] I have added or updated tests where applicable
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 19:20:30 +00:00
DottaandClaude Opus 5.5 9b21dcbdda refactor: remove SPEC docs and repoint spec citations (#20565) 
## Thinking Path

> - Paperclip is the open source app people use to manage AI agents for
work
> - Contributors and agents read the docs under `doc/` before they
change code
> - `doc/SPEC.md` and `doc/SPEC-implementation.md` were out of date.
`doc/SPEC-implementation.md` became an append-only log: in 60 days it
got about 1,645 new lines and lost 181, across 140 commits
> - #20557 changed `AGENTS.md` so the code and its tests are the source
of truth, and the specs are only background
> - This pull request removes the two spec docs. It moves only the
sections that live code comments cite into focused docs, and it repoints
all other references
> - The benefit is one less stale document that agents read and append
to

## Linked Issues or Issue Description

Refs #20557 (merged). This PR is the follow-up that removes the docs. It
is based on `master`, which already contains #20557.

**Issue type**
Outdated content. The docs are out of date and conflict with the code.

**Where is the issue?**
`doc/SPEC.md`, `doc/SPEC-implementation.md`, and the code comments and
docs that cite them.

**What's wrong?**
`doc/SPEC.md` and `doc/SPEC-implementation.md` are out of date, but
agents still read them and append a paragraph with each fix.

**Suggested fix**
Delete both files. Move the sections that code cites into focused docs.
Repoint the references.

## What Changed

- Deleted `doc/SPEC.md` and `doc/SPEC-implementation.md`.
- Moved these spec sections, because live code comments cite them:
- §9.3.1 "Shared default-open issue writes" → new section in
`doc/agent-permission-defaults.md`.
- §9.8.1 "Issue-thread interaction resolver contract" →
`doc/execution-semantics.md`, "Issue-thread interaction resolution".
That section already had most of the contract. I added only the missing
parts: the `supersedeOnUserComment` default, the canonical policy list,
the addressee and cap rules, the rule that surfaces show the server's
effective audience, and the cross-issue budget charge. The short policy
summary at the top of the section now points to the list, so the
definitions occur one time only.
- §9.9 "Task Watchdog Authority Contract" → new "Authority contract"
section in `doc/TASK-WATCHDOG.md`. It holds the allowed and disallowed
mutation lists and the deferred capabilities. The atomic recovery batch,
restoration verification, and interaction rules were already in
`doc/execution-semantics.md` §11, so the new section links there and
does not repeat them. I did not move the "Downstream acceptance
criteria" checklist or the sentence about the CTO and Security sign-off.
These were implementation-time notes; the tests now hold that coverage.
- §9.10 "Company Skill Policy Contract" → new
`doc/company-skill-policy.md`. No focused doc had this contract. I did
not move the "Required regression tests" phase checklist.
- Did not move any other part of the specs.
- Repointed code comments that cited spec sections:
- `packages/shared/src/issue-write-denial.ts` (§9.3) →
`doc/agent-permission-defaults.md`
- `server/src/routes/issues.ts` (§9.9) → `doc/TASK-WATCHDOG.md`,
"Authority contract"
- `server/src/routes/issues.ts` (§9.3, §9.8.1 cross-issue budget) →
`doc/agent-permission-defaults.md`
- `ui/src/lib/interaction-audience.ts` and
`ui/src/components/InteractionGovernancePanel.tsx` (§9.8.1) →
`doc/execution-semantics.md`
- `ui/src/lib/skill-policy-denial.ts`, `ui/src/pages/CompanySkills.tsx`,
`ui/src/pages/CompanySkills.production.tsx`,
`ui/src/pages/SkillStudio.tsx` (§9.10) → `doc/company-skill-policy.md`
- `server/src/services/recovery/ownership-invariant-report.ts` (monitor
deadline wait path) → `doc/execution-semantics.md` §8. That section
already lists the one-shot monitor as a valid wait path.
- Resynced the git blob pin for
`packages/shared/src/issue-write-denial.ts` in
`tests/runner-e2e/source-sdk-public-inputs.json`. The comment edit
changed the file's hash, and the Runner support suite checks that pin. I
used `node tests/runner-e2e/source-sdk-public-inputs.mjs <HEAD>
--write`; only that one pin changed.
- Updated `AGENTS.md` §1 so it does not name the deleted files.
- Removed the links to the deleted files from
`doc/plugins/PLUGIN_SPEC.md`.
- Updated the `doc-maintenance` agent skill
(`.agents/skills/doc-maintenance/`) so it no longer audits or stages
`doc/SPEC.md`.
- Removed or replaced references in `maps/README.md`,
`maps/feature-map/README.md`, `maps/storybook-map/README.md`,
`doc/PRODUCT.md`, `doc/CLIPHUB.md`, `doc/DEPLOYMENT-MODES.md`,
`doc/DEVELOPING.md`, `doc/evals.md`, `doc/spec/agent-runs.md`,
`doc/AGENTCOMPANIES_SPEC_INVENTORY.md`,
`doc/connections/CONNECTOR-PLAYBOOK.md`,
`doc/plugins/ideas-from-opencode.md`, `doc/execution-semantics.md`,
`doc/TASK-WATCHDOG.md`, and `.devin/wiki.json`.
- In `doc/execution-semantics.md`, the "admission is independent of
cleanup and accounting" reference now states the rule inline: only a
hard stop, an explicit pause, or an inactive company refuses admission.
- Kept these references as they are: dated plans in `doc/plans/` and
`docs/plans/`, the dated log
`doc/logs/2026-05-24-cli-api-parity-e2e-log.md`, `releases/`, the
transcript fixture `ui/src/fixtures/runTranscriptFixtures.ts` (it uses
the filename only as sample text), and
`docs/specs/external-task-protocol.md` (it refers to the Symphony
project's own `SPEC.md`).

## Verification

- `git grep -n "SPEC-implementation\|SPEC\.md"` (without `releases/`,
and ignoring `PLUGIN_SPEC.md` citations) finds only the kept references
listed above.
- `grep -rn "§9\.\(3\|8\|9\|10\)" ui/src server/src packages/shared/src`
finds no results.
- `node scripts/check-maps.mjs`: "Maps aligned: 35 areas, 192 capability
IDs".
- `pnpm check:token-gates`: "All gates clean".
- `pnpm exec vitest run --config tests/runner-e2e/vitest.config.ts
tests/runner-e2e/source-sdk-public-type.test.ts
tests/runner-e2e/source-sdk-evidence.test.ts`: 45 passed. These failed
in CI before the pin resync.
- `pnpm verify`: exit 2 ("Broader verification required") because
`AGENTS.md` is a global CI input. Required CI supplies the broad
coverage for this revision.
- All code changes are comments only. No runtime behavior changes. The
title uses `refactor:` because comments in source files changed.

## Risks

- Low risk. The change removes docs and edits comments only.
- A reader can lose context from spec sections that I did not move. The
text stays in git history. The code and its tests are the source of
truth.
- External links to the two deleted files will return 404.

> For core feature work, check [`ROADMAP.md`](ROADMAP.md) first and
discuss it in `#dev` before opening the PR. Feature PRs that overlap
with planned core work may need to be redirected — check the roadmap
first. See `CONTRIBUTING.md`.

## Model Used

- Anthropic Claude Opus 5.5 (`claude-opus-5-5`), in Claude Code with
tool use (shell, file edits). Standard context window.

## Checklist

- [x] I have included a thinking path that traces from project context
to this change
- [x] I have specified the model used (with version and capability
details)
- [x] I have checked ROADMAP.md and confirmed this PR does not duplicate
planned core work
- [x] I have searched GitHub for duplicate or related PRs and linked
them above
- [x] I have either (a) linked existing issues with `Fixes: #` / `Closes
#` / `Refs #` OR (b) described the issue in-PR following the relevant
issue template
- [x] I have not referenced internal/instance-local Paperclip issues or
links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip`
URLs)
- [x] My branch name describes the change (e.g. `docs/...`, `fix/...`)
and contains no internal Paperclip ticket id or instance-derived details
- [x] I have run tests locally and they pass (focused checks above; CI
supplies broad coverage)
- [x] I have added or updated tests where applicable (not applicable:
docs and comments only)
- [x] I have updated relevant documentation to reflect my changes
- [x] I have considered and documented any risks above
- [ ] All Paperclip CI gates are green
- [ ] Greptile is 5/5 with no open P2s, recommendations, or follow-ups
- [x] I will address all Greptile and reviewer comments before
requesting merge

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-11-10 13:07:59 -06:00